Unix File System
Encyclopedia
The Unix file system is a file system
used by many Unix
and Unix-like
operating systems. It is also called the Berkeley
Fast File System, the BSD
Fast File System or FFS. It is a distant descendant of the original filesystem used by Version 7 Unix
.
Inodes are numbered sequentially, starting at 0. The first two inodes are reserved for historical reasons, followed by the inode for the root directory
, which is always inode 2.
Directory files contain only the list of filenames in the directory and the inode associated with each file. All file metadata is kept in the inode.
graduate student, optimized this for 4.2BSD's FFS (Fast File System) by inventing cylinder groups. This breaks the disk up into smaller chunks, each with its own inode clump and data blocks.
The intent of BSD FFS is to try to localize associated data blocks and metadata in the same cylinder group; and, ideally, all of the contents of a directory (both data and metadata for all the files) in the same or nearby cylinder group, thus reducing fragmentation
caused by scattering a directory's contents over a whole disk.
Some of the performance parameters in the superblock included number of tracks and sectors, disk rotation speed, head speed, and alignment of the sectors between tracks. In a fully optimized system, the head could be moved between close tracks to read scattered sectors from alternating tracks while waiting for the platter to spin around.
As disks grew larger and larger, sector level optimization became obsolete (especially with disks that used linear sector numbering and variable sectors per track). With larger disks and larger files, fragmented reads became more of a problem. To combat this, BSD originally increased the filesystem block size from one sector to 1K in 4.0BSD; and, in FFS, increased the filesystem block size from 1K to 8K. This has several effects. The chances of a file's sectors being contiguous is much greater. The amount of overhead to list the file's blocks is reduced. The number of bytes representable by any given number of blocks is increased; since the maximum number of blocks is limited by a fixed bit-width block number, this also allows for larger disks.
With larger block sizes, disks with many small files would waste a lot of space, so BSD added block level fragmentation (also called block suballocation, tail merging or tail packing
), where the last partial block of data from several files may be stored in a single "fragment" block instead of multiple mostly empty blocks .
/Solaris, System V Release 4, HP-UX
, and Tru64 UNIX
, have adopted UFS. Most of them adapted UFS to their own uses, adding proprietary extensions that may not be recognized by other vendors' versions of Unix. Surprisingly, many have continued to use the original block size and data field widths as the original UFS, so some degree of (read) compatibility remains across platforms. Compatibility between implementations as a whole is spotty at best and should be researched before using it across multiple platforms where shared data is a primary intent.
As of Solaris 7, Sun Microsystems
included UFS Logging, which brought filesystem journaling
to UFS. Solaris UFS also has extensions for large files and large disks and other features.
In 4.4BSD and BSD
Unix systems derived from it, such as FreeBSD
, NetBSD
, OpenBSD
, and DragonFlyBSD, the implementation of UFS1 and UFS2 is split into two layers — an upper layer that provides the directory structure and supports metadata (permissions, ownership, etc.) in the inode structure, and lower layers that provide data containers implemented as inodes. This was done to support both the traditional FFS and the LFS log-structured file system with shared code for common functions. The upper layer is called "UFS", and the lower layers are called "FFS" and "LFS". In some of those systems, the term "FFS" is used for the combination of the FFS lower layer and the UFS upper layer, and the term "LFS" is used for the combination of the LFS lower layer and the UFS upper layer.
Kirk McKusick implemented block reallocation, a technique that reorders the blocks in the file system just before the writes are done to reduce fragmentation and control file system aging. He also implemented soft updates
, a mechanism that maintains the file system consistency without limiting the performance in the way the traditional sync mode did. This has the side effect of reducing the requirement of file system checking after a crash or power failure. To overcome the remaining issues after a failure, a background fsck utility was introduced.
In UFS2, Kirk McKusick and Poul-Henning Kamp
extended the FreeBSD FFS and UFS layers to add 64-bit block pointers (allowing volumes to grow up to 8 zettabytes), variable-sized blocks (similar to extents), extended flag fields, additional 'birthtime' stamps, extended attribute support and POSIX1.e ACLs. UFS2 became the default UFS version starting with FreeBSD 5.0. FreeBSD also introduced soft updates
and the ability to make file system snapshots
for both UFS1 and UFS2. These have since been ported to NetBSD, but eventually soft updates (called soft dependencies in NetBSD) was removed from NetBSD 6.0 in favor of the less complex file system journaling mechanism called WAPBL (also referred as logging), which was added to FFS in NetBSD 5.0. OpenBSD has supported soft updates since version 2.9 http://www.openbsd.org/29.html and has had UFS2 support (no ACLs) since version 4.2 http://www.openbsd.org/42.html. Since FreeBSD 7.0, UFS also supports filesystem journaling
using the gjournal GEOM
provider. FreeBSD 9.0 adds support for lightweight journaling on top of softupdates(SU+J), which greatly reduces and need for background fsck, and uses NFS-style ACLs by default.
Linux
includes a UFS implementation for binary compatibility at the read level with other Unixes, but since there is no standard implementation for the vendor extensions to UFS, Linux does not have full support for writing to UFS. The native Linux ext2
filesystem was inspired by UFS1 but doesn't support fragments and there are no plans to implement softupdates. (In some 4.4BSD-derived systems, the UFS layer can use an ext2 layer as a container layer, just as it can use FFS and LFS.)
NeXTStep
, which was BSD-derived, also used a version of UFS. In Apple's Mac OS X
, it is available as an alternative to HFS+
, their proprietary filesystem. However, as of Mac OS X v10.5
, one cannot install Mac OS X "Leopard" on a UFS-formatted volume. In addition, one cannot upgrade older versions of Mac OS X installed on UFS-formatted volumes to Leopard; upgrading requires reformatting the startup volume. Currently there is a 4GB file limit for disks formatted as UFS in Mac OSX.
The PlayStation 3
game console uses UFS2 on HDD. The PlayStation 2
HDD used UFS as well.
File system
A file system is a means to organize data expected to be retained after a program terminates by providing procedures to store, retrieve and update data, as well as manage the available space on the device which contain it. A file system organizes data in an efficient manner and is tuned to the...
used by many Unix
Unix
Unix is a multitasking, multi-user computer operating system originally developed in 1969 by a group of AT&T employees at Bell Labs, including Ken Thompson, Dennis Ritchie, Brian Kernighan, Douglas McIlroy, and Joe Ossanna...
and Unix-like
Unix-like
A Unix-like operating system is one that behaves in a manner similar to a Unix system, while not necessarily conforming to or being certified to any version of the Single UNIX Specification....
operating systems. It is also called the Berkeley
University of California, Berkeley
The University of California, Berkeley , is a teaching and research university established in 1868 and located in Berkeley, California, USA...
Fast File System, the BSD
Berkeley Software Distribution
Berkeley Software Distribution is a Unix operating system derivative developed and distributed by the Computer Systems Research Group of the University of California, Berkeley, from 1977 to 1995...
Fast File System or FFS. It is a distant descendant of the original filesystem used by Version 7 Unix
Version 7 Unix
Seventh Edition Unix, also called Version 7 Unix, Version 7 or just V7, was an important early release of the Unix operating system. V7, released in 1979, was the last Bell Laboratories release to see widespread distribution before the commercialization of Unix by AT&T in the early 1980s...
.
Design
A UFS volume is composed of the following parts:- a few blocks at the beginning of the partition reserved for boot blocks (which must be initialized separately from the filesystem)
- a superblock, containing a magic numberMagic number (programming)In computer programming, the term magic number has multiple meanings. It could refer to one or more of the following:* A constant numerical or text value used to identify a file format or protocol; for files, see List of file signatures...
identifying this as a UFS filesystem, and some other vital numbers describing this filesystem's geometry and statistics and behavioral tuning parameters - a collection of cylinder groups. Each cylinder group has the following components:
- a backup copy of the superblock
- a cylinder group header, with statistics, free lists, etc., about this cylinder group, similar to those in the superblock
- a number of inodeInodeIn computing, an inode is a data structure on a traditional Unix-style file system such as UFS. An inode stores all the information about a regular file, directory, or other file system object, except its data and name....
s, each containing file attributes - a number of data blocksBlock (data storage)In computing , a block is a sequence of bytes or bits, having a nominal length . Data thus structured are said to be blocked. The process of putting data into blocks is called blocking. Blocking is used to facilitate the handling of the data-stream by the computer program receiving the data...
Inodes are numbered sequentially, starting at 0. The first two inodes are reserved for historical reasons, followed by the inode for the root directory
Root directory
In computer file systems, the root directory is the first or top-most directory in a hierarchy. It can be likened to the root of a tree — the starting point where all branches originate.-Metaphor:...
, which is always inode 2.
Directory files contain only the list of filenames in the directory and the inode associated with each file. All file metadata is kept in the inode.
History and evolution
Early versions of Unix filesystems were referred to simply as FS. FS only included the boot block, superblock, a clump of inodes, and the data blocks. This worked well for the small disks early Unixes were designed for, but as technology advanced and disks got larger, moving the head back and forth between the clump of inodes and the data blocks they referred to caused thrashing. Marshall Kirk McKusick, then a BerkeleyUniversity of California, Berkeley
The University of California, Berkeley , is a teaching and research university established in 1868 and located in Berkeley, California, USA...
graduate student, optimized this for 4.2BSD's FFS (Fast File System) by inventing cylinder groups. This breaks the disk up into smaller chunks, each with its own inode clump and data blocks.
The intent of BSD FFS is to try to localize associated data blocks and metadata in the same cylinder group; and, ideally, all of the contents of a directory (both data and metadata for all the files) in the same or nearby cylinder group, thus reducing fragmentation
File system fragmentation
In computing, file system fragmentation, sometimes called file system aging, is the inability of a file system to lay out related data sequentially , an inherent phenomenon in storage-backed file systems that allow in-place modification of their contents. It is a special case of data fragmentation...
caused by scattering a directory's contents over a whole disk.
Some of the performance parameters in the superblock included number of tracks and sectors, disk rotation speed, head speed, and alignment of the sectors between tracks. In a fully optimized system, the head could be moved between close tracks to read scattered sectors from alternating tracks while waiting for the platter to spin around.
As disks grew larger and larger, sector level optimization became obsolete (especially with disks that used linear sector numbering and variable sectors per track). With larger disks and larger files, fragmented reads became more of a problem. To combat this, BSD originally increased the filesystem block size from one sector to 1K in 4.0BSD; and, in FFS, increased the filesystem block size from 1K to 8K. This has several effects. The chances of a file's sectors being contiguous is much greater. The amount of overhead to list the file's blocks is reduced. The number of bytes representable by any given number of blocks is increased; since the maximum number of blocks is limited by a fixed bit-width block number, this also allows for larger disks.
With larger block sizes, disks with many small files would waste a lot of space, so BSD added block level fragmentation (also called block suballocation, tail merging or tail packing
Block suballocation
Block suballocation is a feature of some computer file systems which allows large blocks or allocation units to be used while making efficient use of "slack" space at the end of large files, space which would otherwise be lost for other use to internal fragmentation.In file systems that don't...
), where the last partial block of data from several files may be stored in a single "fragment" block instead of multiple mostly empty blocks .
Implementations
Vendors of some proprietary Unix systems, such as SunOSSunOS
SunOS is a version of the Unix operating system developed by Sun Microsystems for their workstation and server computer systems. The SunOS name is usually only used to refer to versions 1.0 to 4.1.4 of SunOS...
/Solaris, System V Release 4, HP-UX
HP-UX
HP-UX is Hewlett-Packard's proprietary implementation of the Unix operating system, based on UNIX System V and first released in 1984...
, and Tru64 UNIX
Tru64 UNIX
Tru64 UNIX is a 64-bit UNIX operating system for the Alpha instruction set architecture , currently owned by Hewlett-Packard . Previously, Tru64 UNIX was a product of Compaq, and before that, Digital Equipment Corporation , where it was known as Digital UNIX .As its original name suggests, Tru64...
, have adopted UFS. Most of them adapted UFS to their own uses, adding proprietary extensions that may not be recognized by other vendors' versions of Unix. Surprisingly, many have continued to use the original block size and data field widths as the original UFS, so some degree of (read) compatibility remains across platforms. Compatibility between implementations as a whole is spotty at best and should be researched before using it across multiple platforms where shared data is a primary intent.
As of Solaris 7, Sun Microsystems
Sun Microsystems
Sun Microsystems, Inc. was a company that sold :computers, computer components, :computer software, and :information technology services. Sun was founded on February 24, 1982...
included UFS Logging, which brought filesystem journaling
Journaling file system
A journaling file system is a file system that keeps track of the changes that will be made in a journal before committing them to the main file system...
to UFS. Solaris UFS also has extensions for large files and large disks and other features.
In 4.4BSD and BSD
Berkeley Software Distribution
Berkeley Software Distribution is a Unix operating system derivative developed and distributed by the Computer Systems Research Group of the University of California, Berkeley, from 1977 to 1995...
Unix systems derived from it, such as FreeBSD
FreeBSD
FreeBSD is a free Unix-like operating system descended from AT&T UNIX via BSD UNIX. Although for legal reasons FreeBSD cannot be called “UNIX”, as the direct descendant of BSD UNIX , FreeBSD’s internals and system APIs are UNIX-compliant...
, NetBSD
NetBSD
NetBSD is a freely available open source version of the Berkeley Software Distribution Unix operating system. It was the second open source BSD descendant to be formally released, after 386BSD, and continues to be actively developed. The NetBSD project is primarily focused on high quality design,...
, OpenBSD
OpenBSD
OpenBSD is a Unix-like computer operating system descended from Berkeley Software Distribution , a Unix derivative developed at the University of California, Berkeley. It was forked from NetBSD by project leader Theo de Raadt in late 1995...
, and DragonFlyBSD, the implementation of UFS1 and UFS2 is split into two layers — an upper layer that provides the directory structure and supports metadata (permissions, ownership, etc.) in the inode structure, and lower layers that provide data containers implemented as inodes. This was done to support both the traditional FFS and the LFS log-structured file system with shared code for common functions. The upper layer is called "UFS", and the lower layers are called "FFS" and "LFS". In some of those systems, the term "FFS" is used for the combination of the FFS lower layer and the UFS upper layer, and the term "LFS" is used for the combination of the LFS lower layer and the UFS upper layer.
Kirk McKusick implemented block reallocation, a technique that reorders the blocks in the file system just before the writes are done to reduce fragmentation and control file system aging. He also implemented soft updates
Soft updates
In computer file systems, soft updates is an approach to maintaining disk integrity after a crash or power outage. They are an alternative to journaling file systems....
, a mechanism that maintains the file system consistency without limiting the performance in the way the traditional sync mode did. This has the side effect of reducing the requirement of file system checking after a crash or power failure. To overcome the remaining issues after a failure, a background fsck utility was introduced.
In UFS2, Kirk McKusick and Poul-Henning Kamp
Poul-Henning Kamp
Poul-Henning Kamp is a Danish FreeBSD developer, responsible for implementation of the widely used MD5 password hash algorithm, a vast quantity of systems code, including the FreeBSD GEOM storage layer, GBDE cryptographic storage transform, part of the UFS2 file system implementation, FreeBSD...
extended the FreeBSD FFS and UFS layers to add 64-bit block pointers (allowing volumes to grow up to 8 zettabytes), variable-sized blocks (similar to extents), extended flag fields, additional 'birthtime' stamps, extended attribute support and POSIX1.e ACLs. UFS2 became the default UFS version starting with FreeBSD 5.0. FreeBSD also introduced soft updates
Soft updates
In computer file systems, soft updates is an approach to maintaining disk integrity after a crash or power outage. They are an alternative to journaling file systems....
and the ability to make file system snapshots
Snapshot (computer storage)
In computer systems, a snapshot is the state of a system at a particular point in time. The term was coined as an analogy to that in photography. It can refer to an actual copy of the state of a system or to a capability provided by certain systems....
for both UFS1 and UFS2. These have since been ported to NetBSD, but eventually soft updates (called soft dependencies in NetBSD) was removed from NetBSD 6.0 in favor of the less complex file system journaling mechanism called WAPBL (also referred as logging), which was added to FFS in NetBSD 5.0. OpenBSD has supported soft updates since version 2.9 http://www.openbsd.org/29.html and has had UFS2 support (no ACLs) since version 4.2 http://www.openbsd.org/42.html. Since FreeBSD 7.0, UFS also supports filesystem journaling
Journaling file system
A journaling file system is a file system that keeps track of the changes that will be made in a journal before committing them to the main file system...
using the gjournal GEOM
GEOM
GEOM is the main storage framework for the FreeBSD operating system. It is available in FreeBSD 5.0 and higher and provides a standardized way to access storage layers. GEOM is modular and allows for geom modules to connect to the framework. For example, the geom_mirror module will provide RAID1 or...
provider. FreeBSD 9.0 adds support for lightweight journaling on top of softupdates(SU+J), which greatly reduces and need for background fsck, and uses NFS-style ACLs by default.
Linux
Linux
Linux is a Unix-like computer operating system assembled under the model of free and open source software development and distribution. The defining component of any Linux system is the Linux kernel, an operating system kernel first released October 5, 1991 by Linus Torvalds...
includes a UFS implementation for binary compatibility at the read level with other Unixes, but since there is no standard implementation for the vendor extensions to UFS, Linux does not have full support for writing to UFS. The native Linux ext2
Ext2
The ext2 or second extended filesystem is a file system for the Linux kernel. It was initially designed by Rémy Card as a replacement for the extended file system ....
filesystem was inspired by UFS1 but doesn't support fragments and there are no plans to implement softupdates. (In some 4.4BSD-derived systems, the UFS layer can use an ext2 layer as a container layer, just as it can use FFS and LFS.)
NeXTStep
NEXTSTEP
NeXTSTEP was the object-oriented, multitasking operating system developed by NeXT Computer to run on its range of proprietary workstation computers, such as the NeXTcube...
, which was BSD-derived, also used a version of UFS. In Apple's Mac OS X
Mac OS X
Mac OS X is a series of Unix-based operating systems and graphical user interfaces developed, marketed, and sold by Apple Inc. Since 2002, has been included with all new Macintosh computer systems...
, it is available as an alternative to HFS+
HFS Plus
HFS Plus or HFS+ is a file system developed by Apple Inc. to replace their Hierarchical File System as the primary file system used in Macintosh computers . It is also one of the formats used by the iPod digital music player...
, their proprietary filesystem. However, as of Mac OS X v10.5
Mac OS X v10.5
Mac OS X Leopard is the sixth major release of Mac OS X, Apple's desktop and server operating system for Macintosh computers. Leopard was released on 26 October 2007 as the successor of Tiger , and is available in two variants: a desktop version suitable for personal computers, and a...
, one cannot install Mac OS X "Leopard" on a UFS-formatted volume. In addition, one cannot upgrade older versions of Mac OS X installed on UFS-formatted volumes to Leopard; upgrading requires reformatting the startup volume. Currently there is a 4GB file limit for disks formatted as UFS in Mac OSX.
The PlayStation 3
PlayStation 3
The is the third home video game console produced by Sony Computer Entertainment and the successor to the PlayStation 2 as part of the PlayStation series. The PlayStation 3 competes with Microsoft's Xbox 360 and Nintendo's Wii as part of the seventh generation of video game consoles...
game console uses UFS2 on HDD. The PlayStation 2
PlayStation 2
The PlayStation 2 is a sixth-generation video game console manufactured by Sony as part of the PlayStation series. Its development was announced in March 1999 and it was first released on March 4, 2000, in Japan...
HDD used UFS as well.
External links
- Little UFS2 FAQ
- Linux userspace UFS2 tools.
- Filesystems-HOWTO (part of The Linux Documentation Project, link is not Linux specific.)
- UFS2 Tools: An open source tool for accessing UFS2 (BSD) slices from within Windows
- http://docs.info.apple.com/article.html?artnum=306516 No UFS installation for Leopard
- http://www.freebsd.org/projects/bigdisk/index.html