Lustre (file system)
Encyclopedia
Lustre is a massively parallel distributed file system
, generally used for large scale cluster computing
. The name Lustre is a portmanteau word
derived from Linux
and cluster. Available under the GNU GPL, the project provides a high performance file system for clusters of tens of thousands of node
s with petabyte
s of storage capacity.
Lustre file systems are used in computer clusters ranging from small workgroup clusters to large-scale, multi-site clusters. Fifteen of the top 30 supercomputers in the world use Lustre file systems, including the world's fastest TOP500
supercomputer, K computer
.
Lustre file systems can support tens of thousands of client
systems, tens of petabytes (PBs) of storage and hundreds of gigabytes per second (GB/s) of I/O throughput. Due to Lustre's high scalability
, businesses such as Internet service providers, financial institutions, and the oil and gas industry deploy Lustre file systems in their data centers.
at the time. Braam went on to found his own company Cluster File Systems
, which released Lustre 1.0 in 2003. In 2007, Sun Microsystems
acquired Cluster File Systems Inc. Sun included Lustre with its HPC hardware offerings, with the intent to bring the benefits of Lustre technologies to Sun's ZFS
file system
and the Solaris operating system
. In November 2008, Braam left Sun Microsystems to work on another filesystem, leaving Eric Barton and Andreas Dilger in charge of Lustre architecture and development.
In 2010 Oracle Corporation
, by way of its 2010 acquisition of Sun, began to manage and release Lustre.
In April 2010 Oracle announced it would limit paid support for new Lustre 2.0 deployment to Oracle hardware, or hardware provided by approved third party vendors. Lustre remained available under the GPL license to all users, and existing Lustre 1.8 customers would continue to receive support from Oracle.
In December 2010, Oracle announced cessation of Lustre development. Lustre 1.8 release was placed into maintenance-only support creating uncertainty around the future development of the file system. Following this announcement, new Lustre support and development was provided by a community, including Whamcloud, Xyratex
, OpenSFS, EUROPEAN Open File Systems(EOFS) and others. In the same year, Eric Barton and Andreas Dilger left Oracle for the Lustre-centric startup Whamcloud, where they continue to work on Lustre.
Lustre 1.2.0, released in March 2004, provided Linux kernel 2.6 support, a "size glimpse" feature to avoid lock revocation on files undergoing write, and client side data write-back cache accounting (grant).
Lustre 1.4.0, released in November 2004, provided protocol compatibility between versions, InfiniBand
network support, and support for extents/mballoc in the ldiskfs
on-disk filesystem.
Lustre 1.6.0, released in April 2007, supported mount configuration (“mountconf”) allowing servers to be configured with "mkfs" and "mount", supported dynamic addition of object storage targets (OSTs), enabled Lustre distributed lock manager (LDLM) scalability on symmetric multiprocessing
(SMP) servers, and supported free space management for object allocations.
Lustre 1.8.0, released in May 2009, provided OSS Read Cache, improves recovery in the face of multiple failures, adds basic heterogeneous storage management via OST Pools, adaptive network timeouts, and version-based recovery. It also serves as a transition release, being interoperable with both Lustre 1.6 and Lustre 2.0.
Lustre 2.0.0, released in August 2010, provided a rewritten metadata server stack to provide a basis for Clustered Metadata (CMD) to allow distribution of the Lustre metadata across multiple metadata servers, a new Client IO stack (CLIO) for portability to other client operating systems such as Mac OS
, Windows, and Solaris, and an abstracted Object Storage Device (OSD) back-end for portability to other filesystems such as ZFS
. Lustre 2.x clients cannot interoperate with 1.8 or earlier servers. However, Lustre 1.8.6 and later clients can interoperate with Lustre 2.0 servers. The Lustre 2.x on-disk format is compatible with Lustre 1.8 to allow upgrades of existing filesystems to Lustre 2.
The Lustre file system and associated open source software has been adopted by many partners. Both Red Hat
and SUSE
(Novell
) offer Linux kernels that work without patches on the client for easy deployment.
The MDT, OST, and client can be on the same node, but in typical installations these functions are on separate nodes communicating over a network. The Lustre Network (LNET) layer supports several network interconnects, including native Infiniband
verbs, TCP/IP on Ethernet
and other networks, Myrinet
, Quadrics
, and other proprietary network technologies. Lustre will take advantage of remote direct memory access (RDMA) transfers, when available, to improve throughput and reduce CPU usage.
The storage used for the MDT and OST backing filesystems is partitioned, optionally organized with logical volume management
(LVM) and/or RAID
, and normally formatted as ext4
file systems. The Lustre OSS and MDS servers read, write, and modify data in the format imposed by these file systems.
An OST is a dedicated filesystem that exports an interface to byte ranges of objects for read/write operations. An MDT is a dedicated filesystem that controls file access and tells clients which object(s) make up a file. MDTs and OSTs currently use an enhanced version of ext4
called ldiskfs to store data. Work started in 2008 at Sun to port Lustre to Sun's ZFS
/DMU for back-end data storage and continues as an open source project.
When a client accesses a file, it completes a filename lookup on the MDS. As a result, a file is created on behalf of the client or the layout of an existing file is returned to the client. For read or write operations, the client then interprets the layout in the logical object volume (LOV) layer, which maps the offset and size to one or more objects, each residing on a separate OST. The client then locks the file range being operated on and executes one or more parallel read or write operations directly to the OSTs. With this approach, bottlenecks for client-to-OST communications are eliminated, so the total bandwidth available for the clients to read and write data scales almost linearly with the number of OSTs in the filesystem.
Clients do not directly modify the objects on the OST filesystems, but, instead, delegate this task to OSSes. This approach ensures scalability for large-scale clusters and supercomputers, as well as improved security and reliability. In contrast, shared block-based filesystems such as Global File System
and OCFS
must allow direct access to the underlying storage by all of the clients in the filesystem and increase the risk of filesystem corruption from misbehaving/defective clients.
On some massively parallel processor (MPP) installations, computational processors can access a Lustre file system by redirecting their I/O requests to a dedicated I/O node configured as a Lustre client. This approach is used in the Blue Gene
installation at LLNL.
Another approach used in the past is the liblustre library, which provided userspace applications with direct filesystem access. Liblustre was a user-level library that allows computational processors to mount and use the Lustre file system as a client. Using liblustre, the computational processors could access a Lustre file system even if the service node on which the job was launched is not a Lustre client. Liblustre allowed data movement directly between application space and the Lustre OSSs without requiring an intervening data copy through the kernel, thus providing low latency, high bandwidth access from computational processors to the Lustre file system directly.
If only one OST object is associated with an MDT inode, that object contains all the data in the Lustre file. When more than one object is associated with a file, data in the file is “striped” across the objects similar to RAID 0. Striping a file over multiple objects provides significant performance benefits. When striping is used, the maximum file size is not limited by the size of a single target. Capacity and aggregate I/O bandwidth scale with the number of OSTs a file is striped over. Also, since the locking of each object is managed independently for each OST, adding more stripes (OSTs) scales the file IO locking capability of the filesystem proportionately. Each file in the filesystem can have a different striping layout, so that performance and capacity can be tuned optimally for each file.
in the style of the VMS style to protect the integrity of each file's data and metadata. Access and modification of a Lustre file is completely cache coherent
among all of the clients. Metadata locks are managed by the MDT that stores the inode
for the file, using the 128-bit Lustre File Identifier (FID, composed of the Sequence number and Object ID) as the resource name. The metadata locks are split into multiple bits that protect the lookup of the file (file owner and group, permission and mode, and access control list
(ACL)), the state of the inode (directory size, directory contents, link count, timestamps), and layout (file striping). A client can fetch multiple metadata lock bits for a single inode with a single RPC request, but currently they are only ever granted a read lock for the inode. The MDS manages all modifications to the inode in order to avoid lock resource contention
and is currently the only node that gets write locks on inodes.
File data locks are managed by the OST on which each object of the file is striped, using byte-range extent locks. Clients can be granted both overlapping read extent locks for part or all of the file, allowing multiple concurrent readers of the same file, and/or non-overlapping write extent locks for regions of the file. This allows many Lustre clients to access a single file concurrently for both read and write, avoiding bottlenecks during file IO. In practice, because Linux clients manage their data cache in units of pages, the clients will request locks that are always an integer multiple of the page size (4096 bytes on most clients). When a client is requesting an extent lock the OST may grant a lock for a larger extent than requested, in order to reduce the number of lock requests that the client makes. The actual size of the granted lock depends on several factors, including the number of currently-granted locks, whether there are conflicting write locks, and the number of outstanding lock requests. The granted lock is never smaller than the originally-requested extent. OST extent locks use the Lustre FID as the resource name for the lock. Since the number of extent lock servers scales with the number of OSTs in the filesystem, this also scales the aggregate locking performance of the filesystem, and of a single file if it is striped over multiple OSTs.
(SAN) technologies.
LNET supports many commonly-used network types, such as InfiniBand and IP networks, and allows simultaneous availability across multiple network types with routing between them. Remote Direct Memory Access
(RDMA) is permitted when supported by underlying networks such as Quadrics Elan, Myrinet, and InfiniBand. High availability and recovery features enable transparent recovery in conjunction with failover servers.
LNET provides end-to-end throughput over Gigabit Ethernet
(GigE) networks in excess of 100 MB/s, throughput up to 3 GB/s using InfiniBand quad data rate (QDR) links, and throughput over 1 GB/s across 10GigE interfaces.
Lustre MDSes are configured as an active/passive pair, while OSSes are typically deployed in an active/active configuration that provides redundancy without extra overhead. Often the standby MDS is the active MDS for another Lustre file system, so no nodes are idle in the cluster.
at the RIKEN Advanced Institute for Computational Science
, and the second fastest (fastest from Nov 2010 to June 2011), the Tianhe-1A at the National Supercomputing Center in Tianjin, China. Other supercomputers that use the Lustre file system include the third fastest Jaguar supercomputer
at Oak Ridge National Laboratory
(ORNL) and systems at the National Energy Research Scientific Computing Center
located at Lawrence Berkeley National Laboratory
(LBNL), Lawrence Livermore National Laboratory
(LLNL), Pacific Northwest National Laboratory
, Texas Advanced Computing Center
and NASA
in North America, in Asia at Tokyo Institute of Technology
, and one of the largest systems in Europe at CEA
.
, the original developers of Lustre, and more recently Whamcloud, founded by many of the original Lustre developers). A non-exhaustive list of vendors selling bundled computing and Lustre storage systems include Cray
, Dell
, Hewlett-Packard
, BULL
, SGI
, and others. Sun Microsystems no longer sells either systems or storage that include Lustre. Major vendors selling storage hardware with bundled Lustre support include Data Direct Networks (DDN), Dell
, Terascala, Xyratex
, and many others.
Distributed file system
Network file system may refer to:* A distributed file system, which is accessed over a computer network* Network File System , a specific brand of distributed file system...
, generally used for large scale cluster computing
Cluster Computing
Cluster Computing: the Journal of Networks, Software Tools and Applications is a journal for parallel processing, distributed computing systems, and computer communication networks....
. The name Lustre is a portmanteau word
Portmanteau word
A portmanteau or portmanteau word is a blend of two words or morphemes into one new word. A portmanteau word typically combines both sounds and meanings, as in smog, coined by blending smoke and fog. More generally, it may refer to any term or phrase that combines two or more meanings...
derived from Linux
Linux
Linux is a Unix-like computer operating system assembled under the model of free and open source software development and distribution. The defining component of any Linux system is the Linux kernel, an operating system kernel first released October 5, 1991 by Linus Torvalds...
and cluster. Available under the GNU GPL, the project provides a high performance file system for clusters of tens of thousands of node
Node (networking)
In communication networks, a node is a connection point, either a redistribution point or a communication endpoint . The definition of a node depends on the network and protocol layer referred to...
s with petabyte
Petabyte
A petabyte is a unit of information equal to one quadrillion bytes, or 1000 terabytes. The unit symbol for the petabyte is PB...
s of storage capacity.
Lustre file systems are used in computer clusters ranging from small workgroup clusters to large-scale, multi-site clusters. Fifteen of the top 30 supercomputers in the world use Lustre file systems, including the world's fastest TOP500
TOP500
The TOP500 project ranks and details the 500 most powerful known computer systems in the world. The project was started in 1993 and publishes an updated list of the supercomputers twice a year...
supercomputer, K computer
K computer
The K computer – named for the Japanese word , which stands for 10 quadrillion – is a supercomputer being produced by Fujitsu at the RIKEN Advanced Institute for Computational Science campus in Kobe, Japan. In June 2011, TOP500 ranked K the world's fastest supercomputer, with a rating...
.
Lustre file systems can support tens of thousands of client
Client (computing)
A client is an application or system that accesses a service made available by a server. The server is often on another computer system, in which case the client accesses the service by way of a network....
systems, tens of petabytes (PBs) of storage and hundreds of gigabytes per second (GB/s) of I/O throughput. Due to Lustre's high scalability
Scalability
In electronics scalability is the ability of a system, network, or process, to handle growing amount of work in a graceful manner or its ability to be enlarged to accommodate that growth...
, businesses such as Internet service providers, financial institutions, and the oil and gas industry deploy Lustre file systems in their data centers.
History
The Lustre file system architecture was developed as a research project in 1999 by Peter Braam, who was a Senior Systems Scientist at Carnegie Mellon UniversityCarnegie Mellon University
Carnegie Mellon University is a private research university in Pittsburgh, Pennsylvania, United States....
at the time. Braam went on to found his own company Cluster File Systems
Cluster File Systems
Cluster File Systems, Inc. is the company that originally developed the Lustre distributed file system. CFS was a privately held company with offices in the United States, and China.CFS was founded in 2001 by Dr. Peter Braam...
, which released Lustre 1.0 in 2003. In 2007, Sun Microsystems
Sun Microsystems
Sun Microsystems, Inc. was a company that sold :computers, computer components, :computer software, and :information technology services. Sun was founded on February 24, 1982...
acquired Cluster File Systems Inc. Sun included Lustre with its HPC hardware offerings, with the intent to bring the benefits of Lustre technologies to Sun's ZFS
ZFS
In computing, ZFS is a combined file system and logical volume manager designed by Sun Microsystems. The features of ZFS include data integrity verification against data corruption modes , support for high storage capacities, integration of the concepts of filesystem and volume management,...
file system
File system
A file system is a means to organize data expected to be retained after a program terminates by providing procedures to store, retrieve and update data, as well as manage the available space on the device which contain it. A file system organizes data in an efficient manner and is tuned to the...
and the Solaris operating system
Solaris Operating System
Solaris is a Unix operating system originally developed by Sun Microsystems. It superseded their earlier SunOS in 1993. Oracle Solaris, as it is now known, has been owned by Oracle Corporation since Oracle's acquisition of Sun in January 2010....
. In November 2008, Braam left Sun Microsystems to work on another filesystem, leaving Eric Barton and Andreas Dilger in charge of Lustre architecture and development.
In 2010 Oracle Corporation
Oracle Corporation
Oracle Corporation is an American multinational computer technology corporation that specializes in developing and marketing hardware systems and enterprise software products – particularly database management systems...
, by way of its 2010 acquisition of Sun, began to manage and release Lustre.
In April 2010 Oracle announced it would limit paid support for new Lustre 2.0 deployment to Oracle hardware, or hardware provided by approved third party vendors. Lustre remained available under the GPL license to all users, and existing Lustre 1.8 customers would continue to receive support from Oracle.
In December 2010, Oracle announced cessation of Lustre development. Lustre 1.8 release was placed into maintenance-only support creating uncertainty around the future development of the file system. Following this announcement, new Lustre support and development was provided by a community, including Whamcloud, Xyratex
Xyratex
Xyratex is a provider of disk drive manufacturing equipment and data storage subsystems with headquarters of Xyratex International in Havant, United Kingdom and U.S. headquarters in Fremont, CA. Xyratex has two business divisions serving distinct storage markets. They are the Networked Storage...
, OpenSFS, EUROPEAN Open File Systems(EOFS) and others. In the same year, Eric Barton and Andreas Dilger left Oracle for the Lustre-centric startup Whamcloud, where they continue to work on Lustre.
Release history
The Lustre file system was first installed for production use in March 2003 on the MCR Linux Cluster at LLNL, one of the largest supercomputers at the time.Lustre 1.2.0, released in March 2004, provided Linux kernel 2.6 support, a "size glimpse" feature to avoid lock revocation on files undergoing write, and client side data write-back cache accounting (grant).
Lustre 1.4.0, released in November 2004, provided protocol compatibility between versions, InfiniBand
InfiniBand
InfiniBand is a switched fabric communications link used in high-performance computing and enterprise data centers. Its features include high throughput, low latency, quality of service and failover, and it is designed to be scalable...
network support, and support for extents/mballoc in the ldiskfs
Ext4
The ext4 or fourth extended filesystem is a journaling file system for Linux, developed as the successor to ext3.It was born as a series of backward compatible extensions to ext3, many of them originally developed by Cluster File Systems for the Lustre file system between 2003 and 2006, meant to...
on-disk filesystem.
Lustre 1.6.0, released in April 2007, supported mount configuration (“mountconf”) allowing servers to be configured with "mkfs" and "mount", supported dynamic addition of object storage targets (OSTs), enabled Lustre distributed lock manager (LDLM) scalability on symmetric multiprocessing
Symmetric multiprocessing
In computing, symmetric multiprocessing involves a multiprocessor computer hardware architecture where two or more identical processors are connected to a single shared main memory and are controlled by a single OS instance. Most common multiprocessor systems today use an SMP architecture...
(SMP) servers, and supported free space management for object allocations.
Lustre 1.8.0, released in May 2009, provided OSS Read Cache, improves recovery in the face of multiple failures, adds basic heterogeneous storage management via OST Pools, adaptive network timeouts, and version-based recovery. It also serves as a transition release, being interoperable with both Lustre 1.6 and Lustre 2.0.
Lustre 2.0.0, released in August 2010, provided a rewritten metadata server stack to provide a basis for Clustered Metadata (CMD) to allow distribution of the Lustre metadata across multiple metadata servers, a new Client IO stack (CLIO) for portability to other client operating systems such as Mac OS
Mac OS
Mac OS is a series of graphical user interface-based operating systems developed by Apple Inc. for their Macintosh line of computer systems. The Macintosh user experience is credited with popularizing the graphical user interface...
, Windows, and Solaris, and an abstracted Object Storage Device (OSD) back-end for portability to other filesystems such as ZFS
ZFS
In computing, ZFS is a combined file system and logical volume manager designed by Sun Microsystems. The features of ZFS include data integrity verification against data corruption modes , support for high storage capacities, integration of the concepts of filesystem and volume management,...
. Lustre 2.x clients cannot interoperate with 1.8 or earlier servers. However, Lustre 1.8.6 and later clients can interoperate with Lustre 2.0 servers. The Lustre 2.x on-disk format is compatible with Lustre 1.8 to allow upgrades of existing filesystems to Lustre 2.
The Lustre file system and associated open source software has been adopted by many partners. Both Red Hat
Red Hat
Red Hat, Inc. is an S&P 500 company in the free and open source software sector, and a major Linux distribution vendor. Founded in 1993, Red Hat has its corporate headquarters in Raleigh, North Carolina with satellite offices worldwide....
and SUSE
SUSE Linux distributions
SUSE Linux is a computer operating system. It is built on top of the open source Linux kernel and is distributed with system and application software from other open source projects. SUSE Linux is of German origin and mainly developed in Europe. The first version appeared in early 1994, making...
(Novell
Novell
Novell, Inc. is a multinational software and services company. It is a wholly owned subsidiary of The Attachmate Group. It specializes in network operating systems, such as Novell NetWare; systems management solutions, such as Novell ZENworks; and collaboration solutions, such as Novell Groupwise...
) offer Linux kernels that work without patches on the client for easy deployment.
Derivative Software
Lustre 2.1.0, released in September 2011, was a community-wide initiative in response to Oracle suspending development on Lustre 2.x releases. It adds RHEL6 server support and increases the maximum ext4-based OST size from 24TB to 128TB, as well as a number of performance and stability improvements. Lustre 2.1 servers remain interoperable with 1.8.6 and later clients.Architecture
A Lustre file system has three major functional units:- A single metadata server (MDS) that has a single metadataMetadataThe term metadata is an ambiguous term which is used for two fundamentally different concepts . Although the expression "data about data" is often used, it does not apply to both in the same way. Structural metadata, the design and specification of data structures, cannot be about data, because at...
target (MDT) per Lustre filesystem that stores namespace metadata, such as filenames, directories, access permissions, and file layout. The MDT data is stored in a single local disk filesystem.
- One or more object storage servers (OSSes) that store file data on one or more object storage targets (OSTs). Depending on the server’s hardware, an OSS typically serves between two and eight OSTs, with each OST managing a single local disk filesystem. The capacity of a Lustre file system is the sum of the capacities provided by the OSTs.
- Client(s) that access and use the data. Lustre presents all clients with a unified namespace for all of the files and data in the filesystem, using standard POSIXPOSIXPOSIX , an acronym for "Portable Operating System Interface", is a family of standards specified by the IEEE for maintaining compatibility between operating systems...
semantics, and allow concurrent and coherent read and write access to the files in the filesystem.
The MDT, OST, and client can be on the same node, but in typical installations these functions are on separate nodes communicating over a network. The Lustre Network (LNET) layer supports several network interconnects, including native Infiniband
InfiniBand
InfiniBand is a switched fabric communications link used in high-performance computing and enterprise data centers. Its features include high throughput, low latency, quality of service and failover, and it is designed to be scalable...
verbs, TCP/IP on Ethernet
Ethernet
Ethernet is a family of computer networking technologies for local area networks commercially introduced in 1980. Standardized in IEEE 802.3, Ethernet has largely replaced competing wired LAN technologies....
and other networks, Myrinet
Myrinet
Myrinet, ANSI/VITA 26-1998, is a high-speed local area networking system designed by Myricom to be used as an interconnect between multiple machines to form computer clusters. Myrinet has much lower protocol overhead than standards such as Ethernet, and therefore provides better throughput, less...
, Quadrics
Quadrics
Quadrics was a supercomputer company formed in 1996 as a joint venture between Alenia Spazio and the technical team from Meiko Scientific. They produced hardware and software for clustering commodity computer systems into massively parallel systems. Their highpoint was in June 2003 when six out of...
, and other proprietary network technologies. Lustre will take advantage of remote direct memory access (RDMA) transfers, when available, to improve throughput and reduce CPU usage.
The storage used for the MDT and OST backing filesystems is partitioned, optionally organized with logical volume management
Logical volume management
In computer storage, logical volume management or LVM provides a method of allocating space on mass-storage devices that is more flexible than conventional partitioning schemes...
(LVM) and/or RAID
RAID
RAID is a storage technology that combines multiple disk drive components into a logical unit...
, and normally formatted as ext4
Ext4
The ext4 or fourth extended filesystem is a journaling file system for Linux, developed as the successor to ext3.It was born as a series of backward compatible extensions to ext3, many of them originally developed by Cluster File Systems for the Lustre file system between 2003 and 2006, meant to...
file systems. The Lustre OSS and MDS servers read, write, and modify data in the format imposed by these file systems.
An OST is a dedicated filesystem that exports an interface to byte ranges of objects for read/write operations. An MDT is a dedicated filesystem that controls file access and tells clients which object(s) make up a file. MDTs and OSTs currently use an enhanced version of ext4
Ext4
The ext4 or fourth extended filesystem is a journaling file system for Linux, developed as the successor to ext3.It was born as a series of backward compatible extensions to ext3, many of them originally developed by Cluster File Systems for the Lustre file system between 2003 and 2006, meant to...
called ldiskfs to store data. Work started in 2008 at Sun to port Lustre to Sun's ZFS
ZFS
In computing, ZFS is a combined file system and logical volume manager designed by Sun Microsystems. The features of ZFS include data integrity verification against data corruption modes , support for high storage capacities, integration of the concepts of filesystem and volume management,...
/DMU for back-end data storage and continues as an open source project.
When a client accesses a file, it completes a filename lookup on the MDS. As a result, a file is created on behalf of the client or the layout of an existing file is returned to the client. For read or write operations, the client then interprets the layout in the logical object volume (LOV) layer, which maps the offset and size to one or more objects, each residing on a separate OST. The client then locks the file range being operated on and executes one or more parallel read or write operations directly to the OSTs. With this approach, bottlenecks for client-to-OST communications are eliminated, so the total bandwidth available for the clients to read and write data scales almost linearly with the number of OSTs in the filesystem.
Clients do not directly modify the objects on the OST filesystems, but, instead, delegate this task to OSSes. This approach ensures scalability for large-scale clusters and supercomputers, as well as improved security and reliability. In contrast, shared block-based filesystems such as Global File System
Global File System
In computing, the Global File System is a shared disk file system for Linux computer clusters. This is not to be confused with the Google File System, a proprietary distributed filesystem developed by Google....
and OCFS
OCFS
OCFS is a shared disk file system developed by Oracle Corporation and released under the GNU General Public License....
must allow direct access to the underlying storage by all of the clients in the filesystem and increase the risk of filesystem corruption from misbehaving/defective clients.
Implementation
In a typical Lustre installation on a Linux client, a Lustre filesystem driver module is loaded into the kernel and the filesystem is mounted like any other local or network filesystem. Client applications see a single, unified filesystem even though it may be composed of tens to thousands of individual servers and MDT/OST filesystems.On some massively parallel processor (MPP) installations, computational processors can access a Lustre file system by redirecting their I/O requests to a dedicated I/O node configured as a Lustre client. This approach is used in the Blue Gene
Blue Gene
Blue Gene is a computer architecture project to produce several supercomputers, designed to reach operating speeds in the PFLOPS range, and currently reaching sustained speeds of nearly 500 TFLOPS . It is a cooperative project among IBM Blue Gene is a computer architecture project to produce...
installation at LLNL.
Another approach used in the past is the liblustre library, which provided userspace applications with direct filesystem access. Liblustre was a user-level library that allows computational processors to mount and use the Lustre file system as a client. Using liblustre, the computational processors could access a Lustre file system even if the service node on which the job was launched is not a Lustre client. Liblustre allowed data movement directly between application space and the Lustre OSSs without requiring an intervening data copy through the kernel, thus providing low latency, high bandwidth access from computational processors to the Lustre file system directly.
Data objects and file striping
In a traditional UNIX disk file system, an inode data structure contains basic information about each file, such as where the data contained in the file is stored. The Lustre file system also uses inodes, but inodes on MDTs point to one or more OST objects associated with the file rather than to data blocks. These objects are implemented as files on the OSTs. When a client opens a file, the file open operation transfers a set of object pointers and their layout from the MDS to the client, so that the client can directly interact with the OSS node where the object is stored, allowing the client to perform I/O on the file without further communication with the MDS.If only one OST object is associated with an MDT inode, that object contains all the data in the Lustre file. When more than one object is associated with a file, data in the file is “striped” across the objects similar to RAID 0. Striping a file over multiple objects provides significant performance benefits. When striping is used, the maximum file size is not limited by the size of a single target. Capacity and aggregate I/O bandwidth scale with the number of OSTs a file is striped over. Also, since the locking of each object is managed independently for each OST, adding more stripes (OSTs) scales the file IO locking capability of the filesystem proportionately. Each file in the filesystem can have a different striping layout, so that performance and capacity can be tuned optimally for each file.
Locking
Lustre has a distributed lock managerDistributed lock manager
A distributed lock manager provides distributed software applications with a means to synchronize their accesses to shared resources....
in the style of the VMS style to protect the integrity of each file's data and metadata. Access and modification of a Lustre file is completely cache coherent
Cache coherence
In computing, cache coherence refers to the consistency of data stored in local caches of a shared resource.When clients in a system maintain caches of a common memory resource, problems may arise with inconsistent data. This is particularly true of CPUs in a multiprocessing system...
among all of the clients. Metadata locks are managed by the MDT that stores the inode
Inode
In computing, an inode is a data structure on a traditional Unix-style file system such as UFS. An inode stores all the information about a regular file, directory, or other file system object, except its data and name....
for the file, using the 128-bit Lustre File Identifier (FID, composed of the Sequence number and Object ID) as the resource name. The metadata locks are split into multiple bits that protect the lookup of the file (file owner and group, permission and mode, and access control list
Access control list
An access control list , with respect to a computer file system, is a list of permissions attached to an object. An ACL specifies which users or system processes are granted access to objects, as well as what operations are allowed on given objects. Each entry in a typical ACL specifies a subject...
(ACL)), the state of the inode (directory size, directory contents, link count, timestamps), and layout (file striping). A client can fetch multiple metadata lock bits for a single inode with a single RPC request, but currently they are only ever granted a read lock for the inode. The MDS manages all modifications to the inode in order to avoid lock resource contention
Resource contention
In computer science, resource contention is a conflict over access to a shared resource such as random access memory, disk storage, cache memory, internal busses or external network devices....
and is currently the only node that gets write locks on inodes.
File data locks are managed by the OST on which each object of the file is striped, using byte-range extent locks. Clients can be granted both overlapping read extent locks for part or all of the file, allowing multiple concurrent readers of the same file, and/or non-overlapping write extent locks for regions of the file. This allows many Lustre clients to access a single file concurrently for both read and write, avoiding bottlenecks during file IO. In practice, because Linux clients manage their data cache in units of pages, the clients will request locks that are always an integer multiple of the page size (4096 bytes on most clients). When a client is requesting an extent lock the OST may grant a lock for a larger extent than requested, in order to reduce the number of lock requests that the client makes. The actual size of the granted lock depends on several factors, including the number of currently-granted locks, whether there are conflicting write locks, and the number of outstanding lock requests. The granted lock is never smaller than the originally-requested extent. OST extent locks use the Lustre FID as the resource name for the lock. Since the number of extent lock servers scales with the number of OSTs in the filesystem, this also scales the aggregate locking performance of the filesystem, and of a single file if it is striped over multiple OSTs.
Networking
In a cluster with a Lustre file system, the system network connecting the servers and the clients is implemented using Lustre Networking (LNET), which provides the communication infrastructure required by the Lustre file system. Disk storage is connected to the Lustre file system MDSs and OSSs using traditional storage area networkStorage area network
A storage area network is a dedicated network that provides access to consolidated, block level data storage. SANs are primarily used to make storage devices, such as disk arrays, tape libraries, and optical jukeboxes, accessible to servers so that the devices appear like locally attached devices...
(SAN) technologies.
LNET supports many commonly-used network types, such as InfiniBand and IP networks, and allows simultaneous availability across multiple network types with routing between them. Remote Direct Memory Access
Remote Direct Memory Access
In computing, remote direct memory access is a direct memory access from the memory of one computer into that of another without involving either one's operating system...
(RDMA) is permitted when supported by underlying networks such as Quadrics Elan, Myrinet, and InfiniBand. High availability and recovery features enable transparent recovery in conjunction with failover servers.
LNET provides end-to-end throughput over Gigabit Ethernet
Gigabit Ethernet
Gigabit Ethernet is a term describing various technologies for transmitting Ethernet frames at a rate of a gigabit per second , as defined by the IEEE 802.3-2008 standard. It came into use beginning in 1999, gradually supplanting Fast Ethernet in wired local networks where it performed...
(GigE) networks in excess of 100 MB/s, throughput up to 3 GB/s using InfiniBand quad data rate (QDR) links, and throughput over 1 GB/s across 10GigE interfaces.
High availability
Lustre file system high availability features include a robust failover and recovery mechanism, making server failures and reboots transparent. Version interoperability between successive minor versions of the Lustre software enables a server to be upgraded by taking it offline (or failing it over to a standby server), performing the upgrade, and restarting it, while all active jobs continue to run, merely experiencing a delay while the backup server takes over the storage.Lustre MDSes are configured as an active/passive pair, while OSSes are typically deployed in an active/active configuration that provides redundancy without extra overhead. Often the standby MDS is the active MDS for another Lustre file system, so no nodes are idle in the cluster.
Deployments
Lustre is used by many of the TOP500 supercomputers and multi-site clusters. Fifteen of the top 30 supercomputers in the world use Lustre file systems, including the world's fastest supercomputer, K computerK computer
The K computer – named for the Japanese word , which stands for 10 quadrillion – is a supercomputer being produced by Fujitsu at the RIKEN Advanced Institute for Computational Science campus in Kobe, Japan. In June 2011, TOP500 ranked K the world's fastest supercomputer, with a rating...
at the RIKEN Advanced Institute for Computational Science
RIKEN
is a large natural sciences research institute in Japan. Founded in 1917, it now has approximately 3000 scientists on seven campuses across Japan, the main one in Wako, just outside Tokyo...
, and the second fastest (fastest from Nov 2010 to June 2011), the Tianhe-1A at the National Supercomputing Center in Tianjin, China. Other supercomputers that use the Lustre file system include the third fastest Jaguar supercomputer
Jaguar (computer)
Jaguar is a petascale supercomputer built by Cray at Oak Ridge National Laboratory in Oak Ridge, Tennessee. The massively parallel Jaguar has a peak performance of just over 1,750 teraflops . It has 224,256 x86-based AMD Opteron processor cores, and operates with a version of Linux called the...
at Oak Ridge National Laboratory
Oak Ridge National Laboratory
Oak Ridge National Laboratory is a multiprogram science and technology national laboratory managed for the United States Department of Energy by UT-Battelle. ORNL is the DOE's largest science and energy laboratory. ORNL is located in Oak Ridge, Tennessee, near Knoxville...
(ORNL) and systems at the National Energy Research Scientific Computing Center
National Energy Research Scientific Computing Center
The ', or NERSC for short, is a designated user facility operated by Lawrence Berkeley National Laboratory and the Department of Energy. It contains several cluster supercomputers, the largest of which is...
located at Lawrence Berkeley National Laboratory
Lawrence Berkeley National Laboratory
The Lawrence Berkeley National Laboratory , is a U.S. Department of Energy national laboratory conducting unclassified scientific research. It is located on the grounds of the University of California, Berkeley, in the Berkeley Hills above the central campus...
(LBNL), Lawrence Livermore National Laboratory
Lawrence Livermore National Laboratory
The Lawrence Livermore National Laboratory , just outside Livermore, California, is a Federally Funded Research and Development Center founded by the University of California in 1952...
(LLNL), Pacific Northwest National Laboratory
Pacific Northwest National Laboratory
Pacific Northwest National Laboratory is one of the United States Department of Energy National Laboratories, managed by the Department of Energy's Office of Science. The main campus of the laboratory is in Richland, Washington....
, Texas Advanced Computing Center
Texas Advanced Computing Center
The Texas Advanced Computing Center at the University of Texas at Austin, United States, is a research center for advanced computational science, engineering and technology. TACC is located on UT's J.J. Pickle Research Campus....
and NASA
NASA
The National Aeronautics and Space Administration is the agency of the United States government that is responsible for the nation's civilian space program and for aeronautics and aerospace research...
in North America, in Asia at Tokyo Institute of Technology
Tokyo Institute of Technology
The Tokyo Institute of Technology is a public research university located in Greater Tokyo Area, Japan. Tokyo Tech is the largest institution for higher education in Japan dedicated to science and technology. Tokyo Tech enrolled 4,850 undergaraduates and 5006 graduate students for 2009-2010...
, and one of the largest systems in Europe at CEA
Commissariat à l'Énergie Atomique
The Commissariat à l'énergie atomique et aux énergies alternatives or CEA, is a French “public establishment related to industrial and commercial activities” whose mission is to develop all applications of nuclear power, both civilian and military...
.
Commercial support
Commercial support for Lustre is available from a wide array of vendors. In most cases, this support is bundled along with the computing system and/or storage hardware sold by the vendor, who in turn get 3rd-level Lustre support from the Lustre developers (originally Cluster File SystemsCluster File Systems
Cluster File Systems, Inc. is the company that originally developed the Lustre distributed file system. CFS was a privately held company with offices in the United States, and China.CFS was founded in 2001 by Dr. Peter Braam...
, the original developers of Lustre, and more recently Whamcloud, founded by many of the original Lustre developers). A non-exhaustive list of vendors selling bundled computing and Lustre storage systems include Cray
Cray
Cray Inc. is an American supercomputer manufacturer based in Seattle, Washington. The company's predecessor, Cray Research, Inc. , was founded in 1972 by computer designer Seymour Cray. Seymour Cray went on to form the spin-off Cray Computer Corporation , in 1989, which went bankrupt in 1995,...
, Dell
Dell
Dell, Inc. is an American multinational information technology corporation based in 1 Dell Way, Round Rock, Texas, United States, that develops, sells and supports computers and related products and services. Bearing the name of its founder, Michael Dell, the company is one of the largest...
, Hewlett-Packard
Hewlett-Packard
Hewlett-Packard Company or HP is an American multinational information technology corporation headquartered in Palo Alto, California, USA that provides products, technologies, softwares, solutions and services to consumers, small- and medium-sized businesses and large enterprises, including...
, BULL
Groupe Bull
-External links:* * — Friends, co-workers and former employees of Bull and Honeywell* *...
, SGI
Silicon Graphics International
Silicon Graphics International Corp. , is an American manufacturer of computer hardware and software, including high-performance computing solutions, x86-based servers for datacenter deployment, and visualization products...
, and others. Sun Microsystems no longer sells either systems or storage that include Lustre. Major vendors selling storage hardware with bundled Lustre support include Data Direct Networks (DDN), Dell
Dell
Dell, Inc. is an American multinational information technology corporation based in 1 Dell Way, Round Rock, Texas, United States, that develops, sells and supports computers and related products and services. Bearing the name of its founder, Michael Dell, the company is one of the largest...
, Terascala, Xyratex
Xyratex
Xyratex is a provider of disk drive manufacturing equipment and data storage subsystems with headquarters of Xyratex International in Havant, United Kingdom and U.S. headquarters in Fremont, CA. Xyratex has two business divisions serving distinct storage markets. They are the Networked Storage...
, and many others.
See also
- Distributed file systemDistributed file systemNetwork file system may refer to:* A distributed file system, which is accessed over a computer network* Network File System , a specific brand of distributed file system...
- List of file systems, the distributed parallel fault-tolerant file system section