.gif)
Roadrunner (supercomputer)
Encyclopedia
Roadrunner is a supercomputer
built by IBM
at the Los Alamos National Laboratory
in New Mexico
, USA. Currently the world's tenth fastest computer, the US$133-million Roadrunner is designed for a peak performance of 1.7 petaflops
, achieving 1.026 on May 25, 2008 to become the world's first TOP500
Linpack
sustained 1.0 petaflops system. It is a one-of-a-kind supercomputer, built from off the shelf
parts, with many novel design features.
In November 2008, it reached a top performance of 1.456 petaflops, retaining its top spot in the TOP500
list. It is also the fourth-most energy-efficient supercomputer in the world on the Supermicro Green500 list, with an operational rate of 444.94 megaflops per watt of power used.
built the computer for the U.S. Department of Energy's
(DOE) National Nuclear Security Administration
. It is a hybrid design with 12,960 IBM
PowerXCell 8i
and 6,480 AMD Opteron
dual-core processors in specially designed blade server
s connected by Infiniband
. The Roadrunner uses Red Hat Enterprise Linux
along with Fedora
as its operating systems and is managed with xCAT
distributed computing
software. It also uses the Open MPI
Message Passing Interface
implementation.
Roadrunner occupies approximately 560 square metres (6,027.8 sq ft) and became operational in 2008.
The DOE plans to use the computer for simulating how nuclear materials age in order to predict whether the USA's aging arsenal of nuclear weapons are safe and reliable. Other uses for the Roadrunner include the science, financial, automotive and aerospace industries.
server processors manufactured by AMD
using the standard AMD64 architecture
. Attached to each Opteron core is a PowerXCell 8i processor manufactured by IBM
using Power Architecture
and Cell
technology. As a supercomputer, the Roadrunner is considered an Opteron cluster with Cell accelerators, as each node consists of a Cell attached to an Opteron core and the Opterons to each other.
facility, where it broke the 1 petaflops barrier during its fourth attempt on May 25, 2008. The complete system was moved to its permanent location in New Mexico in the summer of 2008, where fine tuning of the applications will continue until final completion in the latter stages of 2009.
, running at 1.8 GHz. These are processors with two general purpose cores each. Opterons are used both in the computational nodes feeding the Cells with useful data and in the system operations and communication nodes passing data between computing nodes and helping the operators running the system. Roadrunner has a total of 6,912 Opteron processors (6,480 computation, 432 operation), for a total of (12,960+864) 13,824 cores.
, running at 3.2 GHz. These processors have one general purpose core (PPE), and eight special performance cores (SPE) for floating point
operations. Roadrunner has a total of 12,960 PowerXCell processors, with 12,960 PPE cores and 103,680 SPE cores, for a total of 116,640 cores.
list, Roadrunner is said to have 122,400 cores. It is important to know which core is counted.
This is a number larger than the one mentioned on Top500. It turns out that the Roadrunner only used 17 Connected Units while doing the LINPACK benchmark, and it was not counting the cores in the operations and communication nodes (they didn't run the benchmark).
Logically, a TriBlade consists of two dual-core Opterons with 16 GB
RAM
and four PowerXCell 8i CPUs with 16 GB Cell RAM.
Physically, a TriBlade consists of one LS21 Opteron
blade
, an expansion blade, and two QS22 Cell
blades. The LS21 has two 1.8 GHz dual-core Opterons with 16 GB memory for the whole blade, providing 8GB for each CPU. Each QS22 has two PowerXCell 8i CPUs, running at 3.2 GHz and 8 GB memory, which makes 4 GB for each CPU. The expansion blade connects the two QS22 via four PCIe x8 links to the LS21, two links for each QS22. It also provides outside connectivity via an Infiniband
4x DDR adapter. This makes a total width of four slots for a single TriBlade. Three TriBlades fit into one BladeCenter H chassis.
file system
through twelve System x3755
servers..
CU system information:.
The final cluster is made up of 18 connected units, which are connected via eight additional (second-stage) Infiniband ISR2012 switches. Each CU is connected through twelve uplinks for each second-stage switch, which makes a total of 96 uplink connections.
Overall system information:
Supercomputer
A supercomputer is a computer at the frontline of current processing capacity, particularly speed of calculation.Supercomputers are used for highly calculation-intensive tasks such as problems including quantum physics, weather forecasting, climate research, molecular modeling A supercomputer is a...
built by IBM
IBM
International Business Machines Corporation or IBM is an American multinational technology and consulting corporation headquartered in Armonk, New York, United States. IBM manufactures and sells computer hardware and software, and it offers infrastructure, hosting and consulting services in areas...
at the Los Alamos National Laboratory
Los Alamos National Laboratory
Los Alamos National Laboratory is a United States Department of Energy national laboratory, managed and operated by Los Alamos National Security , located in Los Alamos, New Mexico...
in New Mexico
New Mexico
New Mexico is a state located in the southwest and western regions of the United States. New Mexico is also usually considered one of the Mountain States. With a population density of 16 per square mile, New Mexico is the sixth-most sparsely inhabited U.S...
, USA. Currently the world's tenth fastest computer, the US$133-million Roadrunner is designed for a peak performance of 1.7 petaflops
FLOPS
In computing, FLOPS is a measure of a computer's performance, especially in fields of scientific calculations that make heavy use of floating-point calculations, similar to the older, simpler, instructions per second...
, achieving 1.026 on May 25, 2008 to become the world's first TOP500
TOP500
The TOP500 project ranks and details the 500 most powerful known computer systems in the world. The project was started in 1993 and publishes an updated list of the supercomputers twice a year...
Linpack
LINPACK
LINPACK is a software library for performing numerical linear algebra on digital computers. It was written in Fortran by Jack Dongarra, Jim Bunch, Cleve Moler, and Gilbert Stewart, and was intended for use on supercomputers in the 1970s and early 1980s...
sustained 1.0 petaflops system. It is a one-of-a-kind supercomputer, built from off the shelf
Commercial off-the-shelf
In the United States, Commercially available Off-The-Shelf is a Federal Acquisition Regulation term defining a nondevelopmental item of supply that is both commercial and sold in substantial quantities in the commercial marketplace, and that can be procured or utilized under government contract...
parts, with many novel design features.
In November 2008, it reached a top performance of 1.456 petaflops, retaining its top spot in the TOP500
TOP500
The TOP500 project ranks and details the 500 most powerful known computer systems in the world. The project was started in 1993 and publishes an updated list of the supercomputers twice a year...
list. It is also the fourth-most energy-efficient supercomputer in the world on the Supermicro Green500 list, with an operational rate of 444.94 megaflops per watt of power used.
Overview
IBMIBM
International Business Machines Corporation or IBM is an American multinational technology and consulting corporation headquartered in Armonk, New York, United States. IBM manufactures and sells computer hardware and software, and it offers infrastructure, hosting and consulting services in areas...
built the computer for the U.S. Department of Energy's
United States Department of Energy
The United States Department of Energy is a Cabinet-level department of the United States government concerned with the United States' policies regarding energy and safety in handling nuclear material...
(DOE) National Nuclear Security Administration
National Nuclear Security Administration
The United States National Nuclear Security Administration is part of the United States Department of Energy. It works to improve national security through the military application of nuclear energy...
. It is a hybrid design with 12,960 IBM
IBM
International Business Machines Corporation or IBM is an American multinational technology and consulting corporation headquartered in Armonk, New York, United States. IBM manufactures and sells computer hardware and software, and it offers infrastructure, hosting and consulting services in areas...
PowerXCell 8i
Cell microprocessor
Cell is a microprocessor architecture jointly developed by Sony, Sony Computer Entertainment, Toshiba, and IBM, an alliance known as "STI". The architectural design and first implementation were carried out at the STI Design Center in Austin, Texas over a four-year period beginning March 2001 on a...
and 6,480 AMD Opteron
Opteron
Opteron is AMD's x86 server and workstation processor line, and was the first processor which supported the AMD64 instruction set architecture . It was released on April 22, 2003 with the SledgeHammer core and was intended to compete in the server and workstation markets, particularly in the same...
dual-core processors in specially designed blade server
Blade server
A blade server is a stripped down server computer with a modular design optimized to minimize the use of physical space and energy. Whereas a standard rack-mount server can function with a power cord and network cable, blade servers have many components removed to save space, minimize power...
s connected by Infiniband
InfiniBand
InfiniBand is a switched fabric communications link used in high-performance computing and enterprise data centers. Its features include high throughput, low latency, quality of service and failover, and it is designed to be scalable...
. The Roadrunner uses Red Hat Enterprise Linux
Red Hat Enterprise Linux
Red Hat Enterprise Linux is a Linux-based operating system developed by Red Hat and targeted toward the commercial market. Red Hat Enterprise Linux is released in server versions for x86, x86-64, Itanium, PowerPC and IBM System z, and desktop versions for x86 and x86-64...
along with Fedora
Fedora (operating system)
Fedora is a RPM-based, general purpose collection of software, including an operating system based on the Linux kernel, developed by the community-supported Fedora Project and sponsored by Red Hat...
as its operating systems and is managed with xCAT
XCAT
xCAT is open-source distributed computing management software used for the deployment and administration of clusters...
distributed computing
Distributed computing
Distributed computing is a field of computer science that studies distributed systems. A distributed system consists of multiple autonomous computers that communicate through a computer network. The computers interact with each other in order to achieve a common goal...
software. It also uses the Open MPI
Open MPI
Open MPI is a Message Passing Interface library project combining technologies and resources from several other projects . It is used by many TOP500 supercomputers including Roadrunner, which was the world's fastest supercomputer from June 2008 to November 2009, and K computer, the fastest...
Message Passing Interface
Message Passing Interface
Message Passing Interface is a standardized and portable message-passing system designed by a group of researchers from academia and industry to function on a wide variety of parallel computers...
implementation.
Roadrunner occupies approximately 560 square metres (6,027.8 sq ft) and became operational in 2008.
The DOE plans to use the computer for simulating how nuclear materials age in order to predict whether the USA's aging arsenal of nuclear weapons are safe and reliable. Other uses for the Roadrunner include the science, financial, automotive and aerospace industries.
Hybrid design
Roadrunner differs from many contemporary supercomputers in that it is a hybrid system, using two different processor architectures. Usually supercomputers only use one, since such a design is easier to design and program for. To realise the full potential of Roadrunner, all software will have to be written specially for this hybrid architecture. The hybrid design consists of dual-core OpteronOpteron
Opteron is AMD's x86 server and workstation processor line, and was the first processor which supported the AMD64 instruction set architecture . It was released on April 22, 2003 with the SledgeHammer core and was intended to compete in the server and workstation markets, particularly in the same...
server processors manufactured by AMD
Advanced Micro Devices
Advanced Micro Devices, Inc. or AMD is an American multinational semiconductor company based in Sunnyvale, California, that develops computer processors and related technologies for commercial and consumer markets...
using the standard AMD64 architecture
Instruction set
An instruction set, or instruction set architecture , is the part of the computer architecture related to programming, including the native data types, instructions, registers, addressing modes, memory architecture, interrupt and exception handling, and external I/O...
. Attached to each Opteron core is a PowerXCell 8i processor manufactured by IBM
IBM
International Business Machines Corporation or IBM is an American multinational technology and consulting corporation headquartered in Armonk, New York, United States. IBM manufactures and sells computer hardware and software, and it offers infrastructure, hosting and consulting services in areas...
using Power Architecture
Power Architecture
Power Architecture is a broad term to describe similar RISC instruction sets for microprocessors developed and manufactured by such companies as IBM, Freescale, AMCC, Tundra and P.A. Semi...
and Cell
Cell (microprocessor)
Cell is a microprocessor architecture jointly developed by Sony, Sony Computer Entertainment, Toshiba, and IBM, an alliance known as "STI". The architectural design and first implementation were carried out at the STI Design Center in Austin, Texas over a four-year period beginning March 2001 on a...
technology. As a supercomputer, the Roadrunner is considered an Opteron cluster with Cell accelerators, as each node consists of a Cell attached to an Opteron core and the Opterons to each other.
Development
Roadrunner has been in development since 2002, and went online in 2006. Due to its novel design and complexity it was constructed in three phases and became fully operational in 2008. Its predecessor was a machine also developed at Los Alamos named Dark Horse. This machine was one of the earliest hybrid architecture systems originally based on ARM and then moved to the Cell processor. It was entirely a 3D design, Its design integrated 3D memory, networking, processors and a number of other advanced technologies. It was designed by a very small and agile team at Los Alamos.Phase 1
The first phase of the Roadrunner was building a standard (albeit quite large) Opteron based cluster, while evaluating the feasibility to further construct and program the future hybrid version. This Phase 1 Roadrunner reached 71 teraflops and has been in full operation at Los Alamos National Laboratory doing advanced weapons simulations since 2006. Even if Roadrunner had not been greenlit for Phase 2, the Phase 1 form would still be a formidable supercomputer and would have ranked, at its time, in the top 10 list of the world's fastest computers.Phase 2
Phase 2 known as “AAIS” (Advanced Architecture Initial System) included building a small hybrid version of the finished system using an older version of the Cell processor. This phase was used to build prototype applications for the hybrid architecture. It went online in January 2007.Phase 3
The goal of Phase 3 was to reach sustained performance in excess of 1 petaflops. Additional Opteron nodes and new PowerXCell processors were added to the design. These PowerXCell processors are five times as powerful as the Cell processors used in Phase 2. It was built to full scale at IBM’s Poughkeepsie, New YorkPoughkeepsie (city), New York
Poughkeepsie is a city in the state of New York, United States, which serves as the county seat of Dutchess County. Poughkeepsie is located in the Hudson River Valley midway between New York City and Albany...
facility, where it broke the 1 petaflops barrier during its fourth attempt on May 25, 2008. The complete system was moved to its permanent location in New Mexico in the summer of 2008, where fine tuning of the applications will continue until final completion in the latter stages of 2009.
Opteron
AMD Opteron 2210Opteron
Opteron is AMD's x86 server and workstation processor line, and was the first processor which supported the AMD64 instruction set architecture . It was released on April 22, 2003 with the SledgeHammer core and was intended to compete in the server and workstation markets, particularly in the same...
, running at 1.8 GHz. These are processors with two general purpose cores each. Opterons are used both in the computational nodes feeding the Cells with useful data and in the system operations and communication nodes passing data between computing nodes and helping the operators running the system. Roadrunner has a total of 6,912 Opteron processors (6,480 computation, 432 operation), for a total of (12,960+864) 13,824 cores.
PowerXCell
IBM PowerXCell 8iCell (microprocessor)
Cell is a microprocessor architecture jointly developed by Sony, Sony Computer Entertainment, Toshiba, and IBM, an alliance known as "STI". The architectural design and first implementation were carried out at the STI Design Center in Austin, Texas over a four-year period beginning March 2001 on a...
, running at 3.2 GHz. These processors have one general purpose core (PPE), and eight special performance cores (SPE) for floating point
Floating point
In computing, floating point describes a method of representing real numbers in a way that can support a wide range of values. Numbers are, in general, represented approximately to a fixed number of significant digits and scaled using an exponent. The base for the scaling is normally 2, 10 or 16...
operations. Roadrunner has a total of 12,960 PowerXCell processors, with 12,960 PPE cores and 103,680 SPE cores, for a total of 116,640 cores.
Number of cores
On the Top500TOP500
The TOP500 project ranks and details the 500 most powerful known computer systems in the world. The project was started in 1993 and publishes an updated list of the supercomputers twice a year...
list, Roadrunner is said to have 122,400 cores. It is important to know which core is counted.
- 13,824 Opteron cores + 116,640 Cell cores = 130,464 cores for both the computing nodes and the operation nodes.
This is a number larger than the one mentioned on Top500. It turns out that the Roadrunner only used 17 Connected Units while doing the LINPACK benchmark, and it was not counting the cores in the operations and communication nodes (they didn't run the benchmark).
- 6,120 Opteron (2 cores) + 12,240 PowerXCell 8i (9 cores) = 122,400 cores
TriBlade
Gigabyte
The gigabyte is a multiple of the unit byte for digital information storage. The prefix giga means 109 in the International System of Units , therefore 1 gigabyte is...
RAM
Ram
-Animals:*Ram, an uncastrated male sheep*Ram cichlid, a species of freshwater fish endemic to Colombia and Venezuela-Military:*Battering ram*Ramming, a military tactic in which one vehicle runs into another...
and four PowerXCell 8i CPUs with 16 GB Cell RAM.
Physically, a TriBlade consists of one LS21 Opteron
Opteron
Opteron is AMD's x86 server and workstation processor line, and was the first processor which supported the AMD64 instruction set architecture . It was released on April 22, 2003 with the SledgeHammer core and was intended to compete in the server and workstation markets, particularly in the same...
blade
Blade server
A blade server is a stripped down server computer with a modular design optimized to minimize the use of physical space and energy. Whereas a standard rack-mount server can function with a power cord and network cable, blade servers have many components removed to save space, minimize power...
, an expansion blade, and two QS22 Cell
Cell (microprocessor)
Cell is a microprocessor architecture jointly developed by Sony, Sony Computer Entertainment, Toshiba, and IBM, an alliance known as "STI". The architectural design and first implementation were carried out at the STI Design Center in Austin, Texas over a four-year period beginning March 2001 on a...
blades. The LS21 has two 1.8 GHz dual-core Opterons with 16 GB memory for the whole blade, providing 8GB for each CPU. Each QS22 has two PowerXCell 8i CPUs, running at 3.2 GHz and 8 GB memory, which makes 4 GB for each CPU. The expansion blade connects the two QS22 via four PCIe x8 links to the LS21, two links for each QS22. It also provides outside connectivity via an Infiniband
InfiniBand
InfiniBand is a switched fabric communications link used in high-performance computing and enterprise data centers. Its features include high throughput, low latency, quality of service and failover, and it is designed to be scalable...
4x DDR adapter. This makes a total width of four slots for a single TriBlade. Three TriBlades fit into one BladeCenter H chassis.
Connected Unit (CU)
A Connected Unit is 60 BladeCenter H full of TriBlades, that is 180 TriBlades. All TriBlades are connected to a 288-port Voltaire ISR2012 Infiniband switch. Each CU also has access to the PanasasPanasas
Panasas, Inc., is a private, multinational computer storage company based in Sunnyvale, California. It specializes in high-performance scale-out network-attached storage optimized for Linux clusters.- History :...
file system
File system
A file system is a means to organize data expected to be retained after a program terminates by providing procedures to store, retrieve and update data, as well as manage the available space on the device which contain it. A file system organizes data in an efficient manner and is tuned to the...
through twelve System x3755
IBM System x
The IBM System x computers form a sub-brand of International Business Machines System brand servers...
servers..
CU system information:.
- 360 dual-core Opterons with 2.88 TiBTebibyteThe tebibyte is a standards-based binary multiple of the byte, a unit of digital information storage. The tebibyte unit symbol is TiB....
RAMRam-Animals:*Ram, an uncastrated male sheep*Ram cichlid, a species of freshwater fish endemic to Colombia and Venezuela-Military:*Battering ram*Ramming, a military tactic in which one vehicle runs into another...
. - 720 PowerXCell 8i cores with 2.88 TiB RAM.
- 12 System x3755 with dual 10-GBit Ethernet each.
- 288-port Voltaire ISR2012 switch with 192 Infiniband 4x DDR links (180 TriBlades and twelve I/O nodes).
Roadrunner cluster
Overall system information:
- 6,480 Opteron processors with 51.8 TiB RAM (in 3,240 LS21 blades)
- 12,960 Cell processors with 51.8 TiB RAM (in 6,480 QS22 blades)
- 216 System x3755 I/O nodes
- 26 288-port ISR2012 Infiniband 4x DDR switches
- 296 racks
- 2.35 MW power
See also
- Central processing unitCentral processing unitThe central processing unit is the portion of a computer system that carries out the instructions of a computer program, to perform the basic arithmetical, logical, and input/output operations of the system. The CPU plays a role somewhat analogous to the brain in the computer. The term has been in...
- Computer architectureComputer architectureIn computer science and engineering, computer architecture is the practical art of selecting and interconnecting hardware components to create computers that meet functional, performance and cost goals and the formal modelling of those systems....
- Computer scienceComputer scienceComputer science or computing science is the study of the theoretical foundations of information and computation and of practical techniques for their implementation and application in computer systems...
- ComputingComputingComputing is usually defined as the activity of using and improving computer hardware and software. It is the computer-specific part of information technology...
- Multi-core