Cray XT3
Encyclopedia
Massively parallel
Massively parallel is a description which appears in computer science, life sciences, medical diagnostics, and other fields.A massively parallel computer is a distributed memory computer system which consists of many individual nodes, each of which is essentially an independent computer in itself,...
MIMD
MIMD
In computing, MIMD is a technique employed to achieve parallelism. Machines using MIMD have a number of processors that function asynchronously and independently. At any time, different processors may be executing different instructions on different pieces of data...
supercomputer
Supercomputer
A supercomputer is a computer at the frontline of current processing capacity, particularly speed of calculation.Supercomputers are used for highly calculation-intensive tasks such as problems including quantum physics, weather forecasting, climate research, molecular modeling A supercomputer is a...
designed by Cray Inc.
Cray
Cray Inc. is an American supercomputer manufacturer based in Seattle, Washington. The company's predecessor, Cray Research, Inc. , was founded in 1972 by computer designer Seymour Cray. Seymour Cray went on to form the spin-off Cray Computer Corporation , in 1989, which went bankrupt in 1995,...
with Sandia National Laboratories
Sandia National Laboratories
The Sandia National Laboratories, managed and operated by the Sandia Corporation , are two major United States Department of Energy research and development national laboratories....
under the codename Red Storm. Cray turned the design into a commercial product in 2004. The XT3 derives much of its architecture from the previous Cray T3E
Cray T3E
The Cray T3E was Cray Research's second-generation massively parallel supercomputer architecture, launched in late November 1995. The first T3E was installed at the Pittsburgh Supercomputing Center in 1996. Like the previous Cray T3D, it was a fully distributed memory machine using a 3D torus...
system, and also from the Intel ASCI Red
ASCI Red
ASCI Red was the first computer built under the Advanced Strategic Computing Initiative . ASCI Red was built by Intel and installed at Sandia in late 1996. The design was based on the Intel Paragon computer...
supercomputer.
The XT3 consists of between 192 and 32,768 processing elements (PEs), where each PE comprises a 2.4 or 2.6 GHz AMD Opteron
Opteron
Opteron is AMD's x86 server and workstation processor line, and was the first processor which supported the AMD64 instruction set architecture . It was released on April 22, 2003 with the SledgeHammer core and was intended to compete in the server and workstation markets, particularly in the same...
processor with up to two cores, a custom "SeaStar" communications chip, and between 1 and 8 GB of RAM
Random-access memory
Random access memory is a form of computer data storage. Today, it takes the form of integrated circuits that allow stored data to be accessed in any order with a worst case performance of constant time. Strictly speaking, modern types of DRAM are therefore not random access, as data is read in...
. The PowerPC 440 based SeaStar device provides a 6.4 gigabyte per second connection to the processor across HyperTransport
HyperTransport
HyperTransport , formerly known as Lightning Data Transport , is a technology for interconnection of computer processors. It is a bidirectional serial/parallel high-bandwidth, low-latency point-to-point link that was introduced on April 2, 2001...
, as well as six 8-gigabyte per second links to neighboring PEs. The PEs are arranged in a 3-dimensional torus
Torus
In geometry, a torus is a surface of revolution generated by revolving a circle in three dimensional space about an axis coplanar with the circle...
topology, with 96 PEs in each cabinet.
The XT3 runs an operating system
Operating system
An operating system is a set of programs that manage computer hardware resources and provide common services for application software. The operating system is the most important type of system software in a computer system...
called UNICOS/lc
Unicos
UNICOS is the name of a range of Unix-like operating system variants developed by Cray for its supercomputers. UNICOS is the successor of the Cray Operating System . It provides network clustering and source code compatibility layers for some other Unixes. UNICOS was originally introduced in 1985...
that partitions the machine into three sections, the largest comprising the Compute nodes, and two smaller sections for Service nodes and IO nodes. In UNICOS/lc 1.x, the Compute PEs run a Sandia developed microkernel
Microkernel
In computer science, a microkernel is the near-minimum amount of software that can provide the mechanisms needed to implement an operating system . These mechanisms include low-level address space management, thread management, and inter-process communication...
called Catamount, which is descended from the SUNMOS
SUNMOS
SUNMOS is an operating system jointly developed by Sandia National Laboratories and the Computer Science Department at the University of New Mexico...
OS of the Intel Paragon
Intel Paragon
The Intel Paragon was a series of massively parallel supercomputers produced by Intel. The Paragon XP/S was a productized version of the experimental Touchstone Delta system built at Caltech, launched in 1992. The Paragon superseded Intel's earlier iPSC/860 system, to which it was closely...
; in UNICOS/lc 2.0, Catamount was replaced by a specially tuned version of Linux called Compute Node Linux
Compute Node Linux
Compute Node Linux is a runtime environment based on the Linux kernel for the Cray XT3, Cray XT4, Cray XT5, Cray XT6, and Cray XE6 supercomputer systems based on SUSE Linux Enterprise Server. CNL forms part of the UNICOS/lc operating system from release 2.0 onwards...
(CNL). Service and IO PEs run the full version of SuSE Linux
Linux
Linux is a Unix-like computer operating system assembled under the model of free and open source software development and distribution. The defining component of any Linux system is the Linux kernel, an operating system kernel first released October 5, 1991 by Linus Torvalds...
and are used for interactive logins, systems management, application compiling and job launch. I/O PEs use physically distinct hardware, in that the node boards include PCI-X
PCI-X
PCI-X, short for PCI-eXtended, is a computer bus and expansion card standard that enhances the 32-bit PCI Local Bus for higher bandwidth demanded by servers. It is a double-wide version of PCI, running at up to four times the clock speed, but is otherwise similar in electrical implementation and...
slots for connections to Ethernet
Ethernet
Ethernet is a family of computer networking technologies for local area networks commercially introduced in 1980. Standardized in IEEE 802.3, Ethernet has largely replaced competing wired LAN technologies....
and Fibre Channel
Fibre Channel
Fibre Channel, or FC, is a gigabit-speed network technology primarily used for storage networking. Fibre Channel is standardized in the T11 Technical Committee of the InterNational Committee for Information Technology Standards , an American National Standards Institute –accredited standards...
networks.
Though the performance of each XT3 model will vary with the speed and number of processors installed, the November 2007 Top500
TOP500
The TOP500 project ranks and details the 500 most powerful known computer systems in the world. The project was started in 1993 and publishes an updated list of the supercomputers twice a year...
results for the Red Storm
Red Storm (computing)
Red Storm is a supercomputer architecture designed for the US Department of Energy’s National Nuclear Security Administration Advanced Simulation and Computing Program. Cray, Inc developed it based on the contracted architectural specifications provided by Sandia National Laboratories...
machine, the largest XT3 machine installed at Sandia, measured 102.7 teraflops on the Linpack benchmark, placing it at #6 on the list. After upgrades in 2008 to install some XT4 nodes with quad-core Opterons, Red Storm achieved 248 teraflops to place at #9 on the November 2008 Top500. The architecture was superseded in 2006 by the Cray XT4
Cray XT4
The Cray XT4 is an updated version of the Cray XT3 supercomputer. It was released on November 18, 2006. It includes an updated version of the SeaStar interconnect router called SeaStar2, processor sockets for Socket AM2 Opteron processors, and 240-pin unbuffered DDR2 memory...
.