Standard Performance Evaluation Corporation
Encyclopedia
The Standard Performance Evaluation Corporation (SPEC) is a non-profit organization
that aims to "produce, establish, maintain and endorse a standardized set" of performance benchmark
s for computers.
SPEC was founded in 1988. SPEC benchmarks are widely used to evaluate the performance of computer systems; the test results are published on the SPEC website. Results are sometimes informally referred to as "SPECmarks" or just "SPEC".
The list of members is available on SPEC's membership page.
The benchmarks aim to test "real-life" situations. There are several benchmarks testing Java scenarios, from simple computation (SPECjbb) to a full system with Java EE, database, disk, and network (SPECjEnterprise). The SPECweb benchmarks test web server
performance by performing various types of parallel HTTP requests.
The SPEC CPU suites test CPU
performance by measuring the run time of several programs such as the compiler
gcc
, the chemistry program gamess
, and the weather program WRF
. The various tasks are equally weighted; no attempt is made to weight them based on their perceived importance. An overall score is based on a geometric mean.
(usually C
, Java or Fortran
), and the interested parties may compile
the code using whatever compiler they prefer for their platform, but may not change the code. Manufacturers have been known to optimize their compilers to improve performance of the various SPEC benchmarks. SPEC has rules that attempt to limit such optimizations.
as the benchmarks include software such as GCC
that is licensed by the GPL. However, the GPL does not require software to be distributed for free, only that recipients be allowed to redistribute any GPLed software that they receive; the license agreement for SPEC specifically exempts items that are under "licenses that require free distribution", and the files themselves are placed in a separate part of the overall software package.
Non-profit organization
Nonprofit organization is neither a legal nor technical definition but generally refers to an organization that uses surplus revenues to achieve its goals, rather than distributing them as profit or dividends...
that aims to "produce, establish, maintain and endorse a standardized set" of performance benchmark
Benchmark (computing)
In computing, a benchmark is the act of running a computer program, a set of programs, or other operations, in order to assess the relative performance of an object, normally by running a number of standard tests and trials against it...
s for computers.
SPEC was founded in 1988. SPEC benchmarks are widely used to evaluate the performance of computer systems; the test results are published on the SPEC website. Results are sometimes informally referred to as "SPECmarks" or just "SPEC".
Membership
Membership allows:- Participation in benchmark development
- Participation in review of results
- Benchmark licenses
The list of members is available on SPEC's membership page.
- Sustaining Membership requires dues payment and typically includes hardware or software companies (e.g. Dell, HP, IBM, Oracle, Red Hat).
- SPEC "Associates" pay a reduced fee and typically include Universities.
- SPEC "Supporting Contributors" are invited to participate in development of a single benchmark, and do not pay fees.
The benchmarks aim to test "real-life" situations. There are several benchmarks testing Java scenarios, from simple computation (SPECjbb) to a full system with Java EE, database, disk, and network (SPECjEnterprise). The SPECweb benchmarks test web server
Web server
Web server can refer to either the hardware or the software that helps to deliver content that can be accessed through the Internet....
performance by performing various types of parallel HTTP requests.
The SPEC CPU suites test CPU
Central processing unit
The central processing unit is the portion of a computer system that carries out the instructions of a computer program, to perform the basic arithmetical, logical, and input/output operations of the system. The CPU plays a role somewhat analogous to the brain in the computer. The term has been in...
performance by measuring the run time of several programs such as the compiler
Compiler
A compiler is a computer program that transforms source code written in a programming language into another computer language...
gcc
GNU Compiler Collection
The GNU Compiler Collection is a compiler system produced by the GNU Project supporting various programming languages. GCC is a key component of the GNU toolchain...
, the chemistry program gamess
GAMESS (US)
GAMESS is a computational chemistry software program that stands for General Atomic and Molecular Electronic Structure System. The original code started on October 1, 1977 as a National Resources for Computations in Chemistry project. In 1981, the code base split into GAMESS and GAMESS variants,...
, and the weather program WRF
Weather Research and Forecasting model
The Weather Research and Forecasting model, , is a specific computer program with a dual use for forecasting and research. It was created through a partnership that includes the National Oceanic and Atmospheric Administration , the National Center for Atmospheric Research , and more than 150 other...
. The various tasks are equally weighted; no attempt is made to weight them based on their perceived importance. An overall score is based on a geometric mean.
Portability
SPEC benchmarks are written in a platform neutral programming languageProgramming language
A programming language is an artificial language designed to communicate instructions to a machine, particularly a computer. Programming languages can be used to create programs that control the behavior of a machine and/or to express algorithms precisely....
(usually C
C (programming language)
C is a general-purpose computer programming language developed between 1969 and 1973 by Dennis Ritchie at the Bell Telephone Laboratories for use with the Unix operating system....
, Java or Fortran
Fortran
Fortran is a general-purpose, procedural, imperative programming language that is especially suited to numeric computation and scientific computing...
), and the interested parties may compile
Compiler
A compiler is a computer program that transforms source code written in a programming language into another computer language...
the code using whatever compiler they prefer for their platform, but may not change the code. Manufacturers have been known to optimize their compilers to improve performance of the various SPEC benchmarks. SPEC has rules that attempt to limit such optimizations.
Licensing
In order to use a benchmark, a license has to be purchased from SPEC; the costs vary from test to test with a typical range from several hundred to several thousand dollars. This pay-for-license model might seem to be in violation of the GPLGNU General Public License
The GNU General Public License is the most widely used free software license, originally written by Richard Stallman for the GNU Project....
as the benchmarks include software such as GCC
GNU Compiler Collection
The GNU Compiler Collection is a compiler system produced by the GNU Project supporting various programming languages. GCC is a key component of the GNU toolchain...
that is licensed by the GPL. However, the GPL does not require software to be distributed for free, only that recipients be allowed to redistribute any GPLed software that they receive; the license agreement for SPEC specifically exempts items that are under "licenses that require free distribution", and the files themselves are placed in a separate part of the overall software package.
Current
- SPEC CPU2006, combined performance of CPU, memoryComputer storageComputer data storage, often called storage or memory, refers to computer components and recording media that retain digital data. Data storage is one of the core functions and fundamental components of computers....
and compiler- CINT2006 ("SPECintSPECintSPECint is a computer benchmark specification for CPU's integer processing power. It is maintained by the Standard Performance Evaluation Corporation . SPECint is the integer performance testing component of the SPEC test suite. The first SPEC test suite, CPU92, was announced in 1992. It was...
"), testing integerInteger (computer science)In computer science, an integer is a datum of integral data type, a data type which represents some finite subset of the mathematical integers. Integral data types may be of different sizes and may or may not be allowed to contain negative values....
arithmetic, with programs such as compilers, interpreters, word processors, chess programs etc. - CFP2006 ("SPECfpSPECfpSPECfp is a computer benchmark designed to test the floating point performance of a computer. It is managed by the Standard Performance Evaluation Corporation. SPECfp is the floating point performance testing component of the SPEC CPU testing suit. The first stander SPECfp was released in 1989 as...
"), testing floating pointFloating pointIn computing, floating point describes a method of representing real numbers in a way that can support a wide range of values. Numbers are, in general, represented approximately to a fixed number of significant digits and scaled using an exponent. The base for the scaling is normally 2, 10 or 16...
performance, with physical simulations, 3D graphics, image processing, computational chemistry etc.
- CINT2006 ("SPECint
- SPECjms2007, Java Message ServiceJava Message ServiceThe Java Message Service API is a Java Message Oriented Middleware API for sending messages between two or more clients. JMS is a part of the Java Platform, Enterprise Edition, and is defined by a specification developed under the Java Community Process as JSR 914...
performance - SPECweb2005, PHP and/or JSP performance.
- SPECviewperf, performance of an OpenGLOpenGLOpenGL is a standard specification defining a cross-language, cross-platform API for writing applications that produce 2D and 3D computer graphics. The interface consists of over 250 different function calls which can be used to draw complex three-dimensional scenes from simple primitives. OpenGL...
3D graphics system, tested with various rendering tasks from real applications - SPECapc, performance of several 3D-intensive popular applications on a given system
- SPEC OMP2001 V3.2, for evaluating performance of parallel systems using OpenMP (http://www.openmp.org) applications.
- SPEC MPI2007, for evaluating performance of parallel systems using MPI (Message Passing InterfaceMessage Passing InterfaceMessage Passing Interface is a standardized and portable message-passing system designed by a group of researchers from academia and industry to function on a wide variety of parallel computers...
) applications. - SPECjvm2008, measuring basic Java performance of a Java Runtime Environment on a wide variety of both client and server systems.
- SPECjEnterprise2010, a multi-tier benchmark for measuring the performance of Java 2 Enterprise Edition (J2EE) technology-based application servers.
- SPECjbb2005, evaluates the performance of server side Java by emulating a three-tier client/server system (with emphasis on the middle tier).
- SPEC MAIL2001, performance of a mail server, testing SMTP and POPPost Office ProtocolIn computing, the Post Office Protocol is an application-layer Internet standard protocol used by local e-mail clients to retrieve e-mail from a remote server over a TCP/IP connection. POP and IMAP are the two most prevalent Internet standard protocols for e-mail retrieval. Virtually all modern...
protocols - SPECpower ssj2008, evaluates the energy efficiency of server systems.
- SPECsfs2008, File server throughput and response time supporting both NFS and CIFS protocol access
- SPECsip_Infrastructure2011, SIPSession Initiation ProtocolThe Session Initiation Protocol is an IETF-defined signaling protocol widely used for controlling communication sessions such as voice and video calls over Internet Protocol . The protocol can be used for creating, modifying and terminating two-party or multiparty sessions...
server performance - SPECvirt_sc2010 ("SPECvirtSPECvirtSPECvirt_sc2010 is a computer benchmark that evaluates the performance of a server computer for virtualization. It is available from the Standard Performance Evaluation Corporation . It was introduced in July, 2010....
"), evaluates the performance of datacenter servers used in virtualized server consolidation environments
Future
- SOA: according to SPEC's web site in late 2010, a subcommittee is investigating benchmarks for Service Oriented Architecture (SOA).
Retired
- SPEC CPU2000
- SPEC CPU95
- SPEC CPU92
- SPEC HPC96
- SPEC HPC2002 (no longer available)
- SPECjAppServer2001
- SPECjAppServer2002
- SPECjAppServer2004
- SPECjbb2000
- SPEC SDM91
- SPEC SFS97_R1
- SPECweb96
- SPECweb99
- SPECweb99_SSL
- SPEC JVM98