Codec listening test
Encyclopedia
A codec listening test is a scientific study
designed to compare two or more lossy audio codec
s, usually with respect to perceived fidelity
or compression efficiency.
Most tests take the form of a double-blind
comparison. Commonly used methods are known as "ABX
" or "ABC/HR" or "MUSHRA
". There are various software packages available for individuals to perform this type of testing themselves with minimal assistance.
in a predetermined number of trials, then the null hypothesis
cannot be rejected and it cannot be proved that there is a perceptible difference between samples A and B. This usually indicates that the encoded version will actually be transparent
to the listener.
, since the audio quality produced by any lossy encoder will be trivially improved by increasing the bitrate. If listeners cannot consistently distinguish a lossy encoder's output from the uncompressed original audio, then it may be concluded that the codec has achieved transparency
.
Popular formats compared in these tests include MP3
, AAC
(and extensions), Vorbis
, Musepack
, and WMA
. The RealAudio Gecko
, ATRAC3, QDesign
, and MP3pro
formats appear in some tests, despite much lower adoption . Many encoder and decoder implementations (both proprietary
and open source) exist for some formats, such as MP3
, which is the oldest and best-known format still in widespread use today.
Experiment
An experiment is a methodical procedure carried out with the goal of verifying, falsifying, or establishing the validity of a hypothesis. Experiments vary greatly in their goal and scale, but always rely on repeatable procedure and logical analysis of the results...
designed to compare two or more lossy audio codec
Codec
A codec is a device or computer program capable of encoding or decoding a digital data stream or signal. The word codec is a portmanteau of "compressor-decompressor" or, more commonly, "coder-decoder"...
s, usually with respect to perceived fidelity
Fidelity
"Fidelity" is the quality of being faithful or loyal. Its original meaning regarded duty to a lord or a king, in a broader sense than the related concept of fealty. Both derive from the Latin word fidēlis, meaning "faithful or loyal"....
or compression efficiency.
Most tests take the form of a double-blind
Double-blind
A blind or blinded experiment is a scientific experiment where some of the people involved are prevented from knowing certain information that might lead to conscious or subconscious bias on their part, invalidating the results....
comparison. Commonly used methods are known as "ABX
ABX test
An ABX test is a method of comparing two kinds of sensory stimuli to identify detectable differences. A subject is presented with two known samples , and one unknown sample X, for three samples total. X is randomly selected from A and B, and the subject identifies X as being either A or B...
" or "ABC/HR" or "MUSHRA
MUSHRA
MUSHRA stands for MUltiple Stimuli with Hidden Reference and Anchor and is a methodology for subjective evaluation of audio quality, to evaluate the perceived quality of the output from lossy audio compression algorithms. It is defined by ITU-R recommendation BS.1534-1. The MUSHRA methodology is...
". There are various software packages available for individuals to perform this type of testing themselves with minimal assistance.
ABX test
In an ABX test, the listener has to identify an unknown sample X as being A or B, with A (usually the original) and B (usually the encoded version) available for reference. The outcome of a test must be statistically significant. This setup ensures that the listener is not biased by his/her expectations, and that the outcome is not likely to be the result of chance. If sample X cannot be determined reliably with a low p-valueP-value
In statistical significance testing, the p-value is the probability of obtaining a test statistic at least as extreme as the one that was actually observed, assuming that the null hypothesis is true. One often "rejects the null hypothesis" when the p-value is less than the significance level α ,...
in a predetermined number of trials, then the null hypothesis
Null hypothesis
The practice of science involves formulating and testing hypotheses, assertions that are capable of being proven false using a test of observed data. The null hypothesis typically corresponds to a general or default position...
cannot be rejected and it cannot be proved that there is a perceptible difference between samples A and B. This usually indicates that the encoded version will actually be transparent
Transparency (data compression)
In data compression or psychoacoustics, transparency is the ideal result of lossy data compression. If a lossy compressed result is perceptually indistinguishable from the uncompressed input, then the compression can be declared to be transparent...
to the listener.
ABC/HR test
In an ABC/HR test, C is the original which is always available for reference. A and B are the original and the encoded version in randomized order. The listener must first distinguish the encoded version from the original (which is the Hidden Reference that the "HR" in ABC/HR stands for), prior to assigning a score as a subjective judgment of the quality. Different encoded versions can be compared against each other using these scores.MUSHRA
In MUSHRA (MUltiple Stimuli with Hidden Reference and Anchor), the listener is presented with the reference (labeled as such), a certain number of test samples, a hidden version of the reference and one or more anchors. The purpose of the anchor(s) is to make the scale be closer to an "absolute scale", making sure that minor artifacts are not rated as having very bad quality.Results
Many double-blind music listening tests have been carried out. The following table lists the results of several listening tests that have been published online. To obtain meaningful results, listening tests must compare codecs' performance at similar or identical bitratesBit rate
In telecommunications and computing, bit rate is the number of bits that are conveyed or processed per unit of time....
, since the audio quality produced by any lossy encoder will be trivially improved by increasing the bitrate. If listeners cannot consistently distinguish a lossy encoder's output from the uncompressed original audio, then it may be concluded that the codec has achieved transparency
Transparency (data compression)
In data compression or psychoacoustics, transparency is the ideal result of lossy data compression. If a lossy compressed result is perceptually indistinguishable from the uncompressed input, then the compression can be declared to be transparent...
.
Popular formats compared in these tests include MP3
MP3
MPEG-1 or MPEG-2 Audio Layer III, more commonly referred to as MP3, is a patented digital audio encoding format using a form of lossy data compression...
, AAC
Advanced Audio Coding
Advanced Audio Coding is a standardized, lossy compression and encoding scheme for digital audio. Designed to be the successor of the MP3 format, AAC generally achieves better sound quality than MP3 at similar bit rates....
(and extensions), Vorbis
Vorbis
Vorbis is a free software / open source project headed by the Xiph.Org Foundation . The project produces an audio format specification and software implementation for lossy audio compression...
, Musepack
Musepack
Musepack or MPC is an open source lossy audio codec, specifically optimized for transparent compression of stereo audio at bitrates of 160–180 kbit/s...
, and WMA
Windows Media Audio
Windows Media Audio is an audio data compression technology developed by Microsoft. The name can be used to refer to its audio file format or its audio codecs. It is a proprietary technology that forms part of the Windows Media framework. WMA consists of four distinct codecs...
. The RealAudio Gecko
Cook Codec
The cook codec is a lossy audio compression codec developed by RealNetworks. It is also known as Cooker, Gecko, RealAudio G2, and RealAudio 8 low bitrate ....
, ATRAC3, QDesign
QDesign
QDesign is a music codec, originally known as LBpack, by Dmitry Shmunk. QDesign corporation bought the codec and hired the developer to create what would become the official audio compression in Apple Computer's QuickTime v3....
, and MP3pro
Mp3PRO
mp3PRO is an audio compression algorithm that combines the MP3 audio format with spectral band replication compression methods. It claims to achieve transparency at lower bitrates than MP3, resulting in a file nearly half the size of standard MP3...
formats appear in some tests, despite much lower adoption . Many encoder and decoder implementations (both proprietary
Proprietary software
Proprietary software is computer software licensed under exclusive legal right of the copyright holder. The licensee is given the right to use the software under certain conditions, while restricted from other uses, such as modification, further distribution, or reverse engineering.Complementary...
and open source) exist for some formats, such as MP3
MP3
MPEG-1 or MPEG-2 Audio Layer III, more commonly referred to as MP3, is a patented digital audio encoding format using a form of lossy data compression...
, which is the oldest and best-known format still in widespread use today.
| Source | Dates | Formats | Bitrate (kbit/s) | Codecs | Musical genres | Samples | Listeners | Winner | Comments |
|---|---|---|---|---|---|---|---|---|---|
| ff123 | 2001 | multiple | ~128 |
|
1 | 16 | Musepack Musepack Musepack or MPC is an open source lossy audio codec, specifically optimized for transparent compression of stereo audio at bitrates of 160–180 kbit/s... and AAC Advanced Audio Coding Advanced Audio Coding is a standardized, lossy compression and encoding scheme for digital audio. Designed to be the successor of the MP3 format, AAC generally achieves better sound quality than MP3 at similar bit rates.... |
||
| ff123 | October 2001-January 2002 | multiple | ~128 |
|
Various | 3 | 25-28 | Musepack Musepack Musepack or MPC is an open source lossy audio codec, specifically optimized for transparent compression of stereo audio at bitrates of 160–180 kbit/s... or Vorbis Vorbis Vorbis is a free software / open source project headed by the Xiph.Org Foundation . The project produces an audio format specification and software implementation for lossy audio compression... |
|
| ff123 | July 2002 | multiple | ~64 |
|
Various | 12 | 24-41 | MP3pro Mp3PRO mp3PRO is an audio compression algorithm that combines the MP3 audio format with spectral band replication compression methods. It claims to achieve transparency at lower bitrates than MP3, resulting in a file nearly half the size of standard MP3... |
Both Vorbis variants were a close second. |
| Roberto Amorim | June 2003 | AAC Advanced Audio Coding Advanced Audio Coding is a standardized, lossy compression and encoding scheme for digital audio. Designed to be the successor of the MP3 format, AAC generally achieves better sound quality than MP3 at similar bit rates.... |
128 CBR Constant bitrate Constant bitrate is a term used in telecommunications, relating to the quality of service. Compare with variable bitrate.When referring to codecs, constant bit rate encoding means that the rate at which a codec's output data should be consumed is constant... |
|
Various | 10 | 11-18 | QuickTime QuickTime QuickTime is an extensible proprietary multimedia framework developed by Apple Inc., capable of handling various formats of digital video, picture, sound, panoramic images, and interactivity. The classic version of QuickTime is available for Windows XP and later, as well as Mac OS X Leopard and... |
|
| Roberto Amorim | July 2003 | multiple | ~128 |
|
Various | 12 | 14-24 | Musepack Musepack Musepack or MPC is an open source lossy audio codec, specifically optimized for transparent compression of stereo audio at bitrates of 160–180 kbit/s... |
AAC, WMA Windows Media Audio Windows Media Audio is an audio data compression technology developed by Microsoft. The name can be used to refer to its audio file format or its audio codecs. It is a proprietary technology that forms part of the Windows Media framework. WMA consists of four distinct codecs... , and Vorbis Vorbis Vorbis is a free software / open source project headed by the Xiph.Org Foundation . The project produces an audio format specification and software implementation for lossy audio compression... tied for close second |
| Roberto Amorim | September 2003 | multiple | ~64 |
|
Various | 12 | 30-43 | Nero HE AAC Nero Digital Nero Digital is a brand name applied to a suite of MPEG-4-compatible video and audio compression codecs developed by Nero AG of Germany and Ateme of France... |
This test showed that listeners preferred 128 kbit/s MP3 audio encoded by LAME LAME LAME is a free software codec used to encode/compress audio into the lossy MP3 file format.-History:The name LAME is a recursive acronym for "LAME Ain't an MP3 Encoder". Around mid-1998, Mike Cheng created LAME 1.0 as a set of modifications against the "8Hz-MP3" encoder source code... to all the tested codecs at 64 kbit/s, with greater than 99% confidence: "No codec delivers the marketing plot [sic] of same quality as MP3 at half the bitrates." |
| Roberto Amorim | January 2004 | MP3 MP3 MPEG-1 or MPEG-2 Audio Layer III, more commonly referred to as MP3, is a patented digital audio encoding format using a form of lossy data compression... |
~128 |
|
Various | 12 | 11-22 | LAME LAME LAME is a free software codec used to encode/compress audio into the lossy MP3 file format.-History:The name LAME is a recursive acronym for "LAME Ain't an MP3 Encoder". Around mid-1998, Mike Cheng created LAME 1.0 as a set of modifications against the "8Hz-MP3" encoder source code... |
Serious issues with Xing and iTunes encodings were discovered after the test, and documented on results page. |
| Roberto Amorim | February 2004 | AAC Advanced Audio Coding Advanced Audio Coding is a standardized, lossy compression and encoding scheme for digital audio. Designed to be the successor of the MP3 format, AAC generally achieves better sound quality than MP3 at similar bit rates.... |
~128 |
|
Various | 12 | 19-29 | iTunes ITunes iTunes is a media player computer program, used for playing, downloading, and organizing digital music and video files on desktop computers. It can also manage contents on iPod, iPhone, iPod Touch and iPad.... |
Open-source FAAC FAAC FAAC or Freeware Advanced Audio Coder is a software project which includes the AAC encoder FAAC and decoder FAAD2. It supports MPEG-2 AAC as well as MPEG-4 AAC. It supports several MPEG-4 Audio object types , file formats , multichannel and gapless encoding/decoding and MP4 metadata tags... codec improved greatly since previous test |
| Roberto Amorim | May 2004 | multiple | ~128 |
|
Various | 18 | 12-27 | aoTuV and Musepack Musepack Musepack or MPC is an open source lossy audio codec, specifically optimized for transparent compression of stereo audio at bitrates of 160–180 kbit/s... |
|
| Roberto Amorim | June 2004 | multiple | 32 CBR Constant bitrate Constant bitrate is a term used in telecommunications, relating to the quality of service. Compare with variable bitrate.When referring to codecs, constant bit rate encoding means that the rate at which a codec's output data should be consumed is constant... |
|
Various | 18 | 47-77 | Nero Digital Nero Digital Nero Digital is a brand name applied to a suite of MPEG-4-compatible video and audio compression codecs developed by Nero AG of Germany and Ateme of France... |
|
| HydrogenAudio user "guruboolez" | July 2004 | multiple | ~175 |
|
Classical Classical music Classical music is the art music produced in, or rooted in, the traditions of Western liturgical and secular music, encompassing a broad period from roughly the 11th century to present times... |
18 | 1 | Musepack Musepack Musepack or MPC is an open source lossy audio codec, specifically optimized for transparent compression of stereo audio at bitrates of 160–180 kbit/s... |
|
| HydrogenAudio user "guruboolez" | August 2005 | multiple | ~180 |
|
Classical Classical music Classical music is the art music produced in, or rooted in, the traditions of Western liturgical and secular music, encompassing a broad period from roughly the 11th century to present times... |
18 | 1 | aoTuV | The author reflects on substantial improvements in Vorbis encoding since his previous test (above):
"Vorbis is now –thanks to Aoyumi [creator of aoTuV]– an excellent audio format for 180 kbit/s encodings (and classical music)." |
| gURuBoOleZZ | August 2005 | MP3 | ~96 |
|
Classic Classical music Classical music is the art music produced in, or rooted in, the traditions of Western liturgical and secular music, encompassing a broad period from roughly the 11th century to present times... , various |
150 classical, 35 various | 1 | aoTuV and AAC Advanced Audio Coding Advanced Audio Coding is a standardized, lossy compression and encoding scheme for digital audio. Designed to be the successor of the MP3 format, AAC generally achieves better sound quality than MP3 at similar bit rates.... tied (classical), aoTuV (various) |
The author selected each participating encoder by pitting multiple encoders against one another in an initial "Darwinian phase." For example, LAME was chosen as the representative MP3 encoder because it clearly outperformed four other MP3 encoders on a subset of the full sample corpus. |
| Sebastian Mares | December 2005 | multiple | ~140 (nominal 128) |
|
Various | 18 | 18-30 | 4-way tie (all except Shine) | "I think this test shows that with the current encoders, the quality at 128 kbit/s is very good... It's time to move to bitrates like 96 kbit/s or even lower (64 kbit/s)." |
| Mp3-tech.org | March 2006 | AAC | 48 |
|
Various | 18 | 10-20 | 5-way tie (all except anchors) | "... it seems that overall, plain HE-AAC might be better than HE-AAC v2 at this bitrate, but a lot more samples would be needed to be able to draw definitive conclusions regarding this. |
| Sebastian Mares | November 2006 | multiple | ~48 |
|
Various | 20 | 22-34 | Nero Digital Nero Digital Nero Digital is a brand name applied to a suite of MPEG-4-compatible video and audio compression codecs developed by Nero AG of Germany and Ateme of France... |
WMA Professional and aoTuV tied for second |
| Sebastian Mares | July 2007 | multiple | ~64 |
|
Various | 18 | 21-33 | Nero Digital Nero Digital Nero Digital is a brand name applied to a suite of MPEG-4-compatible video and audio compression codecs developed by Nero AG of Germany and Ateme of France... and WMA Professional |
|
| Sebastian Mares | October 2008 | MP3 | ~128 |
|
Various | 14 | 26-39 | 5-way tie (all except l3enc L3enc Fraunhofer l3enc was the first public software able to encode PCM files to the MP3 format. The first public version was released in July 1994. This commandline tool was shareware and limited to 112 kbit/s. It was available for MS DOS, Linux, Solaris, SunOS, NeXTstep and IRIX... ) |
"The quality at 128 kbps is very good and MP3 encoders improved a lot since the last test." Also notes that Fraunhofer and Helix codecs are several times faster at encoding than LAME, although virtually identical in terms of perceived audio quality. |
| HydrogenAudio user IgorC (March/April 2011) | March 2011 | multiple | ~64 |
CELT Constrained Energy Lapped Transform is an open, royalty-free audio compression format and a free software codec with especially low algorithmic delay for use in low-latency audio communication. It is a lossy codec, meaning quality is permanently degraded to reduce file size.The algorithms are... complexity 10, VBR 67.5 kbit/s |
Various | 30 | 25-13 | CELT | |
| HydrogenAudio user IgorC (July/August 2011) | July/August 2011 | LC-AAC | ~96 |
|
Various | 20 | 25 | Apple QuickTime |
See also
- Comparison of audio codecsComparison of audio codecsThe following tables compare general and technical information for a variety of audio formats and audio compression formats. For listening tests comparing the perceived audio quality of audio formats and codecs, see the article Codec listening test....
- High fidelity listening tests
External links
- Hydrogenaudio - Community audiophileAudiophileAn audiophile is a person who enjoys listening to recorded music, usually in a home. Some audiophiles are more interested in collecting and listening to music, while others are more interested in collecting and listening to audio components, whose "sound quality" they consider as important as the...
site, host of most non-commercial ABX testing - ff123's ABC/HR Audio Comparison Utility for Windows
- SoundExpert. Continuous blind listening tests of codecs over the internet