MPQ
Encyclopedia
MPQ is an archiving
file format
used in several of Blizzard Entertainment
's games.
MPQs used in Blizzard's games generally contain a game's data files, including graphics, sounds, and level data. The format's capabilities include compression, encryption, file segmentation, extensible file metadata, cryptographic signature and the ability to store multiple versions of the same file for internationalization
and platform-specific differences. MPQ archives can use a variety of compression algorithms which may also be combined.
using a quick, low-collision hashing
algorithm. The index of a specific file within the hash table is the hash of the lowercased filename modulo the size of the hash table, allowing for quick verification of a file's existence within the archive. If multiple files within the archive have the same hash, colliding entries will follow each other in increasing index order (forming a colliding hash cluster). In order to identify the exact entry for the requested file within a colliding hash cluster, each hash table entry stores 2 additional hashes of the lowercased filename, each using the same hashing algorithm but with a different seed value, as well as a locale code and platform code. The end of a colliding hash cluster is detected either by encountering an empty hash table entry or by traversing the entire hash table (including the modulo loopback) back to the initial hash table index.
used for file indexing are encrypted when stored. The encryption process which is used by default uses a known algorithm.
Below are known metadata files.
The following algorithms are currently in use by Blizzard games:
Since there was only one compression algorithm available when MPQs were first deployed in Diablo, those archives used a different archive file metadata flag to indicate compression and did not use a compression header byte.
codec
, BLZ0 (which actually is DivX
). These files are playable in ordinary media players, provided the proper codecs are installed. Blizzard probably used the MPQ extension on those files to hide them from players, who could have spoiled the game by seeing them out of the gameplay. Nevertheless, World of Warcraft
cinematics use the AVI extension.
DBC (WDBC) files are collections of data records used in World of Warcraft
's client-side database. They are found within the MPQ files in the game. .DBC files organize the client-side information necessary to run the game correctly.
File archiver
A file archiver is a computer program that combines a number of files together into one archive file, or a series of archive files, for easier transportation or storage...
file format
File format
A file format is a particular way that information is encoded for storage in a computer file.Since a disk drive, or indeed any computer storage, can store only bits, the computer must have some way of converting information to 0s and 1s and vice-versa. There are different kinds of formats for...
used in several of Blizzard Entertainment
Blizzard Entertainment
Blizzard Entertainment, Inc. is an American video game developer and publisher founded on February 8, 1991 under the name Silicon & Synapse by three graduates of UCLA, Michael Morhaime, Allen Adham and Frank Pearce and currently owned by French company Activision Blizzard...
's games.
MPQs used in Blizzard's games generally contain a game's data files, including graphics, sounds, and level data. The format's capabilities include compression, encryption, file segmentation, extensible file metadata, cryptographic signature and the ability to store multiple versions of the same file for internationalization
Internationalization
In economics, internationalization has been viewed as a process of increasing involvement of enterprises in international markets, although there is no agreed definition of internationalization or international entrepreneurship...
and platform-specific differences. MPQ archives can use a variety of compression algorithms which may also be combined.
File indexing
In order to meet the requirements of speed generally demanded by a computer game, files are indexed in a hash tableHash table
In computer science, a hash table or hash map is a data structure that uses a hash function to map identifying values, known as keys , to their associated values . Thus, a hash table implements an associative array...
using a quick, low-collision hashing
Hash function
A hash function is any algorithm or subroutine that maps large data sets to smaller data sets, called keys. For example, a single integer can serve as an index to an array...
algorithm. The index of a specific file within the hash table is the hash of the lowercased filename modulo the size of the hash table, allowing for quick verification of a file's existence within the archive. If multiple files within the archive have the same hash, colliding entries will follow each other in increasing index order (forming a colliding hash cluster). In order to identify the exact entry for the requested file within a colliding hash cluster, each hash table entry stores 2 additional hashes of the lowercased filename, each using the same hashing algorithm but with a different seed value, as well as a locale code and platform code. The end of a colliding hash cluster is detected either by encountering an empty hash table entry or by traversing the entire hash table (including the modulo loopback) back to the initial hash table index.
Encryption
Both the block table (which contains information on where the file data is located in the archive) and the hash tableHash table
In computer science, a hash table or hash map is a data structure that uses a hash function to map identifying values, known as keys , to their associated values . Thus, a hash table implements an associative array...
used for file indexing are encrypted when stored. The encryption process which is used by default uses a known algorithm.
Revisions
The file header reserves space to contain format version data. Warcraft III ignores format version data of .mpq compliant files it loads and assumes all are version 1.- Version 1 was used before World of WarcraftWorld of WarcraftWorld of Warcraft is a massively multiplayer online role-playing game by Blizzard Entertainment. It is the fourth released game set in the fantasy Warcraft universe, which was first introduced by Warcraft: Orcs & Humans in 1994...
. - Version 2 added an extended header to the format which contained data for an extended block table to allow for larger archive sizes.
Archive metadata
MPQ archives do not have specific structures to store metadata beyond what is absolutely necessary to access archived files. Instead, the convention is to use regular files whose filename is enclosed by parentheses.Below are known metadata files.
- (listfile): Contains a list of the archive's files, one filename per line. May or may not be exhaustive.
- (signature): Contains the weak cryptographic signature of the archive. This type of signature is deprecated.
- (attributes): Contains extended file metadata. Currently known attributes are file creation date, CRC32 checksum and MD5 checksum.
Compression
In modern MPQ archives, each segment (or sector) of a file can be compressed using a combination of compression algorithms. A header byte is prepended to every compressed sector to indicate which compressions were used. The order in which those compressors are applied is hardcoded.The following algorithms are currently in use by Blizzard games:
- PKZIPPKZIPPKZIP is an archiving tool originally written by Phil Katz and marketed by his company PKWARE, Inc. The common "PK" prefix used in both PKZIP and PKWARE stands for "Phil Katz".-History:...
(licensed from PKWARE). The first compression algorithm available. - HuffmanHuffman codingIn computer science and information theory, Huffman coding is an entropy encoding algorithm used for lossless data compression. The term refers to the use of a variable-length code table for encoding a source symbol where the variable-length code table has been derived in a particular way based on...
tree compression combined with ADPCM 4:1 compression (both introduced in StarCraft). Latter algorithm is lossy and only suitable for raw PCM input data. - zlibZlibzlib is a software library used for data compression. zlib was written by Jean-Loup Gailly and Mark Adler and is an abstraction of the DEFLATE compression algorithm used in their gzip file compression program. Zlib is also a crucial component of many software platforms including Linux, Mac OS X,...
(introduced in Warcraft III). - bzip2Bzip2bzip2 is a free and open source implementation of the Burrows–Wheeler algorithm. It is developed and maintained by Julian Seward. Seward made the first public release of bzip2, version 0.15, in July 1996.-Compression efficiency:...
(introduced in World of Warcraft). - LZMALZMAThe Lempel–Ziv–Markov chain algorithm is an algorithm used to perform data compression. It has been under development since 1998 and was first used in the 7z format of the 7-Zip archiver...
(introduced in StarCraft II).
Since there was only one compression algorithm available when MPQs were first deployed in Diablo, those archives used a different archive file metadata flag to indicate compression and did not use a compression header byte.
Warcraft III cinematics
Cutscene cinematics with the MPQ extension are included with Warcraft III and, despite the file extension, are not actual MPQ files. Rather they are AVI files compressed with Blizzard's renamed MPEG-4MPEG-4
MPEG-4 is a method of defining compression of audio and visual digital data. It was introduced in late 1998 and designated a standard for a group of audio and video coding formats and related technology agreed upon by the ISO/IEC Moving Picture Experts Group under the formal standard ISO/IEC...
codec
Codec
A codec is a device or computer program capable of encoding or decoding a digital data stream or signal. The word codec is a portmanteau of "compressor-decompressor" or, more commonly, "coder-decoder"...
, BLZ0 (which actually is DivX
DivX
DivX is a brand name of products created by DivX, Inc. , including the DivX Codec which has become popular due to its ability to compress lengthy video segments into small sizes while maintaining relatively high visual quality.There are two DivX codecs; the regular MPEG-4 Part 2 DivX codec and the...
). These files are playable in ordinary media players, provided the proper codecs are installed. Blizzard probably used the MPQ extension on those files to hide them from players, who could have spoiled the game by seeing them out of the gameplay. Nevertheless, World of Warcraft
World of Warcraft
World of Warcraft is a massively multiplayer online role-playing game by Blizzard Entertainment. It is the fourth released game set in the fantasy Warcraft universe, which was first introduced by Warcraft: Orcs & Humans in 1994...
cinematics use the AVI extension.
DBC (WDBC) files are collections of data records used in World of Warcraft
World of Warcraft
World of Warcraft is a massively multiplayer online role-playing game by Blizzard Entertainment. It is the fourth released game set in the fantasy Warcraft universe, which was first introduced by Warcraft: Orcs & Humans in 1994...
's client-side database. They are found within the MPQ files in the game. .DBC files organize the client-side information necessary to run the game correctly.
Usage in gaming
Blizzard has utilized the MPQ file format for archiving game files in a number of their games, including:- Diablo
- Diablo: HellfireDiablo: HellfireDiablo: Hellfire is an expansion pack that Sierra On-Line produced for the video game Diablo. It was released in 1997 and developed by Synergistic Software, a Sierra division. Hellfire is the only authorized expansion pack released for Diablo. Blizzard Entertainment has never released a...
developed by Synergistic SoftwareSynergistic SoftwareNorthwest Synergistic Software is a contract software developer and a former video game developer. Founded in 1978 under the name Synergistic Software, the company published some of the earliest available games and applications for the Apple II family of computers... - Lords of MagicLords of MagicLords of Magic is a turn-based strategy PC game designed for Windows 95/98 by Sierra Entertainment. The game, which was distributed on CDs, includes a single-player mode and limited multiplayer features on LAN and the Internet as well...
developed by Sierra EntertainmentSierra EntertainmentSierra Entertainment Inc. was an American video-game developer and publisher founded in 1979 as On-Line Systems by Ken and Roberta Williams... - Diablo IIDiablo IIDiablo II is a dark fantasy/horror-themed hack and slash, with elements of the role playing game and dungeon crawl genres. It was released for Windows and Mac OS in 2000 by Blizzard Entertainment, and was developed by Blizzard North. It is a direct sequel to the 1996 hit PC game, Diablo.Diablo II...
- Diablo II: Lord of DestructionDiablo II: Lord of DestructionDiablo II: Lord of Destruction is an expansion pack for the hack and slash action role-playing game Diablo II...
- StarCraftStarCraftStarCraft is a military science fiction real-time strategy video game developed by Blizzard Entertainment. The first game of the StarCraft series was released for Microsoft Windows on 31 March 1998. With more than 11 million copies sold worldwide as of February 2009, it is one of the best-selling...
- StarCraft: Brood WarStarCraft: Brood WarStarCraft: Brood War is the expansion pack for the award winning military science fiction, real-time strategy video game StarCraft. Released in 1998 for Windows and Mac OS, it was co-developed by Saffire and Blizzard Entertainment. The expansion pack introduced new campaigns, map tilesets, music,...
- WarCraft II: Battle.net EditionWarcraft II: Tides of DarknessWarcraft II: Tides of Darkness is a fantasy-themed real-time strategy game published by Blizzard Entertainment and first released for DOS in 1995 and for Mac OS in 1996...
- Warcraft III: Reign of ChaosWarcraft III: Reign of ChaosWarcraft III: Reign of Chaos is a real time strategy computer game released by Blizzard Entertainment on July 3, 2002 . It is the second sequel to Warcraft: Orcs & Humans, and it is the third game set in the Warcraft Universe...
- Warcraft III: The Frozen ThroneWarcraft III: The Frozen ThroneWarcraft III: The Frozen Throne is a real-time strategy computer game developed for Microsoft Windows, Mac OS and Mac OS X by Blizzard Entertainment. It is the official expansion pack to Warcraft III: Reign of Chaos, requiring Reign of Chaos to play...
- World of WarcraftWorld of WarcraftWorld of Warcraft is a massively multiplayer online role-playing game by Blizzard Entertainment. It is the fourth released game set in the fantasy Warcraft universe, which was first introduced by Warcraft: Orcs & Humans in 1994...
- World of Warcraft: The Burning CrusadeWorld of Warcraft: The Burning CrusadeWorld of Warcraft: The Burning Crusade, often referred to as TBC or BC, is the first expansion pack for the MMORPG World of Warcraft...
- World of Warcraft: Wrath of the Lich KingWorld of Warcraft: Wrath of the Lich KingWorld of Warcraft: Wrath of the Lich King, often referred to as WotLK, WLK or Wrath, is the second expansion set for the massively multiplayer online role-playing game World of Warcraft, following The Burning Crusade...
- World of Warcraft: CataclysmWorld of Warcraft: CataclysmWorld of Warcraft: Cataclysm is the third expansion pack for the massively multiplayer online role-playing game World of Warcraft, following the last expansion Wrath of the Lich King. It was officially announced at BlizzCon on August 21, 2009, although dataminers and researchers discovered details...
- StarCraft II: Wings of Liberty
- Diablo IIIDiablo IIIDiablo III is an upcoming dark fantasy/horror-themed action role-playing game in development by Blizzard, making it the third installment in the Diablo franchise...
Specifications
- The MoPaQ Archive Format
- Inside MoPaQ — A guide to the file format. (Not maintained, Win32 only, no source code.).
Libraries
- StormLib is high level opensource cross-platform C++ library for manipulating MPQ archives. It also supports signature verification.
- libmpq is opensource cross-platform C library (and associated tools) for extracting files from MPQ archives.
- jlibmpq java wrapper for libmpq.
- MPQKit is a Mac OS X Objective-C framework for manipulating MPQ archives. It supports signature verification too.
- MpqLib is a .NET 3.5 object oriented library for handling mpq archives and files.
Programs
- SMPQ, an open souce command line program and KDE4KDE4KDE Software Compilation 4 is the current series of releases of KDE Software Compilation. The first major version of this series was released on 11 January 2008, and the latest major version was released on 27 July 2011...
KIOKIOKIO is part of the KDE architecture. It provides access to files, web sites and other resources through a single consistent API. Applications, such as Konqueror which are written using this framework can operate on files stored on remote servers in exactly the same way as they operate on those...
plugin based on StormLib that can create, append or extract files from MPQ archives. - MPQDraft, an open source program hosted on SourceforgeSourceForgeSourceForge Enterprise Edition is a collaborative revision control and software development management system. It provides a front-end to a range of software development lifecycle services and integrates with a number of free software / open source software applications .While originally itself...
that allows users to work with MPQ files. - MPQ Extractor — A Mac OS X extractor utility.
- Ladik's MPQ Editor, a .MPQ file editor for Windows, Storm.dll, StormLib, libraries for reading and writing .MPQ files.
- WoW Model Viewer, a program that can open the World of WarcraftWorld of WarcraftWorld of Warcraft is a massively multiplayer online role-playing game by Blizzard Entertainment. It is the fourth released game set in the fantasy Warcraft universe, which was first introduced by Warcraft: Orcs & Humans in 1994...
MPQ files for viewing of its internal files such as the .MD2MD2 (file format)MD2 is a model format used by id Software's id Tech 2 engine and is thus used by Quake II as well as many other games, most of them using this engine, including SiN and Soldier of Fortune. The format is primarily used for animated player models although it can also be used for static models...
files and .MS3D files.