UEF (file format)
Encyclopedia
Unified Emulator Format (UEF) is a container format for the compressed
storage of audio tapes, ROM
s, floppy disk
s and machine state snapshots for the 8-bit range of computers manufactured by Acorn Computers
. First implemented by Thomas Harte's ElectrEm emulator and related tools, it is now supported by major emulators of Acorn machines and carried by two online archives of Acorn software numbering thousands of titles.
UEF attempts to concisely reproduce media borne signals rather than simply the data represented by them, the intention being an accurate archive
of original media rather than merely a capability to reproduce files stored on them. A selection of metadata
can be included, such as compatibility ratings, position markers, images of packaging and the text of instruction manuals.
The Acorn machines implement the Kansas City standard
(KCS) for tape data encoding and as a result the file format is suitable for creating backup
s of original media for several non-Acorn machines. As of version 0.10 the file format carries BASICODE
signals as well.
TZX is a chunked format with similar scope for the ZX Spectrum
series.
had adopted a convention of hosting ZIP
archives of the raw files on a tape, each raw file accompanied by a sidecar file
, with extension INF, carrying the load and execution addresses from the file header. The INF convention, described and implemented by Wouter Scholten in bbcim (1995), extends the output format of the
, ADFS
) to cover CRCs
and the order of files on tape. While it works adequately for storing user files, it does not preserve the baud
rate of the recording, precise timing information or the non-standard data streams used in copy protected titles.
In the case of disk-based software, it became increasingly convenient to send a sector dump of the disk instead, and by the time of the UEF's introduction the file extensions
of chunks containing the data of interest. The header comprises the magic
string
, and the two-byte version number of the UEF specification in use. A reading application needs to pay attention to the version number, as the unit of measurement in some chunks differs according to the specification version, and one chunk has been redefined between versions.
Each chunk consists of a two-byte ID which determines its meaning, the length of the body in four bytes, and the body itself. An application can readily skip the bodies of chunks it does not need to process. After the last chunk the file simply ends. Currently, UEF chunks do not nest.
The whole UEF file, including the header, may optionally be compressed in gzip
format. By examining the start of the file for a gzip or UEF header, a decompression library can be invoked as appropriate.
, the modulated asynchronous signals of ordinary data blocks, security cycles (modulated synchronous signals, said to be an "identification feature") or gaps where no recognised signal is present. Tape UEF chunks are concatenated in the order they appear, to build up the representation of a whole recording. When generated from a real source tape, each waveform on the tape corresponds directly to a tape chunk, such that the source can be accurately reconstructed (with any non-encodable signals replaced by gaps of equal length.)
Standard Acorn streams (chunk ID: 0x0100) are encoded so that their bytes reappear in the UEF chunk body. From version 0.10, direct support is extended to all asynchronous formats (0x0104) including the
There are some modal variables affecting the interpretation of these chunks: the baud rate, 1200 baud for Acorn signals or 300 baud for KCS; the exact carrier frequency, which determines the playing time of the reconstructed tape; and the phase
of the signal. The latter two may change within a published recording, and their absolute values depend on the tape player, amplifier and sound card used to digitise the signal.
A UEF file can contain markers to separate the tapes of a multiple-tape distribution, and the sides of each tape; positions of interest within each side can also be marked.
Disks are stored as raw sector dumps of each surface, plus their geometry
and a byte identifying the file system
. Previous versions of the specification had provisions to encode disks at the byte stream level, or the magnetic domain level. With SSD and DSD sector dumps serving standard BBC disks well, and the mature FDI
format catering for copy-protected software, the disk image function of UEF is little used.
Sideways
ROMs are likewise stored as raw data, plus an indication of their purpose and a ROM slot recommendation. Again the user base prefers bare ROM dumps for archival.
State snapshot UEF files include standardised chunks to store the major portions of an Acorn Electron or BBC Micro's state: main, shadow and expansion bus memory, the CPU registers and the WD1770 floppy drive controller; also the Electron ULA and the Slogger Master RAM Board, a common Electron add-on. A patch memory chunk rewrites a block of memory at any address, allowing the UEF format to package pokes
. To store state elements not accommodated in the standard chunks, emulators can define their own chunks. A private use area of chunk IDs is reserved for this or any other purpose, although some emulators save state under invalid chunk IDs in the public space.
Multiplexed data is an extension for emulators, used by ElectrEm but without a published specification:
One salient application mentioned by Harte is to superimpose "new graphics on old games", and a single example, a 256-color enhanced Daredevil Dennis, is available from StairwayToHell.com to run in ElectrEm.
Multiplexed data chunks are intended to follow ordinary data chunks in any of the above classes, supplementing the data. Their contents are not meant to be visible to the Acorn computer, whether real or emulated, but otherwise their meaning has not been specified.
Chunks providing content information include the file origin chunk, which identifies the application that generated the UEF file. Inlay scan chunks, intended as a file preview, hold a raw bitmap of the cover art
although an image any larger than a thumbnail
can take up more space than a typical game. The UEF author can also provide the text of an instruction booklet or a URL for more information, a short title for display, minimum machine specification and keyboard mapping for the enclosed software; and where a game does not use the whole screen, the coordinates of the visible area can be given. A minority of UEF files available online contain anything in this class but an origin chunk.
A UEF file can contain multiple classes of data at once, as Harte intended; it is not possible to know which classes it contains without scanning the whole file. In its file selection box ElectrEm displays an icon according to the first data class chunk it finds.
files or a live signal played to the sound card, and transcribes only standard data blocks with accuracy. The 'professional' grade accepts only CSW files, which represent waves preprocessed into rectangular pulse train
s, but it encodes all audio information supported by the UEF specification.
MakeUEF claims to have been the sole creator of all UEF files available on the Web before November 2004, the month of its version 1.0 release. Although the file format was more capable, supporting "gap lengths" since February 2001 at the latest, only "program data" was retained by MakeUEF prior to version 1.0. From November 2004 the fidelity of MakeUEF improved and the file spec was further refined, and an extension of
Data compression
In computer science and information theory, data compression, source coding or bit-rate reduction is the process of encoding information using fewer bits than the original representation would use....
storage of audio tapes, ROM
Read-only memory
Read-only memory is a class of storage medium used in computers and other electronic devices. Data stored in ROM cannot be modified, or can be modified only slowly or with difficulty, so it is mainly used to distribute firmware .In its strictest sense, ROM refers only...
s, floppy disk
Floppy disk
A floppy disk is a disk storage medium composed of a disk of thin and flexible magnetic storage medium, sealed in a rectangular plastic carrier lined with fabric that removes dust particles...
s and machine state snapshots for the 8-bit range of computers manufactured by Acorn Computers
Acorn Computers
Acorn Computers Ltd. was a British computer company established in Cambridge, England, in 1978. The company produced a number of computers which were especially popular in the UK. These included the Acorn Electron, the BBC Micro, and the Acorn Archimedes...
. First implemented by Thomas Harte's ElectrEm emulator and related tools, it is now supported by major emulators of Acorn machines and carried by two online archives of Acorn software numbering thousands of titles.
UEF attempts to concisely reproduce media borne signals rather than simply the data represented by them, the intention being an accurate archive
Archive
An archive is a collection of historical records, or the physical place they are located. Archives contain primary source documents that have accumulated over the course of an individual or organization's lifetime, and are kept to show the function of an organization...
of original media rather than merely a capability to reproduce files stored on them. A selection of metadata
Metadata
The term metadata is an ambiguous term which is used for two fundamentally different concepts . Although the expression "data about data" is often used, it does not apply to both in the same way. Structural metadata, the design and specification of data structures, cannot be about data, because at...
can be included, such as compatibility ratings, position markers, images of packaging and the text of instruction manuals.
The Acorn machines implement the Kansas City standard
Kansas City standard
The Kansas City Standard , or Byte standard, is a digital data format for audio cassette drives. Byte magazine sponsored a symposium in November 1975 in Kansas City, Missouri to develop a standard for storage of digital computer data on inexpensive consumer quality cassettes, at a time when...
(KCS) for tape data encoding and as a result the file format is suitable for creating backup
Backup
In information technology, a backup or the process of backing up is making copies of data which may be used to restore the original after a data loss event. The verb form is back up in two words, whereas the noun is backup....
s of original media for several non-Acorn machines. As of version 0.10 the file format carries BASICODE
BASICODE
BASICODE was a computer project intended to create a unified standard for the BASIC programming language. BASIC was available on many popular home computers, but there were countless variants that were mostly incompatible with each other...
signals as well.
TZX is a chunked format with similar scope for the ZX Spectrum
ZX Spectrum
The ZX Spectrum is an 8-bit personal home computer released in the United Kingdom in 1982 by Sinclair Research Ltd...
series.
History
Before the development of the UEF, archives of Acorn computer software on the World Wide WebWorld Wide Web
The World Wide Web is a system of interlinked hypertext documents accessed via the Internet...
had adopted a convention of hosting ZIP
ZIP (file format)
Zip is a file format used for data compression and archiving. A zip file contains one or more files that have been compressed, to reduce file size, or stored as is...
archives of the raw files on a tape, each raw file accompanied by a sidecar file
Sidecar file
Sidecar files, also known as buddy files or connected files, are files which store data which is not supported by the source file format.For each source file one or more sidecar files can be created...
, with extension INF, carrying the load and execution addresses from the file header. The INF convention, described and implemented by Wouter Scholten in bbcim (1995), extends the output format of the
*INFO command (Acorn DFSDisc Filing System
The Disc Filing System is a computer file system developed by Acorn Computers Ltd, and introduced in 1982 for the Acorn BBC Microcomputer. It was shipped as a ROM to be inserted onto the BBC Micro's motherboard. It has an extremely limited design, and uses a flat directory structure...
, ADFS
Advanced Disc Filing System
The Advanced Disc Filing System is a computing file system particular to the Acorn computer range and RISC OS based successors. Initially based on the rare Acorn Winchester Filing System, it was renamed to the Advanced Disc Filing System when support for floppy discs was added and on later 32 bit...
) to cover CRCs
Cyclic redundancy check
A cyclic redundancy check is an error-detecting code commonly used in digital networks and storage devices to detect accidental changes to raw data...
and the order of files on tape. While it works adequately for storing user files, it does not preserve the baud
Baud
In telecommunications and electronics, baud is synonymous to symbols per second or pulses per second. It is the unit of symbol rate, also known as baud rate or modulation rate; the number of distinct symbol changes made to the transmission medium per second in a digitally modulated signal or a...
rate of the recording, precise timing information or the non-standard data streams used in copy protected titles.
In the case of disk-based software, it became increasingly convenient to send a sector dump of the disk instead, and by the time of the UEF's introduction the file extensions
.ssd and .dsd were already established for single-sided and double-sided raw images of DFS disks, respectively. Distributed bare or in a ZIP archive, they remain popular on archive sites.Aims
In a 2010 post to the Stardot forum, Harte explained at length his reasons for creating the format: being the first to address emulation of the Acorn Electron and its primary medium, tape, Harte wanted a fine-grained and technically optimal representation of media, compared to existing ad-hoc formats; and to package the multiple media elements of a software release into a single file, so that downloading a UEF is "more like obtaining the original product". He went on to observe that it was the tools in use, and "user need", that determined the actual uses to which the UEF had been put.Structure
A UEF file consists of a fixed length header that identifies itself, followed by a linked listLinked list
In computer science, a linked list is a data structure consisting of a group of nodes which together represent a sequence. Under the simplest form, each node is composed of a datum and a reference to the next node in the sequence; more complex variants add additional links...
of chunks containing the data of interest. The header comprises the magic
Magic number (programming)
In computer programming, the term magic number has multiple meanings. It could refer to one or more of the following:* A constant numerical or text value used to identify a file format or protocol; for files, see List of file signatures...
string
UEF File!, a terminating null characterNull character
The null character , abbreviated NUL, is a control character with the value zero.It is present in many character sets, including ISO/IEC 646 , the C0 control code, the Universal Character Set , and EBCDIC...
, and the two-byte version number of the UEF specification in use. A reading application needs to pay attention to the version number, as the unit of measurement in some chunks differs according to the specification version, and one chunk has been redefined between versions.
Each chunk consists of a two-byte ID which determines its meaning, the length of the body in four bytes, and the body itself. An application can readily skip the bodies of chunks it does not need to process. After the last chunk the file simply ends. Currently, UEF chunks do not nest.
The whole UEF file, including the header, may optionally be compressed in gzip
Gzip
Gzip is any of several software applications used for file compression and decompression. The term usually refers to the GNU Project's implementation, "gzip" standing for GNU zip. It is based on the DEFLATE algorithm, which is a combination of Lempel-Ziv and Huffman coding...
format. By examining the start of the file for a gzip or UEF header, a decompression library can be invoked as appropriate.
Content
The Unified Emulator Format models software on cassette as a contiguous sequence of segments, which may be carrier tonesCarrier wave
In telecommunications, a carrier wave or carrier is a waveform that is modulated with an input signal for the purpose of conveying information. This carrier wave is usually a much higher frequency than the input signal...
, the modulated asynchronous signals of ordinary data blocks, security cycles (modulated synchronous signals, said to be an "identification feature") or gaps where no recognised signal is present. Tape UEF chunks are concatenated in the order they appear, to build up the representation of a whole recording. When generated from a real source tape, each waveform on the tape corresponds directly to a tape chunk, such that the source can be accurately reconstructed (with any non-encodable signals replaced by gaps of equal length.)
Standard Acorn streams (chunk ID: 0x0100) are encoded so that their bytes reappear in the UEF chunk body. From version 0.10, direct support is extended to all asynchronous formats (0x0104) including the
8,N,2 format of BASICODE. Otherwise there is a generic chunk (0x0102) to accommodate any arbitrary sequence of bits. Security wave chunks (0x0114) also carry bit streams, encoded in a different form to allow the half-length one bits observed in commercial recordings to be represented.There are some modal variables affecting the interpretation of these chunks: the baud rate, 1200 baud for Acorn signals or 300 baud for KCS; the exact carrier frequency, which determines the playing time of the reconstructed tape; and the phase
Phase (waves)
Phase in waves is the fraction of a wave cycle which has elapsed relative to an arbitrary point.-Formula:The phase of an oscillation or wave refers to a sinusoidal function such as the following:...
of the signal. The latter two may change within a published recording, and their absolute values depend on the tape player, amplifier and sound card used to digitise the signal.
A UEF file can contain markers to separate the tapes of a multiple-tape distribution, and the sides of each tape; positions of interest within each side can also be marked.
Disks are stored as raw sector dumps of each surface, plus their geometry
Cylinder-head-sector
Cylinder-head-sector, also known as CHS, was an early method for giving addresses to each physical block of data on a hard disk drive. In the case of floppy drives, for which the same exact diskette medium can be truly low-level formatted to different capacities, this is still true.Though CHS...
and a byte identifying the file system
File system
A file system is a means to organize data expected to be retained after a program terminates by providing procedures to store, retrieve and update data, as well as manage the available space on the device which contain it. A file system organizes data in an efficient manner and is tuned to the...
. Previous versions of the specification had provisions to encode disks at the byte stream level, or the magnetic domain level. With SSD and DSD sector dumps serving standard BBC disks well, and the mature FDI
Formatted Disk Image
FDI is a file image of a disc created with the Disk2FDI PC software. It is often used to read Amiga or C64 discs and other discs formatted in a way that normally can't be read with a PC. The software uses two methods:...
format catering for copy-protected software, the disk image function of UEF is little used.
Sideways
Sideways address space
The "Sideways" address space on the Acorn BBC Microcomputer and Master-series microcomputer was Acorn's bank switching implementation, providing for permanent system expansion in the days before hard disk drives or even floppy disk drives were commonplace...
ROMs are likewise stored as raw data, plus an indication of their purpose and a ROM slot recommendation. Again the user base prefers bare ROM dumps for archival.
State snapshot UEF files include standardised chunks to store the major portions of an Acorn Electron or BBC Micro's state: main, shadow and expansion bus memory, the CPU registers and the WD1770 floppy drive controller; also the Electron ULA and the Slogger Master RAM Board, a common Electron add-on. A patch memory chunk rewrites a block of memory at any address, allowing the UEF format to package pokes
PEEK and POKE
In computing, PEEK is a BASIC programming language extension used for reading the contents of a memory cell at a specified address. The corresponding command to set the contents of a memory cell is POKE.-Statement syntax:...
. To store state elements not accommodated in the standard chunks, emulators can define their own chunks. A private use area of chunk IDs is reserved for this or any other purpose, although some emulators save state under invalid chunk IDs in the public space.
Multiplexed data is an extension for emulators, used by ElectrEm but without a published specification:
One salient application mentioned by Harte is to superimpose "new graphics on old games", and a single example, a 256-color enhanced Daredevil Dennis, is available from StairwayToHell.com to run in ElectrEm.
Multiplexed data chunks are intended to follow ordinary data chunks in any of the above classes, supplementing the data. Their contents are not meant to be visible to the Acorn computer, whether real or emulated, but otherwise their meaning has not been specified.
Chunks providing content information include the file origin chunk, which identifies the application that generated the UEF file. Inlay scan chunks, intended as a file preview, hold a raw bitmap of the cover art
Cover art
Cover art is the illustration or photograph on the outside of a published product such as a book , magazine, comic book, video game , DVD, CD, videotape, or music album. The art has a primarily commercial function, i.e...
although an image any larger than a thumbnail
Thumbnail
Thumbnails are reduced-size versions of pictures, used to help in recognizing and organizing them, serving the same role for images as a normal text index does for words...
can take up more space than a typical game. The UEF author can also provide the text of an instruction booklet or a URL for more information, a short title for display, minimum machine specification and keyboard mapping for the enclosed software; and where a game does not use the whole screen, the coordinates of the visible area can be given. A minority of UEF files available online contain anything in this class but an origin chunk.
A UEF file can contain multiple classes of data at once, as Harte intended; it is not possible to know which classes it contains without scanning the whole file. In its file selection box ElectrEm displays an icon according to the first data class chunk it finds.
MakeUEF
MakeUEF is a Windows application written by Thomas Harte and expanded by Fraser Ross to convert audio samples into UEF files. Two grades are offered. An 'amateur' version reads WAVWAV
Waveform Audio File Format , is a Microsoft and IBM audio file format standard for storing an audio bitstream on PCs...
files or a live signal played to the sound card, and transcribes only standard data blocks with accuracy. The 'professional' grade accepts only CSW files, which represent waves preprocessed into rectangular pulse train
Pulse wave
A pulse wave or pulse train is a kind of non-sinusoidal waveform that is similar to a square wave, but does not have the symmetrical shape associated with a perfect square wave. It is a term common to synthesizer programming, and is a typical waveform available on many synths. The exact shape of...
s, but it encodes all audio information supported by the UEF specification.
MakeUEF claims to have been the sole creator of all UEF files available on the Web before November 2004, the month of its version 1.0 release. Although the file format was more capable, supporting "gap lengths" since February 2001 at the latest, only "program data" was retained by MakeUEF prior to version 1.0. From November 2004 the fidelity of MakeUEF improved and the file spec was further refined, and an extension of
.hq.uef ("high quality") was adopted to reflect this. The AcornPreservation.org archive only carries the HQ.UEF variety as well as the CSW source files. Its sister site StairwayToHell.com accepts 'amateur' UEF translations and files produced by pre-1.0 MakeUEF. the latter site hosts 1,494 transcriptions of BBC Micro cassette titles and at least 800 of Electron titles.Others
- Several emulators of Acorn machines support UEF natively, to read and write tape data (at original speed or faster) and store state snapshots. Examples include ElectrEm, BeebEm and B-Em.
- FreeUEF by Thomas Harte and the UEFReader Java Sound plugin convert a UEF file to a wave suitable for recording on tape or playing back to a physical computer.
- UberCassette is a cross-platform, multi-format encoder emitting UEF from samples of Acorn cassettes.
- The UEFwalk script validates and extracts data from UEF files.
- The XVUEF patch extends the XvXvxv is a shareware program written by John Bradley to display and modify digital images under the X Window System.While popular in the early 1990s , no official releases have been made since December 1994...
image editor to support the little-used inlay scan chunks of the UEF.