XMODEM
Encyclopedia
XMODEM is a simple file transfer
protocol developed as a quick hack by Ward Christensen
for use in his 1977 MODEM.ASM terminal program. XMODEM became extremely popular in the early bulletin board system
(BBS) market, largely because it was so simple to implement. It was also fairly inefficient, and as modem speeds increased this problem led to the development of a number of modified versions of XMODEM to improve performance or address other problems with the protocol. Chuck Forsberg
collected a number of these into his YMODEM
protocol, but poor implementation led to a further fracturing before they were re-unified by his later ZMODEM
protocol.
floppy disk
s. The packet was prefixed by a simple 3-byte header containing a <SOH
> character, a "block number" from 0-255, and the "inverse" block number—255 minus the block number. Block numbering starts with 1 for the first block sent.
The packet was also suffixed with a single-byte checksum
of the data bytes. The checksum was the sum of all bytes in the packet modulo
256. The modulo operation was easily computed by discarding all but the eight least significant bit
s of the result, or alternatively on an eight bit machine, ignoring arithmetic overflow
which would produce the same effect automatically. In this way the checksum was restricted to an eight bit quantity which was able to be expressed using a single byte. For example, if the data bytes were 130 and 130, the total of these codes is 260 and the resulting checksum is 4 using this method.
The complete packet was thus 132 bytes long, containing 128 bytes of data, for a total throughput efficiency of about 97%.
The file was marked "complete" with a <EOT
> character sent after the last block. This character was not in a packet, but sent alone as a single byte. Since the file length was not sent as part of the protocol, the last packet was padded out with a "known character" that could be dropped. In the original specification this defaulted to
> message back to the sender, which then sent the next packet in sequence. If there was a problem with the checksum, the receiver instead sent a <NAK
>. If a <NAK
> was received, the sender would re-send the packet, and continued to try several times, normally ten, before aborting the transfer.
A
The block numbers were also examined in a simple way to check for errors. After receiving a packet successfully, the next packet should have a one-higher number. If it instead received the same block number this was not considered serious, it was implied that the
Transfers were sender-driven, the receiver acting passively on data sent to it. However, the transfer was actually started by the receiver sending a single
machines, and bears several marks of that operating system
. Notably, files on CP/M were always multiples of 128 bytes, and their end was marked within a block with the
in the early 1980s led to XMODEM having to be updated to notice either a
For some time it was suggested that sending a
Many authors introduced extensions to XMODEM to address these and other problems. Many asked for these extensions to be included as part of a new XMODEM standard. However, Ward Christensen refused to do this, as it was precisely the lack of these features, and the associated coding needed to support them, that led to XMODEM's widespread use. As he explained:
For automated transfers between two sites, a number of add-ons to the XMODEM protocol were implemented over time. These generally assumed the sender would continue sending file after file, with the receiver attempting to trigger the next file by sending a
, author of the original FidoNet
mailers.
TeLink avoided MODEM7's problems by standardizing a new "zero packet" containing information about the original file. This included the file's name, size, and timestamp
, which were placed in a regular 128 byte XMODEM block. Whereas a normal XMODEM transfer would start with the sender sending "block 1", the TeLink header packet was labeled "block 0".
Again, a normal XMODEM implementation would simply discard the packet, the assumption being that the packet number had been corrupted. But this led to a potential time delay if the packet were discarded, as the sender could not be sure it was being
The basic "block 0" system became a standard in the FidoNet community, and was re-used by a number of future protocols like SEAlink
and YMODEM
.
in place of the 8-bit checksum. CRC's encode not only the data in the packet, but its location as well, allowing it to notice the bit-replacement errors that a checksum would miss. Statistically, this made the chance of detecting an error less than 16 bytes long 99.9969%, and even higher for longer data.
XMODEM-CRC was designed to be backwardly compatible with XMODEM. To do this, the receiver simply sent a C (capital C) character instead of a
Unfortunately this attempt at backward compatibility had a downside. Since it was possible that the initial C character would be lost or corrupted, it could not be assumed that the receiver did not support XMODEM-CRC if the first attempt to trigger the transfer failed. The receiver thus tried to start the transfer three times with C, waiting three seconds between each attempt. This meant that if the user selected XMODEM-CRC while attempting to talk to any XMODEM, as it was intended, there was a potential 10 second delay before the transfer started.
To avoid the delay, the sender and receiver would generally list XMODEM-CRC separately from XMODEM, allowing the user to select "basic" XMODEM if the sender didn't explicitly list it. Ironically, any software that did support -CRC in their basic XMODEM transfer, as it was intended, surreptitiously suggested the user should not attempt to use -CRC. To the average user, XMODEM-CRC was essentially a "second protocol", and treated as such.
As modem speeds increase, the fixed delay needed to send the
A number of new versions of XMODEM were introduced in order to address these problems. Like earlier extensions, these versions tended to be backward-compatible with the original XMODEM, and like those extensions, this led to a further fracturing of the XMODEM landscape in the user's terminal emulator. In the end, dozens of versions of XMODEM would emerge.
system was SEAdog, written by the same author as the then-popular .arc data compression
format. SEAdog included a wide variety of improvements, including SEAlink
, an improved transfer protocol.
SEAlink used a method known as sliding windows to avoid the inter-packet delay. To do this, the protocol did not wait for the
SEAlink later added a number of other improvements, and was generally a useful protocol. However it remained rare, and was typically only found in FidoNet mailers.
XMODEM-1K was an expanded version of XMODEM-CRC, which indicated the longer block size in the sender by starting a packet with the
XMODEM-1K was actually one of the many improvements to XMODEM introduced by Chuck Forsberg
in his YMODEM
protocol. Forsberg suggested that the various improvements were optional, expecting software authors to implement as many of them as possible. Instead they generally implemented the bare minimum, leading to a profusion of semi-compatible implementations, and eventually, the splitting out of the name "YMODEM" into "XMODEM-1K" and a variety of YMODEMs. Thus XMODEM-1K actually post-dates YMODEM, but remained fairly common anyway.
A backwards compatible extensions of XMODEM with 32k and 64k block lengths was created by Adontec for better performance on high-speed error free connections like ISDN or TCP/IP networks.
Since this feature is only an alteration of the receiver-side behaviour, it does not require any changes in the protocol or on the sender's side.
Pre-acknowledge was also possible for YMODEM
. It was made obsolete by variants such as YMODEM-g or ZMODEM
.
File transfer
File transfer is a generic term for the act of transmitting files over a computer network or the Internet. There are numerous ways and protocols to transfer files over a network. Computers which provide a file transfer service are often called file servers. Depending on the client's perspective the...
protocol developed as a quick hack by Ward Christensen
Ward Christensen
Ward Christensen, born in West Bend, Wisconsin, U.S., is the founder of the CBBS bulletin board, the first bulletin board system ever brought online...
for use in his 1977 MODEM.ASM terminal program. XMODEM became extremely popular in the early bulletin board system
Bulletin board system
A Bulletin Board System, or BBS, is a computer system running software that allows users to connect and log in to the system using a terminal program. Once logged in, a user can perform functions such as uploading and downloading software and data, reading news and bulletins, and exchanging...
(BBS) market, largely because it was so simple to implement. It was also fairly inefficient, and as modem speeds increased this problem led to the development of a number of modified versions of XMODEM to improve performance or address other problems with the protocol. Chuck Forsberg
Chuck Forsberg
Chuck Forsberg developed two data transmission protocols popular for uploading and downloading files from dial-up bulletin board systems in the 1990s. He received a Dvorak Award for Excellence in Telecommunications in 1992 for developing ZMODEM....
collected a number of these into his YMODEM
YMODEM
YMODEM is a protocol for file transfer used between modems. YMODEM was developed by Chuck Forsberg as the successor to XMODEM and MODEM7, and was first implemented in his CP/M YAM program...
protocol, but poor implementation led to a further fracturing before they were re-unified by his later ZMODEM
ZMODEM
ZMODEM is a file transfer protocol developed by Chuck Forsberg in 1986, in a project funded by Telenet in order to improve file transfers on their X.25 network...
protocol.
XMODEM
XMODEM, like most file transfer protocols, breaks up the original data into a series of "packets" that are sent to the receiver, along with additional information allowing the receiver to determine whether that packet was correctly received.Packet structure
The original XMODEM used a 128-byte data packet, the basic block size used on CP/MCP/M
CP/M was a mass-market operating system created for Intel 8080/85 based microcomputers by Gary Kildall of Digital Research, Inc...
floppy disk
Floppy disk
A floppy disk is a disk storage medium composed of a disk of thin and flexible magnetic storage medium, sealed in a rectangular plastic carrier lined with fabric that removes dust particles...
s. The packet was prefixed by a simple 3-byte header containing a <SOH
C0 and C1 control codes
Most character encodings, in addition to representing printable characters, may also represent additional information about the text, such as the position of a cursor, an instruction to start a new line, or a message that the text has been received...
> character, a "block number" from 0-255, and the "inverse" block number—255 minus the block number. Block numbering starts with 1 for the first block sent.
The packet was also suffixed with a single-byte checksum
Checksum
A checksum or hash sum is a fixed-size datum computed from an arbitrary block of digital data for the purpose of detecting accidental errors that may have been introduced during its transmission or storage. The integrity of the data can be checked at any later time by recomputing the checksum and...
of the data bytes. The checksum was the sum of all bytes in the packet modulo
Modulo operation
In computing, the modulo operation finds the remainder of division of one number by another.Given two positive numbers, and , a modulo n can be thought of as the remainder, on division of a by n...
256. The modulo operation was easily computed by discarding all but the eight least significant bit
Least significant bit
In computing, the least significant bit is the bit position in a binary integer giving the units value, that is, determining whether the number is even or odd. The lsb is sometimes referred to as the right-most bit, due to the convention in positional notation of writing less significant digits...
s of the result, or alternatively on an eight bit machine, ignoring arithmetic overflow
Arithmetic overflow
The term arithmetic overflow or simply overflow has the following meanings.# In a computer, the condition that occurs when a calculation produces a result that is greater in magnitude than that which a given register or storage location can store or represent.# In a computer, the amount by which a...
which would produce the same effect automatically. In this way the checksum was restricted to an eight bit quantity which was able to be expressed using a single byte. For example, if the data bytes were 130 and 130, the total of these codes is 260 and the resulting checksum is 4 using this method.
The complete packet was thus 132 bytes long, containing 128 bytes of data, for a total throughput efficiency of about 97%.
The file was marked "complete" with a <EOT
End-of-transmission character
In telecommunication, an end-of-transmission character is a transmission control character. Its intended use is to indicate the conclusion of a transmission that may have included one or more texts and any associated message headings....
> character sent after the last block. This character was not in a packet, but sent alone as a single byte. Since the file length was not sent as part of the protocol, the last packet was padded out with a "known character" that could be dropped. In the original specification this defaulted to
Transfer details
Files were transferred one packet at a time. When received, the packet's checksum was calculated by the receiver and compared to the one received from the sender at the end of the packet. If the two matched, the receiver sent an <ACKAcknowledge character
In telecommunications, an acknowledge character is a transmission control character transmitted by the receiving station as an acknowledgement, i.e...
> message back to the sender, which then sent the next packet in sequence. If there was a problem with the checksum, the receiver instead sent a <NAK
NaK
NaK, or sodium-potassium alloy, an alloy, of potassium , and sodium , is usually liquid at room temperature. Various commercial grades are available. NaK is highly reactive with water and may catch fire when exposed to air, so must be handled with special precautions...
>. If a <NAK
NaK
NaK, or sodium-potassium alloy, an alloy, of potassium , and sodium , is usually liquid at room temperature. Various commercial grades are available. NaK is highly reactive with water and may catch fire when exposed to air, so must be handled with special precautions...
> was received, the sender would re-send the packet, and continued to try several times, normally ten, before aborting the transfer.
A
The block numbers were also examined in a simple way to check for errors. After receiving a packet successfully, the next packet should have a one-higher number. If it instead received the same block number this was not considered serious, it was implied that the
Transfers were sender-driven, the receiver acting passively on data sent to it. However, the transfer was actually started by the receiver sending a single
Minor problems
XMODEM was written for CP/MCP/M
CP/M was a mass-market operating system created for Intel 8080/85 based microcomputers by Gary Kildall of Digital Research, Inc...
machines, and bears several marks of that operating system
Operating system
An operating system is a set of programs that manage computer hardware resources and provide common services for application software. The operating system is the most important type of system software in a computer system...
. Notably, files on CP/M were always multiples of 128 bytes, and their end was marked within a block with the
MS-DOS
MS-DOS is an operating system for x86-based personal computers. It was the most commonly used member of the DOS family of operating systems, and was the main operating system for IBM PC compatible personal computers during the 1980s to the mid 1990s, until it was gradually superseded by operating...
in the early 1980s led to XMODEM having to be updated to notice either a
For some time it was suggested that sending a
Major problems
XMODEM was designed for simplicity, without much knowledge of other file transfer protocols – which were fairly rare anyway. Due to its simplicity, there were a number of very basic errors that could cause a transfer to fail, or worse, result in an incorrect file which went unnoticed by the protocol. Most of this was due to the use of a simple checksum for error correction, which is susceptible to missing errors in the data if two bits are reversed, which can happen with a suitably short burst of noise. Additionally, similar damage to the header or checksum could lead to a failed transfer in cases where the data itself was undamaged.Many authors introduced extensions to XMODEM to address these and other problems. Many asked for these extensions to be included as part of a new XMODEM standard. However, Ward Christensen refused to do this, as it was precisely the lack of these features, and the associated coding needed to support them, that led to XMODEM's widespread use. As he explained:
- It was a quick hack I threw together, very unplanned (like everything I do), to satisfy a personal need to communicate with some other people. ONLY the fact that it was done in 8/77, and that I put it in the public domain immediately, made it become the standard that it is...
- ...People who suggest I make SIGNIFICANT changes to the protocol, such as 'full duplex', 'multiple outstanding blocks', 'multiple destinations', etc etc don't understand that the incredible simplicity of the protocol is one of the reasons it survived.
Batch Transfers
Another problem with XMODEM was that it required the transfer to be user-driven. Typically this meant the user would navigate on the sender's system to select the file they wanted, and then invoke the transfer from their end using a command in their terminal emulator.For automated transfers between two sites, a number of add-ons to the XMODEM protocol were implemented over time. These generally assumed the sender would continue sending file after file, with the receiver attempting to trigger the next file by sending a
MODEM7
MODEM7, also known as MODEM7 batch or Batch XMODEM, was the first known extension of the XMODEM protocol. An XMODEM file transfer starts with the receiver sending a singleTeLink
MODEM7 sent the filename as normal text, which meant it could be corrupted by the same problems that XMODEM was attempting to avoid. This led to the introduction of TeLink by Tom JenningsTom Jennings
Tom Jennings is a Los Angeles-based artist and technician. He is the creator of FidoNet, the first message and file networking system for BBSes...
, author of the original FidoNet
FidoNet
FidoNet is a worldwide computer network that is used for communication between bulletin board systems. It was most popular in the early to mid 1990s, prior to the introduction of easy and affordable access to the Internet...
mailers.
TeLink avoided MODEM7's problems by standardizing a new "zero packet" containing information about the original file. This included the file's name, size, and timestamp
Timestamp
A timestamp is a sequence of characters, denoting the date or time at which a certain event occurred. A timestamp is the time at which an event is recorded by a computer, not the time of the event itself...
, which were placed in a regular 128 byte XMODEM block. Whereas a normal XMODEM transfer would start with the sender sending "block 1", the TeLink header packet was labeled "block 0".
Again, a normal XMODEM implementation would simply discard the packet, the assumption being that the packet number had been corrupted. But this led to a potential time delay if the packet were discarded, as the sender could not be sure it was being
The basic "block 0" system became a standard in the FidoNet community, and was re-used by a number of future protocols like SEAlink
SEAlink
SEAlink is a file transfer protocol that is backward compatible with XMODEM but features a sliding window system for improved throughput. SEAlink was written in 1986 as a part of the SEAdog FidoNet mailer written by System Enhancement Associates, creators of the famous ARC program...
and YMODEM
YMODEM
YMODEM is a protocol for file transfer used between modems. YMODEM was developed by Chuck Forsberg as the successor to XMODEM and MODEM7, and was first implemented in his CP/M YAM program...
.
XMODEM-CRC
The checksum used in the original protocol was extremely simple, and errors within the packet could go unnoticed. This led to the introduction of XMODEM-CRC by John Byrns, which used a 16-bit CRCCyclic redundancy check
A cyclic redundancy check is an error-detecting code commonly used in digital networks and storage devices to detect accidental changes to raw data...
in place of the 8-bit checksum. CRC's encode not only the data in the packet, but its location as well, allowing it to notice the bit-replacement errors that a checksum would miss. Statistically, this made the chance of detecting an error less than 16 bytes long 99.9969%, and even higher for longer data.
XMODEM-CRC was designed to be backwardly compatible with XMODEM. To do this, the receiver simply sent a C (capital C) character instead of a
Unfortunately this attempt at backward compatibility had a downside. Since it was possible that the initial C character would be lost or corrupted, it could not be assumed that the receiver did not support XMODEM-CRC if the first attempt to trigger the transfer failed. The receiver thus tried to start the transfer three times with C, waiting three seconds between each attempt. This meant that if the user selected XMODEM-CRC while attempting to talk to any XMODEM, as it was intended, there was a potential 10 second delay before the transfer started.
To avoid the delay, the sender and receiver would generally list XMODEM-CRC separately from XMODEM, allowing the user to select "basic" XMODEM if the sender didn't explicitly list it. Ironically, any software that did support -CRC in their basic XMODEM transfer, as it was intended, surreptitiously suggested the user should not attempt to use -CRC. To the average user, XMODEM-CRC was essentially a "second protocol", and treated as such.
Higher throughput
Since the XMODEM protocol required the sender to stop and wait for anAs modem speeds increase, the fixed delay needed to send the
A number of new versions of XMODEM were introduced in order to address these problems. Like earlier extensions, these versions tended to be backward-compatible with the original XMODEM, and like those extensions, this led to a further fracturing of the XMODEM landscape in the user's terminal emulator. In the end, dozens of versions of XMODEM would emerge.
SEAlink
One of the first "third party" mailers for the FidoNetFidoNet
FidoNet is a worldwide computer network that is used for communication between bulletin board systems. It was most popular in the early to mid 1990s, prior to the introduction of easy and affordable access to the Internet...
system was SEAdog, written by the same author as the then-popular .arc data compression
Data compression
In computer science and information theory, data compression, source coding or bit-rate reduction is the process of encoding information using fewer bits than the original representation would use....
format. SEAdog included a wide variety of improvements, including SEAlink
SEAlink
SEAlink is a file transfer protocol that is backward compatible with XMODEM but features a sliding window system for improved throughput. SEAlink was written in 1986 as a part of the SEAdog FidoNet mailer written by System Enhancement Associates, creators of the famous ARC program...
, an improved transfer protocol.
SEAlink used a method known as sliding windows to avoid the inter-packet delay. To do this, the protocol did not wait for the
SEAlink later added a number of other improvements, and was generally a useful protocol. However it remained rare, and was typically only found in FidoNet mailers.
XMODEM-1K
Another way to solve the throughput problem is to increase the packet size. Although the fundamental problem of latency remains, the speed at which it becomes a problem is higher. XMODEM-1K with 1024-byte packets was the most popular such solution. In this case, the throughput at 9600 bit/s is 81%, given the same assumptions as above.XMODEM-1K was an expanded version of XMODEM-CRC, which indicated the longer block size in the sender by starting a packet with the
XMODEM-1K was actually one of the many improvements to XMODEM introduced by Chuck Forsberg
Chuck Forsberg
Chuck Forsberg developed two data transmission protocols popular for uploading and downloading files from dial-up bulletin board systems in the 1990s. He received a Dvorak Award for Excellence in Telecommunications in 1992 for developing ZMODEM....
in his YMODEM
YMODEM
YMODEM is a protocol for file transfer used between modems. YMODEM was developed by Chuck Forsberg as the successor to XMODEM and MODEM7, and was first implemented in his CP/M YAM program...
protocol. Forsberg suggested that the various improvements were optional, expecting software authors to implement as many of them as possible. Instead they generally implemented the bare minimum, leading to a profusion of semi-compatible implementations, and eventually, the splitting out of the name "YMODEM" into "XMODEM-1K" and a variety of YMODEMs. Thus XMODEM-1K actually post-dates YMODEM, but remained fairly common anyway.
A backwards compatible extensions of XMODEM with 32k and 64k block lengths was created by Adontec for better performance on high-speed error free connections like ISDN or TCP/IP networks.
Pre-acknowledge
Over reliable (error-free) connections, the receiver could eliminate the latency issue by "pre-acknowledging" the packets. The receiver would already send ACK while the packet was still being transmitted. This effectively breaks error-correction since a packet is always acknowledged regardless of its integrity (which can only be checked after it has been completely received).Since this feature is only an alteration of the receiver-side behaviour, it does not require any changes in the protocol or on the sender's side.
Pre-acknowledge was also possible for YMODEM
YMODEM
YMODEM is a protocol for file transfer used between modems. YMODEM was developed by Chuck Forsberg as the successor to XMODEM and MODEM7, and was first implemented in his CP/M YAM program...
. It was made obsolete by variants such as YMODEM-g or ZMODEM
ZMODEM
ZMODEM is a file transfer protocol developed by Chuck Forsberg in 1986, in a project funded by Telenet in order to improve file transfers on their X.25 network...
.