rsync is a software application and network protocol for Unix-like
Unix-like
A Unix-like operating system is one that behaves in a manner similar to a Unix system, while not necessarily conforming to or being certified to any version of the Single UNIX Specification....
A computer file is a block of arbitrary information, or resource for storing information, which is available to a computer program and is usually based on some kind of durable storage. A file is durable in the sense that it remains available for programs to use after the current program has finished...
In computing, a folder, directory, catalog, or drawer, is a virtual container originally derived from an earlier Object-oriented programming concept by the same name within a digital file system, in which groups of computer files and other folders can be kept and organized.A typical file system may...
from one location to another while minimizing data
Data
The term data refers to qualitative or quantitative attributes of a variable or set of variables. Data are typically the results of measurements and can be the basis of graphs, images, or observations of a set of variables. Data are often viewed as the lowest level of abstraction from which...
Delta encoding is a way of storing or transmitting data in the form of differences between sequential data rather than complete files; more generally this is known as data differencing...
when appropriate. An important feature of rsync not found in most similar programs/protocols is that the mirror
Mirror (computing)
In computing, a mirror is an exact copy of a data set. On the Internet, a mirror site is an exact copy of another Internet site.Mirror sites are most commonly used to provide multiple sources of the same information, and are of particular value as a way of providing reliable access to large downloads...
ing takes place with only one transmission in each direction. rsync can copy or display directory contents and copy files, optionally using compression
Data compression
In computer science and information theory, data compression, source coding or bit-rate reduction is the process of encoding information using fewer bits than the original representation would use....
Recursion is the process of repeating items in a self-similar way. For instance, when the surfaces of two mirrors are exactly parallel with each other the nested images that occur are a form of infinite recursion. The term has a variety of meanings specific to a variety of disciplines ranging from...
In Unix and other multitasking computer operating systems, a daemon is a computer program that runs as a background process, rather than being under the direct control of an interactive user...
The Transmission Control Protocol is one of the core protocols of the Internet Protocol Suite. TCP is one of the two original components of the suite, complementing the Internet Protocol , and therefore the entire suite is commonly referred to as TCP/IP...
In computer networking, a port is an application-specific or process-specific software construct serving as a communications endpoint in a computer's host operating system. A port is associated with an IP address of the host, as well as the type of protocol used for communication...
of 873, serving files in the native rsync protocol or via a remote shell
Shell (computing)
A shell is a piece of software that provides an interface for users of an operating system which provides access to the services of a kernel. However, the term is also applied very loosely to applications and may include any software that is "built around" a particular component, such as web...
The remote shell is a command line computer program that can execute shell commands as another user, and on another computer across a computer network.The remote system to which rsh connects runs the rshd daemon...
Secure Shell is a network protocol for secure data communication, remote shell services or command execution and other secure network services between two networked computers that it connects via a secure channel over an insecure network: a server and a client...
. In the latter case, the rsync client executable must be installed on the remote machine as well as on the local machine.
Released under the GNU General Public License version 3, rsync is free software
Free software
Free software, software libre or libre software is software that can be used, studied, and modified without restriction, and which can be copied and redistributed in modified or unmodified form either without restriction, or with restrictions that only ensure that further recipients can also do...
Andrew "Tridge" Tridgell is an Australian computer programmer best known as the author of and contributor to the Samba file server, and co-inventor of the rsync algorithm....
and Paul Mackerras wrote the original rsync. Tridgell discusses the design, implementation and performance of rsync in chapters 3 through 5 of his Australian National University
Australian National University
The Australian National University is a teaching and research university located in the Australian capital, Canberra.As of 2009, the ANU employs 3,945 administrative staff who teach approximately 10,000 undergraduates, and 7,500 postgraduate students...
Doctor of Philosophy, abbreviated as Ph.D., PhD, D.Phil., or DPhil , in English-speaking countries, is a postgraduate academic degree awarded by universities...
thesis.
rsync was first announced on 19 June 1996. Rsync 3.0 was released on 1 March 2008.
Uses
rsync was originally written as a replacement for rcp
Rcp (Unix)
rcp stands for the Unix 'remote copy' command. It is a command on the Unix operating systems that is used to remotely copy—to copy one or more files from one computer system to another...
Secure Copy or SCP is a means of securely transferring computer files between a local and a remote host or between two remote hosts. It is based on the Secure Shell protocol....
. As such, it has a similar syntax to its parent programs. Like its predecessors, it still requires a source and a destination to be specified, one of which may be remote. Because of the flexibility, speed and scriptability of rsync, it has become a standard Linux utility and is included in all popular Linux distributions. As a result, rsync has been ported to Windows (via Cygwin
Cygwin
Cygwin is a Unix-like environment and command-line interface for Microsoft Windows. Cygwin provides native integration of Windows-based applications, data, and other system resources with applications, software tools, and data of the Unix-like environment...
One of the earliest applications of rsync was to implement mirroring or backup for multiple Unix clients to a central Unix server using rsync/ssh and standard Unix accounts.
Cron is a time-based job scheduler in Unix-like computer operating systems. Cron enables users to schedule jobs to run periodically at certain times or dates...
, one can schedule automated encrypted rsync-based mirroring between multiple hosts and a central server.
Unison is a file synchronization program. It is used for synchronizing files between two directories, either on one computer, or between a computer and another storage device Unison is a file synchronization program. It is used for synchronizing files between two directories, either on one...
File synchronization in computing is the process of ensuring that computer files in two or more locations are updated via certain rules....
program that uses the rsync algorithm. It is used, for example, for synchronizing two normally-identical directories on two computers that are both subject to editing. In other words, when two devices are synchronized, the user can be sure that the most current version of a file is available on both devices, regardless of where it was last modified.
FreeBSD is a free Unix-like operating system descended from AT&T UNIX via BSD UNIX. Although for legal reasons FreeBSD cannot be called “UNIX”, as the direct descendant of BSD UNIX , FreeBSD’s internals and system APIs are UNIX-compliant...
The Apache HTTP Server, commonly referred to as Apache , is web server software notable for playing a key role in the initial growth of the World Wide Web. In 2009 it became the first web server software to surpass the 100 million website milestone...
The preferred (and simplest) way to mirror the PuTTY
PuTTY
PuTTY is a free and open source terminal emulator application which can act as a client for the SSH, Telnet, rlogin, and raw TCP computing protocols and as a serial console client...
In mathematics and computer science, an algorithm is an effective method expressed as a finite list of well-defined instructions for calculating a function. Algorithms are used for calculation, data processing, and automated reasoning...
invented by the Australian computer programmer Andrew Tridgell
Andrew Tridgell
Andrew "Tridge" Tridgell is an Australian computer programmer best known as the author of and contributor to the Samba file server, and co-inventor of the rsync algorithm....
for efficiently transmitting a structure (such as a file) across a communications link when the receiving computer already has a similar, but not identical, version of the same structure.
The recipient splits its copy of the file into fixed-size non-overlapping chunks and computes two checksum
Checksum
A checksum or hash sum is a fixed-size datum computed from an arbitrary block of digital data for the purpose of detecting accidental errors that may have been introduced during its transmission or storage. The integrity of the data can be checked at any later time by recomputing the checksum and...
The MD4 Message-Digest Algorithm is a cryptographic hash function developed by Ronald Rivest in 1990. The digest length is 128 bits. The algorithm has influenced later designs, such as the MD5, SHA-1 and RIPEMD algorithms....
A hash function is any algorithm or subroutine that maps large data sets to smaller data sets, called keys. For example, a single integer can serve as an index to an array...
A rolling hash is a hash function where the input is hashed in a window that moves through the input.A few hash functions allow a rolling hash to be computed very quickly -- the new hash value is rapidly calculated given only the old hash value, the old value removed from the window, and the new...
'. (Version 30 of the protocol, released with rsync version 3.0.0, now uses MD5
MD5
The MD5 Message-Digest Algorithm is a widely used cryptographic hash function that produces a 128-bit hash value. Specified in RFC 1321, MD5 has been employed in a wide variety of security applications, and is also commonly used to check data integrity...
hashes rather than MD4.) It sends these checksums to the sender.
The sender computes the rolling checksum for every chunk of size in its own version of the file, even overlapping chunks. This can be calculated efficiently because of a special property of the rolling checksum: if the rolling checksum of byte
Byte
The byte is a unit of digital information in computing and telecommunications that most commonly consists of eight bits. Historically, a byte was the number of bits used to encode a single character of text in a computer and for this reason it is the basic addressable element in many computer...
s through is , the rolling checksum of bytes through can be computed from , byte , and byte without having to examine the intervening bytes. Thus, if one had already calculated the rolling checksum of bytes 1–25, one could calculate the rolling checksum of bytes 2–26 solely from the previous checksum, and from bytes 1 and 26.
A rolling hash is a hash function where the input is hashed in a window that moves through the input.A few hash functions allow a rolling hash to be computed very quickly -- the new hash value is rapidly calculated given only the old hash value, the old value removed from the window, and the new...
Adler-32 is a checksum algorithm which was invented by Mark Adler in 1995, and is a modification of the Fletcher checksum. Compared to a cyclic redundancy check of the same length, it trades reliability for speed. Adler-32 is more reliable than Fletcher-16, and slightly less reliable than Fletcher-32...
zlib is a software library used for data compression. zlib was written by Jean-Loup Gailly and Mark Adler and is an abstraction of the DEFLATE compression algorithm used in their gzip file compression program. Zlib is also a crucial component of many software platforms including Linux, Mac OS X,...
The Fletcher checksum is an algorithm for computing a position-dependent checksum devised by John G. Fletcher at Lawrence Livermore Labs in the late 1970s. A description of the algorithm and an analysis of the performance characteristics of a particular implementation were published in the IEEE...
.
The sender then compares its rolling checksums with the set sent by the recipient to determine if any matches exist. If they do, it verifies the match by computing the hash for the matching block and by comparing it with the hash for that block sent by the recipient.
The sender then sends the recipient those parts of its file that did not match the recipient's blocks, along with information on where to merge these blocks into the recipient's version. This makes the copies identical. However, there is a small probability that differences between chunks in the sender and recipient are not detected, and thus remains uncorrected. This requires a simultaneous hash collision in MD5 and the rolling checksum. It is possible to generate MD5 collisions, and the rolling checksum is not cryptographically strong, but the chance for this to occur by accident is nevertheless extremely remote. With 128 bits from MD5 plus 32 bits from the rolling checksum, and assuming maximum entropy in these bits, the probability of a hash collision with this combined checksum is 2−(128+32) = 2−160. The actual probability is a few times higher, since good checksums approach maximum output entropy but very rarely achieve it.
If the sender's and recipient's versions of the file have many sections in common, the utility needs to transfer relatively little data to synchronize the files.
While the rsync algorithm forms the heart of the rsync application that essentially optimizes transfers between two computers over TCP/IP, the rsync application supports other key features that aid significantly in data transfers or backup. They include compression and decompression of data block by block using zlib
Zlib
zlib is a software library used for data compression. zlib was written by Jean-Loup Gailly and Mark Adler and is an abstraction of the DEFLATE compression algorithm used in their gzip file compression program. Zlib is also a crucial component of many software platforms including Linux, Mac OS X,...
at sending and receiving ends, and support for protocols such as ssh
Secure Shell
Secure Shell is a network protocol for secure data communication, remote shell services or command execution and other secure network services between two networked computers that it connects via a secure channel over an insecure network: a server and a client...
that enables encrypted transmission of compressed and efficient differential data using rsync algorithm. Instead of ssh, stunnel
Stunnel
Stunnel is an open-source multi-platform computer program, used to provide universal TLS/SSL tunneling service.Stunnel can be used to provide secure encrypted connections for clients or servers that do not speak TLS or SSL natively. It runs on a variety of operating systems , including most...
can also be used to create an encrypted tunnel to secure the data transmitted.
Finally, rsync is capable of limiting the bandwidth consumed during a transfer, a useful feature that few other standard file transfer protocols offer.
Variations
A utility called uses the rsync algorithm to generate delta file
Delta encoding
Delta encoding is a way of storing or transmitting data in the form of differences between sequential data rather than complete files; more generally this is known as data differencing...
s with the difference from file A to file B (like the utility diff
Diff
In computing, diff is a file comparison utility that outputs the differences between two files. It is typically used to show the changes between one version of a file and a former version of the same file. Diff displays the changes made per line for text files. Modern implementations also...
, but in a different delta format). The delta file can then be applied to file A, turning it into file B (similar to the patch
Patch (Unix)
patch is a Unix program that updates text files according to instructions contained in a separate file, called a patch file. The patch file is a text file that consists of a list of differences and is produced by running the related diff program with the original and updated file as arguments...
utility).
Unlike diff, the process of creating a delta file has two steps: first a signature file is created from file A, and then this (relatively small) signature and file B are used to create the delta file. Also unlike diff, rdiff works well with binary file
Binary file
A binary file is a computer file which may contain any type of data, encoded in binary form for computer storage and processing purposes; for example, computer document files containing formatted text...
s.
Using rdiff, a utility called rdiff-backup has been created, capable of maintaining a backup
Backup
In information technology, a backup or the process of backing up is making copies of data which may be used to restore the original after a data loss event. The verb form is back up in two words, whereas the noun is backup....
mirror of a file or directory either locally or remotely over the network, on another server. rdiff-backup stores incremental rdiff deltas with the backup, with which it is possible to recreate any backup point.
duplicity is a variation on rdiff-backup that allows for backups without cooperation from the storage server, as with simple storage services like Amazon S3
Amazon S3
Amazon S3 is an online storage web service offered by Amazon Web Services. Amazon S3 provides storage through web services interfaces...
. It works by generating the hashes for each block in advance, encrypting them, and storing them on the server, then retrieving them when doing an incremental backup. The rest of the data is also stored encrypted for security purposes.
rsyncrypto is a utility to encrypt files in an rsync-friendly fashion. The rsyncrypto algorithm ensures that two almost identical files, when encrypted with rsyncrypto and the same key, will produce almost identical encrypted files. This allows for the low-overhead data transfer achieved by rsync while providing encryption for secure transfer and storage of sensitive data in a remote location.
An alternative to manually scripting rsync is the Free Software (FLOSS) GUI program BackupPC
BackupPC
BackupPC is a free Disk-to-disk backup software suite with a web-based frontend. The cross-platform server will run on any Linux, Solaris, or UNIX based server. No client is necessary, as the server is itself a client for several protocols that are handled by other services native to the client OS...
, which performs automatic scheduled backups to rsync servers.
As of Mac OS X 10.5 and later, there is a special -E or --extended-attributes switch which allows retaining much of the HFS
Hierarchical File System
Hierarchical File System is a file system developed by Apple Inc. for use in computer systems running Mac OS. Originally designed for use on floppy and hard disks, it can also be found on read-only media such as CD-ROMs...
file metadata when syncing between two machines supporting this feature. This is achieved by transmitting the proprietary Resource Fork
Resource fork
The resource fork is a construct of the Mac OS operating system used to store structured data in a file, alongside unstructured data stored within the data fork. A resource fork stores information in a specific form, such as icons, the shapes of windows, definitions of menus and their contents, and...
along with the Data Fork.
Practical applications
rsync can be used as a method to intelligently copy or backup files from one location to another. For example, within the iTunes music library, all music files are located within an artist folder, with an album subdirectory. If you have an external hard drive that serves as a backup of your music, it can be frustratingly slow to go through the recent additions of (e.g., a new album within an old artist) music on your computer, and to make sure that they are backed up on the external hard drive. If you were to just copy the artist folders, it would disregard the subdirectories, asking you to replace the entire directory. However, rsync can be used to scan all of the files in your music library, as well as the subdirectories, and to add only the ones that are not present on the external hard drive.
BackupAssist is a backup software product created for use with the Microsoft Windows environments. Developed by Cortex I.T Labs, it is aimed primarily at the small to medium business market and provides protection for Windows Server, Hyper-V, Active Directory, Microsoft Exchange and Microsoft SQL...
luckyBackup is a powerful, fast and reliable free backup application.It provides a GUI based on the cross-platform Qt framework and is not fundamentally console based or web based as many of the clients from the list of backup software are. It shares the data differencing and copying tool, rsync,...
Grsync is a Graphical User Interface for the rsync synchronization tool under Linux / Unix System. There is also a port of Grsync on Windows platform...
- Summary :It provides backup scheduling based on time as well as on resource availability. For example plugging in a USB drive can trigger a backup, if the root of the USB drive contains a QtdSync file....
Duplicati is a software suite that provides easy encrypted, versioned, remote backup of files requiring little of the remote server. Duplicati enables the user to create full and incremental backups allowing recovery of data at any of the backup times...
Remote Differential Compression is a client–server synchronization algorithm that allows the contents of two files to be synchronized by communicating only the differences between them...
Unison is a file synchronization program. It is used for synchronizing files between two directories, either on one computer, or between a computer and another storage device Unison is a file synchronization program. It is used for synchronizing files between two directories, either on one...
in its own version of the file, even overlapping chunks. This can be calculated efficiently because of a special property of the rolling checksum: if the rolling checksum of byte
through 
is 
, the rolling checksum of bytes 
through 
can be computed from 
, byte 
, and byte 
without having to examine the intervening bytes. Thus, if one had already calculated the rolling checksum of bytes 1–25, one could calculate the rolling checksum of bytes 2–26 solely from the previous checksum, and from bytes 1 and 26.