Maildir
Encyclopedia
The Maildir e-mail
format is a common way of storing e-mail messages, where each message is kept in a separate file
with a unique name, and each folder is a directory
. The local filesystem handles file locking
as messages are added, moved and deleted.
(often named
, the author of qmail
, djbdns
, and other software. Although the original specification was written specifically for Bernstein's qmail
, it is general enough to be implemented in many programs.
and other software, wrote an extension to the Maildir format called Maildir++ to support subfolders and mail quotas. Maildir++ directories contain subdirectories with names that start with a '.' (dot) that are also Maildir++ folders. This extension is therefore a violation of the Maildir specification, which provides an exhaustive list of the possible contents of a Maildir, however it is a compatible violation and other Maildir software supports Maildir++.
The delivery process stores the message in the maildir by creating and writing to
When the mail user agent process finds messages in the
(to separate the unique part of the filename from the actual information), a '2', a comma
and various flag
s. The '2' specifies, loosely speaking
, the version of the information that follows the comma. '2' is the only currently officially specified version, '1' being an experimental version. One can only assume that it was used while the Maildir format was under development. The specification defines flags which show whether the message has been read, deleted and so on: the initial (capital) letter of Passed, Replied, Seen, Trashed, Draft, and Flagged. Dovecot
uses lowercase letters to match 26 IMAP keywords, which may include standardised keywords such as $MDNSent, and user defined flags.
designed Maildir to be safely writable by multiple concurrent writers without any form of explicit locking, even over NFS. This works reasonably well in practice, but can result in strangeness. During directory listing, any files that are renamed after the first
, for example, uses its own non-standard locking with Maildir.
Maildir thus suffers from the inherent scaling limitations of any one-email-per-file email storage design.
and some configurations of Novell Storage Services
.
Software running on these systems can use an alternative separator (such as ";", or "-"), and it is often trivial to patch free and open source software
to use a different separator.
As there is currently no agreement on what character this alternative separator should be, there can be interoperability difficulties between different Maildir-supporting programs on these systems. However, not all Maildir-related software needs to know what the separator character is, because not all Maildir-related software needs to be able to read or modify the flags of a message ("read", "replied to" etc.); software that merely delivers to a Maildir, or archives old messages from it based only on date, should work no matter what separator is in use. If only the MUA needs to read or modify message flags, and only one is used, then non-standard alternative separators may be used without interoperability problems.
For example:
E-mail
Electronic mail, commonly known as email or e-mail, is a method of exchanging digital messages from an author to one or more recipients. Modern email operates across the Internet or other computer networks. Some early email systems required that the author and the recipient both be online at the...
format is a common way of storing e-mail messages, where each message is kept in a separate file
Computer file
A computer file is a block of arbitrary information, or resource for storing information, which is available to a computer program and is usually based on some kind of durable storage. A file is durable in the sense that it remains available for programs to use after the current program has finished...
with a unique name, and each folder is a directory
Directory (file systems)
In computing, a folder, directory, catalog, or drawer, is a virtual container originally derived from an earlier Object-oriented programming concept by the same name within a digital file system, in which groups of computer files and other folders can be kept and organized.A typical file system may...
. The local filesystem handles file locking
File locking
File locking is a mechanism that restricts access to a computer file by allowing only one user or process access at any specific time. Systems implement locking to prevent the classic interceding update scenario ....
as messages are added, moved and deleted.
Specifications
A Maildir directoryDirectory (file systems)
In computing, a folder, directory, catalog, or drawer, is a virtual container originally derived from an earlier Object-oriented programming concept by the same name within a digital file system, in which groups of computer files and other folders can be kept and organized.A typical file system may...
(often named
Maildir) usually has three subdirectories named tmp, new, and cur.Maildir
The original Maildir specification was written by Daniel J. BernsteinDaniel J. Bernstein
Daniel Julius Bernstein is a mathematician, cryptologist, programmer, and professor of mathematics at the University of Illinois at Chicago...
, the author of qmail
Qmail
qmail is a mail transfer agent that runs on Unix. It was written, starting December 1995, by Daniel J. Bernstein as a more secure replacement for the popular Sendmail program...
, djbdns
Djbdns
The djbdns software package is a DNS implementation created by Daniel J. Bernstein due to his frustrations with repeated BIND security holes. A $1000 prize for the first person to find a privilege escalation security hole in djbdns was awarded in March 2009 to Matthew Dempsky., djbdns's tinydns...
, and other software. Although the original specification was written specifically for Bernstein's qmail
Qmail
qmail is a mail transfer agent that runs on Unix. It was written, starting December 1995, by Daniel J. Bernstein as a more secure replacement for the popular Sendmail program...
, it is general enough to be implemented in many programs.
Maildir++
Sam Varshavchik, the author of the Courier Mail ServerCourier Mail Server
The Courier mail server is a mail transfer agent server that provides ESMTP, IMAP, POP3, SMAP, webmail, and mailing list services with individual components. It is best known for its IMAP server component....
and other software, wrote an extension to the Maildir format called Maildir++ to support subfolders and mail quotas. Maildir++ directories contain subdirectories with names that start with a '.' (dot) that are also Maildir++ folders. This extension is therefore a violation of the Maildir specification, which provides an exhaustive list of the possible contents of a Maildir, however it is a compatible violation and other Maildir software supports Maildir++.
Technical operation
The process that delivers an e-mail message writes it to a file in thetmp directory with a unique filename. The current algorithm for generating the unique filename combines the time, the host name, and a number of pseudo-random parameters to ensure uniqueness.The delivery process stores the message in the maildir by creating and writing to
tmp/unique, and then moving this file to new/unique. The moving is commonly done by hard linking the file to new and then unlinking the file from tmp, but some implementations simply rename it there. This sequence guarantees that a maildir-reading program will not see a partially-written message, as MUAs never look in tmp.When the mail user agent process finds messages in the
new directory it moves them to cur (using rename - link then unlink strategy may result in having the message duplicated) and appends an informational suffix to the filename before reading them. The information suffix consists of a colonColon (punctuation)
The colon is a punctuation mark consisting of two equally sized dots centered on the same vertical line.-Usage:A colon informs the reader that what follows the mark proves, explains, or lists elements of what preceded the mark....
(to separate the unique part of the filename from the actual information), a '2', a comma
Comma
A comma is a type of punctuation mark . The word comes from the Greek komma , which means something cut off or a short clause.Comma may also refer to:* Comma , a type of interval in music theory...
and various flag
Flag (computing)
In computer programming, flag can refer to one or more bits that are used to store a binary value or code that has an assigned meaning, but can refer to uses of other data types...
s. The '2' specifies, loosely speaking
Handwaving
Handwaving is a pejorative label applied to the action of displaying the appearance of doing something, when actually doing little, or nothing. For example, it is applied to debate techniques that involve logical fallacies. It is also used in working situations where productive work is expected,...
, the version of the information that follows the comma. '2' is the only currently officially specified version, '1' being an experimental version. One can only assume that it was used while the Maildir format was under development. The specification defines flags which show whether the message has been read, deleted and so on: the initial (capital) letter of Passed, Replied, Seen, Trashed, Draft, and Flagged. Dovecot
Dovecot (software)
Dovecot is an open source IMAP and POP3 server for Linux/UNIX-like systems, written primarily with security in mind. Developed by Timo Sirainen, Dovecot was first released in July 2002...
uses lowercase letters to match 26 IMAP keywords, which may include standardised keywords such as $MDNSent, and user defined flags.
Inconsistent state with lockless operation
Daniel J. BernsteinDaniel J. Bernstein
Daniel Julius Bernstein is a mathematician, cryptologist, programmer, and professor of mathematics at the University of Illinois at Chicago...
designed Maildir to be safely writable by multiple concurrent writers without any form of explicit locking, even over NFS. This works reasonably well in practice, but can result in strangeness. During directory listing, any files that are renamed after the first
readdir system call and before the last readdir call may not appear in the listing. This causes the listing process to believe that the message was deleted, while in reality only its flags were changed. When the process lists the messages again, the "deleted" message suddenly reappears. Some mail-accessing programs layer their own locking on top of Maildir in an attempt to prevent this kind of problem. DovecotDovecot (software)
Dovecot is an open source IMAP and POP3 server for Linux/UNIX-like systems, written primarily with security in mind. Developed by Timo Sirainen, Dovecot was first released in July 2002...
, for example, uses its own non-standard locking with Maildir.
Locking & scaling
There remains the implicit locking used by the filesystem in updating the directories; non-cluster filesystems will typically only allow one kernel thread at a time to be updating what is in a directory, so therename system call will provide the locking needed. Maildir is not lock-free, only explicit-lock-free. For many small to medium mail-systems, this scales adequately even over NFS, but as you grow to large systems handling many concurrent deliveries, the constant changing of the directory contents across multiple directories will result in repeated cache invalidation of the different NFS clients, so READDIR RPCs will have to re-issued, scaling poorly. Further, many filesystems have per-directory file-count limits.Maildir thus suffers from the inherent scaling limitations of any one-email-per-file email storage design.
Filesystem compatibility
The Maildir standard cannot be implemented without modification on systems that do not accept colons in filenames. This includes Microsoft WindowsMicrosoft Windows
Microsoft Windows is a series of operating systems produced by Microsoft.Microsoft introduced an operating environment named Windows on November 20, 1985 as an add-on to MS-DOS in response to the growing interest in graphical user interfaces . Microsoft Windows came to dominate the world's personal...
and some configurations of Novell Storage Services
Novell Storage Services
Novell Storage Services is a file system used by the Novell NetWare operating system. Recently support of NSS was introduced to SUSE Linux via low-level network NCPFS protocol...
.
Software running on these systems can use an alternative separator (such as ";", or "-"), and it is often trivial to patch free and open source software
Free and open source software
Free and open-source software or free/libre/open-source software is software that is liberally licensed to grant users the right to use, study, change, and improve its design through the availability of its source code...
to use a different separator.
As there is currently no agreement on what character this alternative separator should be, there can be interoperability difficulties between different Maildir-supporting programs on these systems. However, not all Maildir-related software needs to know what the separator character is, because not all Maildir-related software needs to be able to read or modify the flags of a message ("read", "replied to" etc.); software that merely delivers to a Maildir, or archives old messages from it based only on date, should work no matter what separator is in use. If only the MUA needs to read or modify message flags, and only one is used, then non-standard alternative separators may be used without interoperability problems.
Mail servers
- bincimap IMAP server
- DovecotDovecot (software)Dovecot is an open source IMAP and POP3 server for Linux/UNIX-like systems, written primarily with security in mind. Developed by Timo Sirainen, Dovecot was first released in July 2002...
IMAP server - Courier Mail ServerCourier Mail ServerThe Courier mail server is a mail transfer agent server that provides ESMTP, IMAP, POP3, SMAP, webmail, and mailing list services with individual components. It is best known for its IMAP server component....
SMTP and IMAP server, for which the Maildir++ format was invented - EximEximExim is a mail transfer agent used on Unix-like operating systems. Exim is free software distributed under the terms of the GNU General Public License, and it aims to be a general and flexible mailer with extensive facilities for checking incoming e-mail....
SMTP server - PostfixPostfix (software)In computing, Postfix is a free and open-source mail transfer agent that routes and delivers electronic mail. It is intended as a fast, easier-to-administer, and secure alternative to the widely-used Sendmail MTA....
SMTP server - QmailQmailqmail is a mail transfer agent that runs on Unix. It was written, starting December 1995, by Daniel J. Bernstein as a more secure replacement for the popular Sendmail program...
SMTP server, for which Maildir format was invented - XMailXMailXMail is a free open-source, developed by Davide Libenzi, POP3/ESMTP mail server for Unix and Microsoft Windows operating systems. Unlike other email servers, XMail does not require that each user have a system account, instead it uses flat-file databases to store user information...
, an open sourceOpen sourceThe term open source describes practices in production and development that promote access to the end product's source materials. Some consider open source a philosophy, others consider it a pragmatic methodology...
cross-platform (*nix and WindowsMicrosoft WindowsMicrosoft Windows is a series of operating systems produced by Microsoft.Microsoft introduced an operating environment named Windows on November 20, 1985 as an add-on to MS-DOS in response to the growing interest in graphical user interfaces . Microsoft Windows came to dominate the world's personal...
) SMTP and POP3 server - MeTA1 SMTP server
Delivery agents
- ProcmailProcmailprocmail is a mail delivery agent capable of sorting incoming mail into various directories and filtering out spam messages. Procmail is widely used on Unix-based systems and stable, but no longer maintained; users who wish a maintained program are advised to use an alternative MDA, such as...
- MaildropMaildropmaildrop is a Mail delivery agent used by the Courier Mail Server. The maildrop MDA also includes filtering functionality.maildrop receives mail via stdin and delivers in both Maildir and mbox formats....
- GetmailGetmailgetmail is a simple mail retrieval agent intended as a replacement for fetchmail, implemented in Python. It can retrieve mail from POP3, IMAP4 and Standard Dial-up POP3 Service servers, with or without SSL...
, a Maildir-aware mail-retrieval and delivery agent alternative to FetchmailFetchmailFetchmail is an open source software utility for POSIX-compliant operating systems which is used to retrieve e-mail from a remote POP3, IMAP, ETRN or ODMR mail server to the user's local system. It was developed from the popclient program, written by Carl Harris.Its chief significance is perhaps... - mbsync
- OfflineIMAP
- mswatch
- mpop
Mail readers
- BalsaBalsa (e-mail client)Balsa is a lightweight email client for GNOME.Balsa has a graphical front end, support for MIME attachments coming and going, directly supports POP3 and IMAP protocols. It has a spell checker and direct support for PGP and GPG for encryption. It has some basic filtering capabilities, and natively...
previously the official GNOME mail reader (prior to Evolution) - ConeCone (software)Cone is a text-based e-mail client and news client for Unix-like operating systems. It is developed by the Courier Mail Server developers. Its name stands for "console newsreader and emailer"....
a curses-based mail reader - GnusGnusGnus is a message reader running under GNU Emacs and XEmacs. It supports reading and composing both news and e-mail.Gnus blurs the distinction between news and e-mail, treating them both as "articles" that come from different sources. News articles are kept separate by group, and e-mail can be...
- mailxMailxmailx is a Unix utility program for sending and receiving mail, also known as a Mail User Agent program. It is an improved version of the mail utility.mailx is a lightweight console application mail program which has a command syntax similar to ed....
- GNUMailGNUMailGNUMail is a free and open source, cross-platform mail client for GNUstep and Cocoa. It is the official mail client of GNUstep and is also used in Étoilé. It was inspired by NeXTMail , the prime father of Apple Mail...
- KMail, KDE mail reader
- MuttMutt (e-mail client)Mutt is a text-based email client for Unix-like systems. It was originally written by Michael Elkins in 1995 and released under the GNU General Public License version 2 or any later version....
- Evolution, official GNOME mail client
- Wanderlust
Mail index and search tools
- Beagle (software)Beagle (software)Beagle is a search system for Linux and other such modern Unix-like systems, enabling the user to search documents, chat logs, email and contact lists in a similar way to Spotlight in Mac OS X, and Windows Search or Google Desktop under Linux or Microsoft Windows...
can index Maildirs and many other information storage formats - Mairix is a program for indexing and searching email messages stored in maildir, MH or mbox folders
- notmuch is a program for indexing and searching email messages stored in maildir
Software that supports Maildir by implication
The list of software that can be used with Maildir is in fact much larger if you consider how this software can be plugged together, and the role of network access protocols.For example:
- The SendmailSendmailSendmail is a general purpose internetwork email routing facility that supports many kinds of mail-transfer and -delivery methods, including the Simple Mail Transfer Protocol used for email transport over the Internet....
MTA does not support any mail delivery format (although many assume that it does). Sendmail uses a separate delivery process calledmail.local. Procmail (and other programs that support Maildir) can be used in place ofmail.local, so Sendmail can rightly be said to support Maildir as much as it supports any other format. - Many mail readers do not support Maildir but do support remote access formats such as IMAP. Since there are several IMAP mail stores that support Maildir, any mail reader that supports IMAP such as Microsoft OutlookMicrosoft OutlookMicrosoft Outlook is a personal information manager from Microsoft, available both as a separate application as well as a part of the Microsoft Office suite...
, PinePine (e-mail client)Pine is a freeware, text-based email client developed at the University of Washington. The first version of this client was written in 1989. Source code was available for only the Unix version under a license written by the University of Washington...
, or Mozilla ThunderbirdMozilla ThunderbirdMozilla Thunderbird is a free, open source, cross-platform e-mail and news client developed by the Mozilla Foundation. The project strategy is modeled after Mozilla Firefox, a project aimed at creating a web browser...
can be used to access Maildir folders. - FetchmailFetchmailFetchmail is an open source software utility for POSIX-compliant operating systems which is used to retrieve e-mail from a remote POP3, IMAP, ETRN or ODMR mail server to the user's local system. It was developed from the popclient program, written by Carl Harris.Its chief significance is perhaps...
does not support Maildir (or any local delivery format) but since it talks to an SMTP server or local delivery agent, any of those listed above can be used to deliver mail from Fetchmail to Maildirs.