Cyrus IMAP HOWTO

IMAP allows the user to read email from many different locations and accounts because email folders are stored on the server, locally, at the home or the office, for instance. Even saved and read messages are stored on the IMAP server. POP only stores new unread messages on the server, and the read and saved messages are stored locally. To force POP to not delete email once it has been read can create many copies of the same email, creating a waste of space and confusion. However, IMAP is usually more complicated to set up. If you only have one email account, POP is most likely your best choice. If you want to access more than one account, from different locations, IMAP will probably serve you most efficiently.

Cyrus IMAP provides a robust, scalable mail architecture. It provides client access through IMAP and POP3, the two most common mail presentation protocols defined by open standards. Through the use of the SASL authentication, it supports a variety of authentication mechanisms and back-ends. It supports wrapping IMAP and POP3 in SSL and negotiating TLS in-channel. It supports multiple storage pools for mailboxes, so adding mailbox space is relatively straight-forward.

The mail spool is typically located in /var/spool/mail. This one fact remains fairly consistent. Below here, there may be three different sorts of layouts, depending the selected level of mailbox hashing. If hashimapspool is zero, then there will just be directories here that map directly to mailboxes based on the first letter of the mailbox name. For example, a mailbox for user "wcooley" will be located at /var/spool/mail/w/user/wcooley. For very large installations, however, this tends to result in a disproporationate number of mailboxes under "s" compared with "z."

If hashimapspool is 1 or true, then the effect of fulldirhash comes into play. If it is not set or zero, then mailboxes are set by the username. For example, for user "wcooley", the mailbox is located in /var/spool/mail/w/user/wcooley. However, if fulldirhash is set to 1 or true, then the mailbox directories are hashed with a true hash function to a prime number "buckets," which are represented as a single character. For example, /var/spool/imap/I/user/wcooley. The use of a true hash function provides nearly even distribution of mailboxes in into directories.

To provide a consistent lookup mechanism for mailbox names, the mbpath program is provided to allow a consistent interface for determining the actual mailbox path without having to account for the hashing yourself.

The dohash and undohash scripts purport to convert between the three levels of mailbox hashing. However, they may not work and are not terribly well documented, so it's probably best to choose the correct one from the beginning. Unless you plan on mucking about with the mailboxes (which you probably should not do), there is no harm in using the full hashing from the get-go.

Metadata for the mailstore is stored by default in /var/imap, although for FHS compliance, packages tend to use /var/lib/imap. (FIXME)

Configuration is stored in /etc/imapd.conf and /etc/cyrus.conf. (FIXME)

Binaries could be nearly anywhere, from /usr/cyrus to /usr/lib/cyrus-imapd, with programs intended to be used directly (such as cyradm) linked to from /usr/bin. (FIXME)

The master process acts as the controller for all of the Cyrus sub-servers. It performs tasks similar to those performed by cron, inetd, and init scripts, although only for Cyrus processes--it is not a system-wide replacement for any of these. It is configured through the cyrus.conf file, which has three sections: START, SERVICES, and EVENTS.

Each section has entries which are restricted to a single line and composed of a tag name and parameters. The tag name should be unique and the exact parameters are specific to the particular section. When Cyrus is built with support for TCP Wrappers, the tag name is used as the daemon name for hosts.allow and hosts.deny.

All sections have a cmd, which lists the program and parameters to be used when the entry is invoked. This entry is relative to the --with-cyrus-prefix supplied at build time.

The START section contains commands that are run then master is started, much like an init script. Usually, this section contains nothing more than a recover section, for checking the Cyrus databases before starting processes that use those databases. If your system is configured to use idled, it is also started here, because it does not directly listen on a network or UNIX socket.

The following is a minimal listing of the START section (taken from the example master/conf/normal.conf from the source distribution).

START {
  # do not delete this entry!
  recover       cmd="ctl_cyrusdb -r"
                                                                                                                                                                     
  # this is only necessary if using idled for IMAP IDLE
#  idled                cmd="idled"
}

The SERVICES section contains entries for the various sub-servers that run under the system and listen on network sockets. It is very much like inetd in this respect.

The listen parameter lists the TCP or UDP port or UNIX domain socket to be listened-on for the particular service. UNIX domain sockets are paths in the filesystem and are differentiated from ports by starting with a slash (/). For TCP and UDP sockets, you can give either the actual port number or the service name, as resolved through /etc/services or other NSS means. TCP is the default protocol, although UDP can be used with the proto parameter. prefork and maxchild provide control over the number of processes initially forked to help manage load and the maximum number of child processes to be started, to prevent too many processes from overloading the system. The maxfds parameter limits the number of open file handles that the each process my have open.

The following is an example listing of the SERVICES (taken from the example master/conf/normal.conf from the source distribution).

# UNIX sockets start with a slash and are put into /var/imap/socket
SERVICES {
  # add or remove based on preferences
  imap          cmd="imapd" listen="imap" prefork=0
  imaps         cmd="imapd -s" listen="imaps" prefork=0
  pop3          cmd="pop3d" listen="pop3" prefork=0
  pop3s         cmd="pop3d -s" listen="pop3s" prefork=0
  sieve         cmd="timsieved" listen="sieve" prefork=0
                                                                                                                                                                     
  # these are only necessary if receiving/exporting usenet via NNTP
#  nntp         cmd="nntpd" listen="nntp" prefork=0
#  nntps                cmd="nntpd -s" listen="nntps" prefork=0
                                                                                                                                                                     
  # at least one LMTP is required for delivery
#  lmtp         cmd="lmtpd" listen="lmtp" prefork=0
  lmtpunix      cmd="lmtpd" listen="/var/imap/socket/lmtp" prefork=0
                                                                                                                                                                     
  # this is only necessary if using notifications
#  notify       cmd="notifyd" listen="/var/imap/socket/notify" proto="udp" prefork=1
}

The EVENTS section is similar to cron in that it schedules commands to be run. Generally, these commands are for maintenance of the various databases Cyrus uses.

This section has a period, which is an interval in minutes to run the listed command, and an at, which lists a time in 24-hour format to run the listed command.

The following is an example listing from the same master/conf/normal.conf.

EVENTS {
  # this is required
  checkpoint    cmd="ctl_cyrusdb -c" period=30
 
  # this is only necessary if using duplicate delivery suppression,
  # Sieve or NNTP
  delprune      cmd="cyr_expire -E 3" at=0400
 
  # this is only necessary if caching TLS sessions
  tlsprune      cmd="tls_prune" at=0400
}

As you might have guessed, the imapd sub-server provides IMAP access to the mail store to IMAP clients. It can also provide SSL-wrapped IMAP with the -s parameter.

The pop3d sub-server provides POP3 access to the mail store. It can also provide SSL-wrapped IMAP with the -s parameter.

The timsieved sub-server provides the network service for managing Sieve scripts installed on the server.

The lmtpd sub-server receives deliveries into the mail store, either through TCP/IP or a UNIX domain socket.

The notifyd sub-server provides an interface sending notifications. I've never actually seen this in use and am not sure what it's purpose is, because the docs are a little sketchy. (FIXME).

The fud sub-server provides an interface for client access to information about the mailbox, such as new mail count, total message count, and recent message count. I've not really seen this in use. (FIXME).

Unlike the other sub-servers, idled is run in the START section rather than the SERVICE section. idled can monitor a mailbox and signal to listening imapd and pop3d that new mail has arrived.

(New with 2.2.) Provides an NNTP interface for shared mailboxes. (FIXME)

Provides murder update support.

Provides a Sendmail socket map daemon which can be used by Sendmail to verify that a recipient exists before accepting the message.

The abritron process provides readership statistics for mailboxes. It does not seem to be built by default. It is unclear how commonly used or well-maintained it is.

Sorts output from arbitron.

The cyradm command is a Perl script which allows you to manage mailboxes.

(Guess???) The cyrdump command dumps mailboxes to a flat-text to standard output, saving seen flags and other state. (NB: There appears to be no reverse command.)

Expires entries from the duplicate delivery database. Can also be used to control automatic expiration of messages based on mailbox annotations.

Reads articles from an NNTP peer and delivers them into Cyrus.

The chk_cyrus command performs consistency checks on Cyrus mailstore.

The ctl_cyrusdb command performs maintenance on the Cyrus databases, such as recovering, check-pointing, and archiving.

The ctl_deliver command performs maintenance on the Cyrus duplicate delivery database.

The ctl_mboxlist command performs maintenance on the Cyrus mailbox list.

The cvt_cyrusdb command is used convert between the various database formats used by Cyrus. It can be used, for example, to perform a dump of data from binary databases to plain-text flat-file databases, which can be more reliably backed-up.

The deliver command used to be the point of entry for mail coming from the mail transfer agent (Sendmail, Postfix, etc). It reads messages from standard input and delivers into the system. However, with the implementation of LMTP, it is considered obsolete. In fact, it is just an LMTP client which delivers to lmtpd. It should not be used by the MTA to deliver mail into the system unless the MTA does not support LMTP. It can still be useful, however, with external programs such as procmail, which does not operate as an LMTP client. (FIXME: procmail has an option now to be an LMTP client.) The -q may still be useful to post over-quota alerts to users' mailboxes.

The dohash utility is a Perl script which can be used to migrate from older directory layouts and to implement spool directory hashing.

The ipurge utility expires and deletes messages from mailboxes or partitions. It can be configured purge based on message age or mailbox size.

mbpath translates a mailbox name into a filesystem path, taking into consideration spool hashing and partitions.

The mkimap command is a Perl script which creates the directory heirarchies requires by Cyrus. It is generally only needed when installing from source.

(Guess???) Creates newsgroup heirarchies from news "active" files.

Not to be confused with the quota command used for viewing filesystem quotas, this command reports on quota usage and can also be used to repair inconsistencies. (NB: The Cyrus quota command is in section 8 of the man pages; the filesystem quota is in section 1.)

The rehash script can be used to convert between spool hash types: none, basic and full.

The reconstruct command is used to rebuild mailboxes, which might have suffered database corruption or restored from backup. Note that, despite the fact that the man page has listed a -m option to rebuild the master mailbox list, it has never actually worked. It is important, therefore, to maintain a backup of the mailbox list, perferably as plain-text.

The squatter utility may be one of the most overlooked utilities in the Cyrus distribution. It builds full-text mailbox indexes which allow very fast server-side searching.

tls_prune removed expired TLS sessions from the session database.

The translatesieve command is used to update Sieve scripts when implementing alternative namespaces.

The undohash command flattens directory heirarchies which had been previously hashed.

This command migrates the timsieved storage from older versions, which used to use a hardlink to indicate the active script but now uses a symlink.

User mailboxes live under 'user.<username>'. To the user, this mailbox is called simply INBOX; sub-folders such as 'user.<username>.Sent', appear to the user as 'INBOX.Sent' (NB: altnamespace).

Cyrus IMAP's mailbox storage is not Maildir, although it is similar in some respects. You should not expect to access the mailboxes directly as you might other servers; only access messages through POP3 or IMAP.

The mailbox storage does use databases, but it cannot use SQL. People occasionally ask about using SQL, but it does not seem to be an appropriate use of SQL; the data is not especially relational. The data is hashed for fast lookups, which is better done by file-based databases when data is not relational. Most people who are ask about SQL storage have high-availability in mind, which is easy with certain databases (like MySQL); most highly-available Cyrus implementations use SANs and clustered filesystems.

Mailboxes have to be created before being used, either with 'cyradm' or another tool (NB: autocreate patches).

Deleting a message in IMAP requires two steps: First, set the \Deleted flag on the message. Second, EXPUNGE the mailbox. Many MUAs first move messages to a Trash folder, but given the recoverability inherent in a two-step deletion, it is generally considered a design flaw of the MUAs.

To configure Cyrus to receive message via LTMP, make sure the lmtpunix line in cyrus.conf is uncommented and looks something like the following:

SERVICES {
# ...
  lmtpunix      cmd="lmtpd" listen="/var/lib/imap/socket/lmtp" prefork=0
# ...
}

On the Postfix side, ensure that the lmtp service in master.cf is not chrooted:

# ==========================================================================
# service type  private unpriv  chroot  wakeup  maxproc command + args
#               (yes)   (yes)   (yes)   (never) (50)
# ==========================================================================
lmtp      unix  -       -       n       -       -       lmtp

and set the mailbox_transport to the LMTP socket listed in cyrus.conf:

mailbox_transport = lmtp:unix:/var/lib/imap/socket/lmtp

FIXME

The cyradm command is the main tool that Cyrus administrators use. It allows an administrator to create, delete and rename mailboxes and manage quotas and ACLs.

Assuming that your system is configured, running cyradm should simply be a matter of typing the command, although it is often located in an out-of-the-PATH directory, so you might need the full path. Inevitably, a person will have problems authenticating the first time because he has neglected to limit the advertised SASL mechanisms and cyradm chooses a stronger mechanism than he has support for. If you are unable to login, specify the --auth option with "login". You might also not be working as cyrus, so the "--user cyrus" option may also be necessary. Of course, the IMAP server itself must'nt be forgotten, even if it is only "localhost". So that simple command of starting cyradm ends up being something like this:

$ cyradm --user cyrus --auth login localhost

Upon successfully launching cyradm, you will be presented with a prompt inviting you to start mucking about with things. A good place to start is, of course, "help."

$ cyradm --user cyrus --auth login localhost
IMAP Password:localhost.localdomain> help
authenticate, login, auth         authenticate to server
chdir, cd                         change current directory
createmailbox, create, cm         create mailbox
deleteaclmailbox, deleteacl, dam  remove ACLs from mailbox
deletemailbox, delete, dm         delete mailbox
disconnect, disc                  disconnect from current server
exit, quit                        exit cyradm
help, ?                           show commands
info                              display mailbox/server metadata
listacl, lam, listaclmailbox      list ACLs on mailbox
listmailbox, lm                   list mailboxes
listquota, lq                     list quotas on specified root
listquotaroot, lqr, lqm           show quota roots and quotas for mailbox
mboxcfg, mboxconfig               configure mailbox
reconstruct                       reconstruct mailbox (if supported)
renamemailbox, rename, renm       rename (and optionally relocate) mailbox
server, servername, connect       show current server or connect to server
setaclmailbox, sam, setacl        set ACLs on mailbox
setinfo                           set server metadata
setquota, sq                      set quota on mailbox or resource
version, ver                      display version info of current server
xfermailbox, xfer                 transfer (relocate) a mailbox to a different server

You may, of course, use any SASL mechanism you have configured instead of "login", but login tends to work more readily than more sophisticated mechanisms. cyradm need not be run on localhost either. The administration protocol is integrated in the IMAP protocol itself, so the tool has a level of network transparency. It does not, however, support SSL/TLS, which is why using it on localhost is a good idead.

The command shell will pass any command it does not recognize to your regular shell, which may be confusing at first.

Resist the temptation to give administrative rights to accounts other than dedicated accounts--logging in with administrative rights with a regular mail client may cause unpredictable results. It is not out of sheer laziness that all of my examples use "cyrus" as the admin user.

Listing users is done with the 'listmailbox' command, usually run as 'lm'. There are not a lot of options with 'lm', except in the wildcards that may be used. As with shell wildcards, '*' matches anything; '%', on the other hand, matches anything except the hierarchy separator. This allows only listing the top-level set of folders. For example, let's list all of the folders:

localhost.localdomain> lm
news.announce (\HasNoChildren)  user.foo.bar (\HasNoChildren)   
user.foo (\HasChildren)         user.wcooley (\HasNoChildren)   

Now let's list only users' folders:

localhost.localdomain> lm user.*
user.foo (\HasChildren)        user.wcooley (\HasNoChildren)  
user.foo.bar (\HasNoChildren)  

Now, let's list only the top-level user folders:

localhost.localdomain> lm user.%
user.foo (\HasChildren)        user.wcooley (\HasNoChildren)  

In order for a user to receive e-mail with Cyrus, his mailbox must first be created. This is unlike many IMAP servers, where the user's INBOX is created initially upon delivery. Because Cyrus does not rely on UNIX accounts to authenticate users nor does it operate with any user's privileges, it cannot know whether a mailbox is valid.

There are patches which allow automatic creation of a user's INBOX (and pre-defined subfolders) upon login. Provided that your users will login before the expect to receive e-mail and that all available accounts should have mailboxes, this is a good method of removing some administrative burden.

Creating a mailbox manually is simply a matter of using the 'createmailbox' command, usually abbreviated 'cm', and providing the correct namespace. For users, that namespace is 'user.', so a mailbox for user 'wcooley' will be created as 'user.wcooley'.

localhost.localdomain> cm user.wcooley
localhost.localdomain> lm user.wcooley
user.wcooley (\HasNoChildren)  

To delete a mailbox, you must fix give the admin user 'create' permissions on the mailbox. To the surprise of many, the admin does not have this permission by default. Then you can delete the mailbox with 'dm'.

localhost> lam user.test
test lrswipcda
localhost.localdomain> dm user.test
deletemailbox: Permission denied
localhost> sam user.test cyrus c 
localhost> lam user.test
test lrswipcda
cyrus c
localhost> dm user.test