An extensible FreeBSD mail system - part 1

Karl Levik
Karl Levik Published: Updated:

Introduction and motivation

In the past I have been running mail systems on various GNU/Linux servers. Since I decided to migrate my websites to FreeBSD I now needed to prove to myself and the world that could set up a similar system there. But why did I want to subject myself to the pain of running my own mail system in the first place?

Mostly, it's about wanting to remain independent from third-party services as much as possible, not wanting to trade my data for convenience. And if the ever smaller number of mega-corporations that run email for power and profit are having a "bad day", then I want to be able to sit back and laugh at them with unrestrained schadenfreude! A secondary but still important motivation is to get hands-on experience with core technologies that have been powering modern communications for several decades.

In this first article I will just show an overview of how this works. As such, most of this should be valid for other Unix-like OSes as well.

A few things to note:

  1. This is a basic stack, so is not suitable as-is for any large-scale operation. However, it is also a modern and extensible stack because this mail software can be complemented with additional components, making it possible to scale up to work for larger organisations.

  2. As a basic stack, though, we can get away with significantly less effort. I'm using the FreeBSD base system and packages only, so there is no compiling of ports! And ironically, despite my day-job as a database engineer, there is currently no database system involved, except for in the case of Roundcube1 - and Roundcube is optional if you don't need a webmail interface.

Roundcube webmail Roundcube, licenced under GPL3-or-later, is a web-based IMAP client

Overview of the mail software stack

mail software stack Slightly simplified view of the mail server stack documented in this series, illustrating sending and receiving mail. The Nginx, php-fpm, Roundcube and database components can be replaced by an MUA such as Thunderbird or mutt.

Core components and definitions

A standard mail software stack can be said to consist of the following components, although many variations exist in the wild2.

  • MUA: Mail User Agent: Also known as the email client, this is the part of the stack that you, the user, can actually see and interact with directly. It's used for composing email and sending it to the MSA/MTA via SMTP, as well as to receive emails via the MDA. In this series I will use Roundcube for MUA. (Roundcube is a web application which obviously needs a web server and PHP server for which I use NGINX and PHP-FPM, respectively.)
  • MRA: Mail Retrieval Agent: This is the receive part of the MUA. Fetchmail is an example of a stand-alone MRA.
  • MSA: Mail Submission Agent: The role of the MSA is to authenticate and receive email messages, from the MUA and pass it on to a local MTA. While stand-alone MSA implementations exist, this function is often performed by the MTA.
  • MTA:Mail Transfer Agent: Also known as a mail exchanger or MX host, this transfers email messages from one host to another via the SMTP. I will use Postfix, but note that the default MTA as of FreeBSD 14.0 is the DragonFly Mail Agent (dma).
  • MDA: Mail Delivery Agent, also known as the Local Delivery Agent (LDA): When an email arrives at a MTA and doesn't need to be passed on to another MTA, then it's delivered to the MDA which puts it into a mailbox. (Various mailbox formats exist, see below section.) The MDA provides IMAP and/or POP3 access to the mailboxes. I will use Dovecot as the MDA/LDA in this series.

Other components

Additionally, as seen in the diagram above, you will need access to DNS, the Domain Name Service, to map between domain names and IP addresses, and to store your various mail records.

I will also use the packages opendkim for Domain Keys Identified Mail, and dovecot-pigeonhole for Pigeonhole which adds support for mail filters ('milters') via the Sieve language and the ManageSieve protocol in Dovecot.

And as mentioned, the OS is FreeBSD, currently 14.1-RELEASE, and I'm using the Packet Filter (pf) from the base system for firewall purposes. (Please see my previous article on how to extend this with Fail2Ban, so you can block IP addresses attempting to break into your mail servers.)

The mail protocols

More general protocols, such as TLS and SASL, are also frequently used in mail stacks, but these are the core mail protocols:

  • SMTP - Simple Mail Transfer Protocol, on port 25 for server-to-server MTA, and port 465 for submission over TLS to the MSA. Sometimes you will see the acronym ESMTP which stands for extended SMTP, i.e. an extension to the protocol. These days "SMTP" has come to mean "ESMTP". An important part of SMTP is the error codes and the Delivery Status Notifications (DSNs)3, as you need these to help debug problems with email delivery.
  • IMAP - Internet Message Access Protocol, on port 993 for retrieving mail from your server-side mailbox via IMAPS, i.e. secure IMAP over TLS.
  • POP3 - the Post Office Protocol v3, on port 110 or secure POP3 (POP3S) on port 995 via TLS for retrieving mail from your server-side mail box, these days replaced by IMAP and will not be covered in this series.
  • LMTP - Local Mail Transfer Protocol, often on port 24, is similar to SMTP, but with a few differences. It should never be on port 25 and is not intended for WAN. The server does not implement a mail queue, so any failed recipients will need to be handled by the client.

Mailbox formats

The xbiff utility

The venerable xbiff
utility for the X
Window System. "When
mail arrives, the flag
goes up and the
mailbox beeps."

The mailbox or Message Store can have two different formats:

  • mbox: The original format, which has one file per mail folder. If you have more than one process attempting to access the mailbox at the same time, this could lead to locking and file corruption. This could be local MUAs on the server (such as mutt) or IMAP or POP3 servers (such as Dovecot) acting on behalf of remote MUAs, or even a utility such as xbiff which is used to notify users that they have new emails waiting to be read.
  • Maildir: The newer format, which has one file per mail message, which is a lot more scalable than mbox.

References

Notes

  1. In a super minimal stack, you can get away with just FreeBSD (or any other Unix-like OS), an MUA such as mutt, and an MTA+MDA such as sendmail. ↩︎

  2. For example, it could be noted that of course Microsoft email is a little different: They use Messaging Application Programming Interface (MAPI) instead of SMTP for submitting email to the MSA, and they use MAPI together with ActiveSync instead of IMAP. ↩︎

  3. Simple Mail Transfer Protocol (SMTP) Enhanced Status Codes Registry, hosted by IANA. ↩︎