Where do mails come from, anyway?

So recently I was moving my e-mail setup from one server to another and in the process decided to wrap it up in a nice, tidy Docker container, called Mail-o-Rama. I did this not for isolation or because I planned on moving servers again in the near future, but mainly to document everything I needed to get things up and running. Fiddling with SMTP servers, DKIM proxies, IMAP stuff etc. can be time consuming and error prone.

During that time I asked myself a couple of simple questions:

What about the e-mails that are generated from my local system, like from crond? How do they come to be, anyway? Is it a system call? Are there POSIX standards?

Nobody I asked could spontaneously explain how cron or other tools were really creating e-mails, so I decided to dig deeper and – shudder – read some C source code.

vixie-cron

The “classic” cron daemon on Debian/Ubuntu is the one written by Paul Vixie, hence the name “vixie cron”. It’s a good starting point to dig deeper. The FSF maintains the link to its source code, but beware, Debian for example applies a massive 300KB patch before compiling their version, so a few minute details differ from the stock vixie cron source.

Grepping through the code and searching for “mail”, we find a couple of interesting spots. First of, there’s this in the config.h:

#if !defined(_PATH_SENDMAIL)
# define _PATH_SENDMAIL "/usr/lib/sendmail"
#endif /*SENDMAIL*/

It checks if there is already a constant _PATH_SENDMAIL and if not, simply assumes a “sendmail” binary at /usr/lib/sendmail (spoiler, on many systems including Debian this binary is actually at /usr/sbin/sendmail). But who could have set _PATH_SENDMAIL? It’s actually set in Linux’ own paths.h. Good to know.

In the same file, we also find this:

      /*
       * choose one of these MAILCMD commands.  I use
       * /bin/mail for speed; it makes biff bark but doesn't
       * do aliasing.  /usr/lib/sendmail does aliasing but is
       * a hog for short messages.  aliasing is not needed
       * if you make use of the MAILTO= feature in crontabs.
       * (hint: MAILTO= was added for this reason).
       */

#define MAILCMD _PATH_SENDMAIL          /*-*/
#define MAILARGS "%s -FCronDaemon -odi -oem -or0s %s"   /*-*/
      /* -Fx   = set full-name of sender
       * -odi  = Option Deliverymode Interactive
       * -oem  = Option Errors Mailedtosender
       * -or0s = Option Readtimeout -- don't time out
       */

/* #define MAILCMD "/bin/mail"      /*-*/
/* #define MAILARGS "%s -d  %s"     /*-*/
      /* -d = undocumented but common flag: deliver locally?
       */

/* #define MAILCMD "/usr/mmdf/bin/submit" /*-*/
/* #define MAILARGS "%s -mlrxto %s"   /*-*/

Now we’re getting closer. Apparently cron simply starts an external process, MAILCMD, with a few command line options to send mail. How does that work exactly? Thankfully the source for this is very easy to grasp, even for non-C developers like myself. Take a look at the do_command.c:

    if (ch != EOF) {
      register FILE *mail;
      register int  bytes = 1;
      int   status = 0;

      Debug(DPROC|DEXT,
        ("[%d] got data (%x:%c) from grandchild\n",
          getpid(), ch, ch))

      /* get name of recipient.  this is MAILTO if set to a
       * valid local username; USER otherwise.
       */
      if (mailto) {
        /* MAILTO was present in the environment
         */
        if (!*mailto) {
          /* ... but it's empty. set to NULL
           */
          mailto = NULL;
        }
      } else {
        /* MAILTO not present, set to USER.
         */
        mailto = usernm;
      }

      /* if we are supposed to be mailing, MAILTO will
       * be non-NULL.  only in this case should we set
       * up the mail command and subjects and stuff...
       */

      if (mailto) {
        register char **env;
        auto char mailcmd[MAX_COMMAND];
        auto char hostname[MAXHOSTNAMELEN];

        (void) gethostname(hostname, MAXHOSTNAMELEN);
        (void) sprintf(mailcmd, MAILARGS,
                 MAILCMD, mailto);
        if (!(mail = cron_popen(mailcmd, "w"))) {
          perror(MAILCMD);
          (void) _exit(ERROR_EXIT);
        }
        fprintf(mail, "From: root (Cron Daemon)\n");
        fprintf(mail, "To: %s\n", mailto);
        fprintf(mail, "Subject: Cron <%s@%s> %s\n",
          usernm, first_word(hostname, "."),
          e->cmd);
# if defined(MAIL_DATE)
        fprintf(mail, "Date: %s\n",
          arpadate(&TargetTime));
# endif /* MAIL_DATE */
        for (env = e->envp;  *env;  env++)
          fprintf(mail, "X-Cron-Env: <%s>\n",
            *env);
        fprintf(mail, "\n");

        /* this was the first char from the pipe
         */
        putc(ch, mail);
      }

      /* we have to read the input pipe no matter whether
       * we mail or not, but obviously we only write to
       * mail pipe if we ARE mailing.
       */

      while (EOF != (ch = getc(in))) {
        bytes++;
        if (mailto)
          putc(ch, mail);
      }

      /* only close pipe if we opened it -- i.e., we're
       * mailing...
       */

      if (mailto) {
        Debug(DPROC, ("[%d] closing pipe to mail\n",
          getpid()))
        /* Note: the pclose will probably see
         * the termination of the grandchild
         * in addition to the mail process, since
         * it (the grandchild) is likely to exit
         * after closing its stdout.
         */
        status = cron_pclose(mail);
      }

      /* if there was output and we could not mail it,
       * log the facts so the poor user can figure out
       * what's going on.
       */
      if (mailto && status) {
        char buf[MAX_TEMPSTR];

        sprintf(buf,
      "mailed %d byte%s of output but got status 0x%04x\n",
          bytes, (bytes==1)?"":"s",
          status);
        log_it(usernm, getpid(), "MAIL", buf);
      }

    } /*if data from grandchild*/

Easy enough: popen() a process, write a simple email with subject line to its stdin and we’re done here.

Note that Debian extends this section a bit, so if no sendmail could be found (for example because you deleted your sendmail binary), a warning would be printed and you can tell by looking at your system journal.

So now we know how (vixie-)cron actually creates an email: it uses something called “sendmail”. Let’s figure out what this sendmail thing is.

sendmail

sendmail is both a piece of software and a quasi standard when it comes to creating e-mails programmatically. You can get the original sendmail implementation from Proofpoint, but your own system is most likely not using the original.

Postfix

See, if you install Postfix you also get a sendmail binary. But it’s not the original sendmail, it’s just a shim, a wrapper around Postfix’ own postdrop functionality to accept e-mails. It behaves in the same way as sendmail (which makes sense), but is a totally different implementation. If you download the Postfix source code, you will eventually find the src/sendmail/sendmail.c file, whose manual section clearly states that it’s a “Postfix to Sendmail compatibility interface”.

Exim4

Exim similarly also provides a sendmail-compatible interface, but in its case it usually just comes in the form of a /usr/sbin/sendmail -> /usr/bin/exim4 symlink, because the Exim4 binary in itself is already compatible.

Others, like msmtp

If you look at other MTAs like msmtp, you will (in case of Debian or Alpine) find that they are often accompanied by a -mta package which provides a sendmail compatible wrapper, like msmtp plus msmtp-mta.

What about the mail command?

The mail command on Debian is part of the mailutils package. For sending e-mails, also simply uses sendmail, but passes in a few more convenient headers. The actual binary it uses can usually be configured through your /etc/mailrc file.

Summary

What have we learned so far? We now know that sendmail (the interface, that is) is an integral part of your Linux machine’s e-mail setup. Programs like cron or mail use it to have a single point through which all e-mails go. Alternative MTA implementations like Postfix, Exim4, ssmtp, msmtp etc. all provide a sendmail-compatible implementation to make integration with existing software easier.

What happens when sendmail is invoked is up to the MTA. Full-featured MTAs like sendmail (the original), Postfix or Exim4 can be configured to do many fancy things. Smaller MTAs like msmtp will simply forward the e-mail via SMTP to an external SMTP server.

Back to Docker

Armed with this knowledge, we can see two different approaches with our Docker setup:

  1. We can provide our own sendmail program, which would forward the e-mail to the Docker container. Thankfully, OpenSMTPD also brings with it its own sendmail, so we could just perform a docker exec <container> sendmail to have the e-mail delivered inside the container. Since cron uses popen and does not invoke a shell, we unfortunately cannot simply put a one line shellscript at /usr/sbin/sendmail, but we need a standalone binary.

  2. Alternatively, we can use a minimal “sendmail to external SMTP forwarder” like msmtp (and msmtp-mta) and configure it to relay all incoming e-mail to localhost, port 25.