Moving my emails (sort of) into The Cloud

I've been meaning to migrate my email to the “cloud” for some time now (well, a little local cloudlet), but my last attempt a couple of years or so ago came a cropper & scared me off.  But now I've gone back to it, and it's all working a treat.

For pretty-much ever, I've been using Fetchmail on my Linux box to gather emails from various accounts (mine & the family's) in the background and squirrel it away in the local mail spool.  Fetchmail sucks down POP/IMAP/etc. emails, then squirts them into the local box's sendmail, which finally (nowadays) uses procmail to actually locally deliver the emails.  It is procmail that can (and for me, does) use SpamAssassin to filter emails.  One day, I'll look at replacing the ageing & unwieldy sendmail with the simpler & cleaner alternative PostFix.  But not yet.

In the olden days, mozilla would then use file-copy to get /var/spool/mail inboxes into it's local (mbox-format) folder structure.  This became deprecated, so ever since then I've been using the excellent Dovecot email server to manage things.  It serves up stored emails (in this case, mbox files named after the users in the spool directory) via POP3 and IMAP, and this is what we've been configuring Thunderbird to use.

Although the other members of the family have been using POP, as that seemed sensible to use with Thunderbird's “combined inboxes” mode, I'd been using IMAP, as I like to occasionally access my unread emails remotely (initially using SquirrelMail as a webmail service on the box).

This, however, meant seeing two inboxes in TB, and having a message filter set up to move all incoming IMAP mail to the TB local inbox.  Plus, if I forgot and left TB running, I would lose the remote capability, as the emails would have been sucked off the IMAP server.

I had always kind of wanted to keep my prime inbox on the server so that I could keep accessing recent stuff, but then filing it around made it even more messy.  Plus, without grief you can't have sub-directories of Inbox (which I do for various things).  I'd also been worried about the long-term safety of keeping lots of things in the single-file mbox format (used by /var/spool/mail).

So a while ago I tried to reconfigure Dovecot to keep Inbox as an mbox file in /var/spool/mail, but have sub-directories (and a few other things) in the safer maildir format.  For whatever reason, that went horribly wrong, so I left it as-is.

Now, however, I am accessing mail from my phone more often, & there's the likelihood of tablets kicking about, and all wanting The Email on them.  Also, now I've moved from SquirrelMail to the rather tasty RoundCube which makes web access even more palatable.  So I had another bash, now time and application versions have moved on.

I decided after some consideration to abandon the plan to keep /var/spool/mail active.  It had caused grief before, and it felt a lot cleaner to just give everyone one big mail store.  No faffing about with Dovecot's “namespaces”.

I'd been storing IMAP data in ~/usr/mail for each user, so I went with ~/usr/mail/store for the new maildir config., just to prevent the new config. treading on the toes of existing files (& in case I needed to revert!).  That's just me being anal:  most people seem to use something messily-top-level like ~/MailDir.

I shut down fetchmail to prevent anything coming in while I was working, played around a lot with Dovecot and procmail settings, and used extensive local ‘mail -s 'test' someuser’ to prove that everything seemed ot be going OK.  Then I started it all up, bringing each fetchmail account online one at a time to double check stuff was working (it wasn't the first time, due to a typo in the procmailrc file!  Thankfully I was creating backups via procmail ...).

The fetchmail config. didn't change.

Here's the procmail config. I'm not using (not the final slashes in the paths to denote the use of maildir.  You can see the old config commented out at the bottom.  I'm also keeping verbose logging logging on for a week or so, as well as the backup files.

# Use maildir format (trailing slash)

# Temp backup
:0 c:

# Already going to spam (prevent recursion)?
* !LOGNAME ?? ^spam$
    # Run spamassassin

    # Anything recognised as spam, send to spam
    * ^X-Spam-(Flag|Status):.*YES
    ! spam

## :0:
## * ^X-Spam-Flag:.*YES
## /var/spool/mail/spam

The mbox-format backup files ~/mail-backup can be used if something goes awry (which happened once).  Simply change their ownership from root to the user in question, and move them into Thunderbird's Local Folders location.  Then the next time TB starts up, it just sees them as local emails in a folder called mail-backup, and you can file them manually into the Inbox then delete it.

To test the non-recursive re-post of spam (yes, I went there!), I used the following email snippet.  Procmail had used to write spam emails directly to /var/spool/mail/spam, which had an appropriate ownership, but with maildir it would dump the emails in new files owned by root (as this is in common config.) which couldn't be then read by my dedicated spam user.  I wanted to keep this set-up (most others seem to have per-user spam stores, set up individually in ~/.procmailrc), so after some failed attempts went for simply forwarding spam on to the spam user, with a cop-out for not checking again.

HELO test
MAIL FROM: my@email.address
RCPT TO: myuser
Subject: Fake spam

more fake spam

I injected it with:

cat /tmp/fakespam | socat - TCP4:localhost:25

Now here are the relevant snippets of Dovecot's config. (these, on Fedora, are sprinkled within /etc/dovecot/conf.d/ files:

mail_location = maildir:~/usr/mail/store/
namespace inbox {
    inbox = yes

I also moved from using local PAM authentication (the users' actual box login passwords, which always seemed a bit dicey for webmail access) to a dedicated passwd file in /etc/dovecot:

#!include auth-system.conf.ext
!include auth-passwdfile.conf.ext

# example line from the users passwd file
someone:{CRAM-MD5}a3a19b0f11f8f9f964ffa6423f3f98227b55897ab34e1465a42eff9e3d6ed0ba:1020:1020:Some One:/home/someone:/bin/bash

The Dovecot command ‘doveadm pw’ is a god-send for creating hashed passwords you can cut'n'paste into the file.  Don't forget ‘dovecot -n’ to check the dovecot config. after you've fiddled too!

Most of the other default Dovecot config. I was able to leave as-is.

The only other thing I struggled with was migrating the existing mbox files out of Thunderbird and up into Dovecot.  It comes with a dconv tool to do mailbox conversions, mirrors and backups, but I failed utterly to get something sensible out of it (probably because the source emails weren't created by Dovecot in the first place, but Thunderbird).

In the end, the simplest solution to converting the mbox format file to maildir was ... not to.  I simply started up Thunderbird (and in the case of the family, created new IMAP accounts to replace the old POP ones), and dragged the mails and folders from the Local Folders store into the IMAP one.  Simples.  I only came a slight cropper when copying a couple of shop email folders which had dots in their name:  maildir doesn't support this, so I had to use dashes instead.

I found I wanted to stop Dovecot logging every time someone polled their email, as my maillog doesn't really need to be written to that frequently.  I also check mails in a local service (to keep a USB LCD display able to report unread emails).  This used to count “From” lines in /var/spool/mail/* but of course can't now.  So it uses a modified perl script I found to get the free count.  This needs to know users' passwords, hence another reason to use separate email passwords.  I keep the script root-read-only and accessible only via sudo.

This snippet in rsyslog.conf stopped Dovecot spamming the maillog file (no pun intended):

# Log all the mail messages in one place.
# But no .info login/logout crap (only notice or higher)
mail.notice                    -/var/log/maillog

Here's the script (originally just for GMail):

  1. #!/usr/bin/perl -w
  2. # by gxmsgx
  3. # description: get the count of unread messages on gmail imap
  5. use strict;
  6. use Mail::IMAPClient;
  8. my $server   = '';
  9. my $username = $ARGV[0];
  11. my %password = (
  12. 'list'  => 'password1',
  13. 'of'    => 'password2',
  14. 'users' => 'password3',
  15. );
  17. if (not defined($username)  or  not defined($password{$username}))
  18. {
  19. print "-1\n";
  20. exit 1;
  21. }
  23. my $client = Mail::IMAPClient->new(
  24. Server   => $server,
  25. User     => $username,
  26. Password => $password{$username},
  27. Ssl      => 1,
  28. )
  29. or die "new(): $@";
  31. if ($client->IsAuthenticated()) {
  32. my $msgct;
  34. $client->select("INBOX");
  35. $msgct = $client->unseen_count||'0';
  36. print "$msgct\n";
  37. }
  39. $client->logout();

Also useful was the slew of debug and logging options in Dovecot.

So, now all my mail is in the cloud (well, my box's version of it), and I can get at it from anywhere.   While my box is powered on (which is mostly all the time).

[update] I've amended the global procmailrc file so that I can use a local ~/.procmailrc without having full paths, e.g.:

#deliver emails with PVR in the subject to the INBOX/PVR subfolder.
* ^Subject:.*PVR

[update-2] And again, to get it to detect recursion better (by username), whilst allowing it to detect spam my ISP has already flagged with another keyword.


About fnx

PSN: the_phoenix
This entry was posted in Software, Tech and tagged , , , , , , , . Bookmark the permalink.

Leave a Reply

Your email address will not be published. Required fields are marked *