As I am moving out of Gmail, I had to find a way to deal with the 21GB mail archive from the past 12 years.

Google lets you export all your data from its various services, including email. After a day or so you get a download link that contains all your mail in one single file in MBOX format.

MBOX is a text format so it allows itself to be searched, but that would only tell you that what you are looking for is somewhere in that 21GB file.

I could also import it into my mail client as a local archive, by dropping the MBOX file in the Local Folder of Thunderbird with Finder. That provides me with a similar access and search capability as I had for all that mail in Gmail. However, if I would like to do more with my archive, mine it for things, and re-use stuff by piping it into other workflows having it in Thunderbird would not be enough.

Mailsteward puts MBOX into MySQL
So I searched for a way to more radically open my archive up to search. I came across DevonThink, but that seemed a bit overkill as it does so much more than merely digesting a mail archive, and as such provides way too much overlap with my Evernote. (Although I may rethink that in the future, if I decide to also move out of Evernote, as after Gmail it is my biggest third party service that contains lots of valuable information.) I looked for something simpler, that just does what I need, putting e-mail into sql, and that is how I found Mailsteward Pro.

There are three versions of Mailsteward, and I needed the Pro version, as it is the one that works with MySQL and thus can handle the volume of mail in my archive. It costs $99 one time, not cheap, but as I was paying for storage with Google as well, over time it pays for itself.

Installing Mailsteward
When installing Mailsteward it assumes you already have a MySQL server running on your system. I use MAMP Pro on my laptop as a local web and mysql server, on which I run different things locally, like a blog based journal and a self-assessment survey tool. MAMP Pro is very easy to install.

You need to take the following steps to allow Mailsteward access to MySQL. In MAMP Pro you need to allow external access to MySQL, but only from within your own system (this basically means applications other than MAMP can access the MySQL server.

Schermafbeelding 2016-07-19 om 16.37.07

Then you create a new database via the PHP Mysqladmin that comes with MAMP. Mailsteward will populate it with the right tables. In my case I aptly named it mailarchives.

Schermafbeelding 2016-07-19 om 10.48.16

Within Mailsteward you then add a connection, listing the database you created, and adding the right ports etc. Note that the socket it requests isn’t an actual file on your system, but does need to point to the right folder within the MAMP installation, which is the Application/MAMP/tmp/mysql folder.

Schermafbeelding 2016-07-19 om 08.41.51

Importing MBOX files
I first tested Mailsteward with my parents e-mail archive that I kept after they passed away last year, to be able to find contact details of their friends. It imported fine. Then I tried to import my Gmail MBOX file. It turns out 21GB is too large to handle in one go for Mailsteward, as it eats away all memory on your Mac. I concluded that I need to split my Gmail MBOX file into multiple smaller ones.

Luckily there is a working script on GitHub that chops MBOX files up in smaller ones, and that allows you to set the filesize you want. I chopped the Gmail MBOX into 21 smaller files of 1GB each. These imported ok into MailSteward. Mailsteward maintains tags and conversation threads.

To run the script, first open it in a text editor and change the filesize limit to what you want (default is 40MB, I changed it to 1GB). Then open Terminal and run the script by typing the following command, where the destination folder does not need to exist:

sudo php mbox_splitter.php yourarchivename.mbox yourdestinationfolder

terminalcommand

That way you end up with a folder that contains all the smaller MBOX files:

Schermafbeelding 2016-07-22 om 16.06.53
Using Mailstewards import feature you then add each of those files, by hand (but luckily you only need to do that once).

Using the archive
Mailsteward allows you to search the archive through its rather simple and bland interface, but you can also tweak the MySQL queries it creates yourself. The additional advantage of having it in MySQL is that I can also access the archive with other tools to search it.

Schermafbeelding_mailsteward

Adding newer mail to the archive
Thunderbird allows me to export e-mail as MBOX files via the Import/Export add-on, which can then be added to the archive by Mailsteward. So that’s a straightforward operation. Likely I can automate it and schedule it to run every month.

Today I changed the way we use e-mail addresses for identification on-line.

Over time my e-mail address(es) has (have) become the carrier of a lot of important stuff. It’s not just a way to communicate with others, but also serves as generic user name on countless website accounts. And likely quite a few of those have had their security breached over time, or are unscrupulous (or even malicious) in their own right.

As part of a talk on privacy by Brenno de Winter (Dutch investigative journalist) that we went to this weekend (see previous posting), he mentioned using unique e-mail addresses (and pw’s of course) for every site you use. Or disposable e-mail addresses for sites you visit only once. That way when one site gets compromised there is no risk of your user credentials being used elsewhere, and if one site sells your email addresses on it is immediately apparent to you who did that.

I have been aware of this advice for a long time, but never saw an easy way to act on it:

  • Most disposable e-mail address (DEA) services offer a temporary e-mail address, usually enough to quickly confirm an e-mail address, after which it gets deleted automatically. This is useful for one time visits / registration at a website, but not for using unique addresses for services you use more often.
  • Some sites do not accept e-mail addresses that are clearly created by DEA type services
  • I own multiple domains, which I could theoretically use for unique mail addresses, but in practice that is much more unlikely. I would need to either create mail addresses before using them to register somewhere, through the domain’s administration panel, or use a catch-all that would simply accept any incoming mail on that domain, including tons of automatic spam flung out to randomly generated e-mail addresses.
  • What I actually need is:

  • The ability to create new e-mail addresses on the fly, simply by using them
  • The ability to both have more permanent unique addresses, as well as single use addresses
  • Using a domain that is not perceived as a DEA service and not easily associated to me (e.g. by visiting its website)
  • Using a domain that I control so I cannot get cut off from unique addresses connected to important user accounts
  • The ability to recognize any of these unique addresses in my regular inbox
  • Something that still filters out spam, while accepting any incoming address
  • So today I decided to investigate further and act on it.
    This is the solution I came up with:

  • I found 33mail.com, built by Andrew Clark (in Dublin/Ireland so under EU regulations), that allows you to create addresses on the fly, and then through a dashboard simply block the ones that get misused at some point. It also forwards to one of your actual e-mail addresses, including letting you (anonymously) reply from the unique address.
  • 33mail.com allows you to connect any other domain to their service, so that instead of using something@myaccount.33mail.com I can use something@myrandomdomain while still using 33mail. This is very useful as it helps to prevent being filtered out because of using a DEA service domain, and keeps the addresses under my control.
  • I registered two new domains, one for me, one for Elmine, and set up their MX DNS records to point to 33mail. So that anything@ourtwodomains.tld goes to 33mail. These domains are, apart from the records at the registrar, not otherwise easily associated to us.
  • I provided two unique email addresses for 33mail to forward to at two other domains I own and use.
  • I set up two auto-forwards for those addresses that 33mail forwards to, which makes it end up in one of my or Elmine’s regular inboxes. In our inbox we have filters that pick up on anything that comes from those forwarding addresses 33mail sends stuff to.
  • This is not a free solution, but it is cheap. The registration of two domains, plus a service package so I can set my own DNS settings, with our regular hoster comes to 45 Euro or so. 33mail charges 8 or 9 Euros for a premium account, which is needed to add your own domain name to their service, and I created a premium account for each of us, as we will be using two seperate domain names. Total cost: 65 Euro/yr.

    Here’s a drawing of the full set-up:

    33mail