9 ms·

Small mail server best current practices

401 points by ebcase 6y ago

deadalus 6y ago
I personally use zoho.com(free plan) for email to and from my domain. What is the use of self-hosting a email server over using Zoho or Gmail?
- acidburnNSA 6y ago
  I like hosting mine for a few reasons: - It's a hobby and I enjoy it - It like fiddling around with advanced features, like wildcard addresses, server side filtering (via dovecot-sieve), and other fun stuff. For instance, a friend once was mulling on facebook on detecting tone of email and rejecting ones that were too mean. I made a simple python script that attempted this and had her try it out. It was funny. (See previous point about hobby) - I figure that being on your own server you're at least slightly less likely to be a victim of a big internal platform hack like twitter's recent one, or state-run ones. - It feels slightly badass. I don't recommend it to people who don't want to do it for fun.
elric 6y ago
Not sure if I want to carry on host other people's email anymore, it's become a huge pain. It's bad enough that Google and MS frequently refuse to deliver mail; without any apparent reason or any humans to discuss the problem with. But now that zimbra is dead (no new open source versions), email hosting is pretty much back to where it was in the 00s.
juskrey 6y ago
Mailinabox handles all of that automatically and nicely. You just need to do some manual DNS work and check your IP against blacklists on http://multirbl.valli.org/ http://multirbl.valli.org/ - and mostly done.
- jeffbee 6y ago
  I think part of the point of the post and the PDF to which it links is that Gmail and the other big operators don't use the blacklists, so for deliverability to the majority of recipients it makes no difference whether you are or are not on those lists.
  - jeffbee 6y ago
    Example of how the blacklists contain essentially no information: every one of Gmail's outbound VIPs is in a SORBS blacklist. Every one I checked is in at least 20 other blacklists. So if you just don't want to get mail from hundreds of millions of people, subscribe to IP blacklists.
    - johnklos 6y ago
      That's not true, unless "VIP" means something that I don't understand. I use SORBS on a rather busy mail server with thousands of messages sent to and from Gmail daily. Aside from very occasional SORBS rejections, which happen more often with Outlook.com and Yahoo, this simply isn't the case.
      - jeffbee 6y ago
        See for yourself: http://multirbl.valli.org/lookup/209.85.128.65.html http://multirbl.valli.org/lookup/209.85.128.65.html
        e12e 6y ago
        Seems to be mostly not listed?
        jeffbee 6y ago
        Sure but blacklists are binary. That’s why they don’t work.
        6y ago
davemtl 6y ago
With all the fixes and standards just to send a cute cat GIFs to your friends on Gmail is such a pain in the butt for most self hosters. If you're doing it, great! Keep it up! All the better for you. But for me, using a e-mail hosting provider with a good reputation is preferable, they've already done all the hard work (my preference is FastMail, but it's not everyone's cup of tea). I've been in the IT field for almost 20 years, I've done the self hosting e-mail thing. In my early teens and 20s, I didn't have anyone to please or take care of, so keeping up was easy with new e-mail standards as they rolled out, 15-20 years later, I'm divorced, work full time and have two kids, I simply don't have the time to self-host anymore. I'd rather give somebody a few $ a month to take care of that for me.
- hkt 6y ago
  I've been so irritated with self hosting I've gone the other way: I'm in the process of setting up an email hosting co-op, with the aim of paying for managed service from a small hosting provider for most things but paying no more than I do now. That way it won't only fall to me to deal with deliverability issues etc.
  - saimiam 6y ago
    See my sibling comment to yours! I think we are both working towards the same goals. Nice to meet you.
- saimiam 6y ago
  > I'd rather give somebody a few $ a month to take care of that for me I'll do it though i'm not FastMail! I'm working on a SES backed email client so SES takes care of deliverability (assuming the user does the basic DKIM, MX, TXT set ups) so it will work out much cheaper to send and receive mails than a FastMail/GSuite type solution. Plus you'll get unlimited disposable emails for free, a shared inbox (if you want to share an inbox with your family to help stay on top of every one's schedules, for exam), and while the client I have built is pretty rudimentary, i'll be adding bells and whistles to make it better. Github @ https://github.com/saiorama/ses-email-client https://github.com/saiorama/ses-email-client Landing page to submit a request @ https://shared-inbox.landen.co/ https://shared-inbox.landen.co/ Note to others - I'm open to collaborating to take back email from the big guys. Hit me up via Github.
jks 6y ago
Addendum: if you host a mailing list server such as Mailman, you'll need to do one of (1) don't change the content or DKIM-signed headers at all (e.g. by adding Subject tags or unsubscribe instructions); (2) rewrite the From: header so that you don't look like you are impersonating the individual senders; (3) use ARC [https://en.wikipedia.org/wiki/Authenticated_Received_Chain https://en.wikipedia.org/wiki/Authenticated_Received_Chain] signatures. I don't know if Gmail and the other big providers actually honor ARC headers yet, but if they ever do, that would seem to be the cleanest solution if you need to add unsubscribe instructions or similar banners.
guerby 6y ago
I wonder if there's an ansible playbook implementing all that?
- ireflect 6y ago
  Sovereign does most of it: github.com/sovereign/sovereign I was an early contributor to Sovereign and used it for many years, but I found that we kept adding features to it and it got too large for my liking. Now I use docker-mailserver: https://github.com/tomav/docker-mailserver https://github.com/tomav/docker-mailserver
  - gertrunde 6y ago
    I looked at Sovereign a while back, and came to the same conclusion, and ended up going with mail-in-a-box, which is possibly a little too far the other way, but at least very low maintenance.
evilsetg 6y ago
Thank you! I recently managed to get a static IP and was looking forward to set my own mail server to be independent of the big providers. Now I have been searching for something exactly like this, a recent, concise overview of the hoops I have to jump through to be allowed to send to the big guys.
- saimiam 6y ago
  I wrote one as well @ https://medium.com/@ssr233/get-disposable-emails-for-your-domain-in-minutes-8c5c8830fe7b https://medium.com/@ssr233/get-disposable-emails-for-your-do... I'm also trying to build my own email client for SES @ https://github.com/saiorama/ses-email-client https://github.com/saiorama/ses-email-client. Happy to collaborate in case you want to work on this issue.
icefo 6y ago
Tangentially related but I'm having some issues related to email beeing refused by Gmail. In my organization we have ~10 linux VM and they are all configured to send email to an exim MTA. For example monitoring stuff. The system mail name is set to vmName.srv.domain.name (domain.name is replaced by our actual domain and this resolve to a local address) so it's quite common to receive mail from root@vmName.srv.domain.name. This cause no issue when it's delivered to the local Cyrus server but when it's transfered to a Gmail address for some reason Gmail reject the email because the headers are bad. I am considering rewriting the address to user.vmName.srv@domain.name in exim when receiving email from local VMs and I am wondering if this could be a bad idea. I'm going to redo this mail server from scratch soon because it has been left untouched for something like 6 years. Some software was compiled on it with shared libraries which then were updated but the software was not recompiled. It's a dumpster fire and I'm actually surprised most of our emails seem to go through. I would also add to the article that's it's a good idea to have some rate limit on outbound email, outbound email spam checks and to check that the user sending email has the right to use that address.
- johnchristopher 6y ago
  Quick question (as someone who doesn't manage email): why not using an external mail provider (like mailjet) ? Privacy concerns and/or internal mails that shouldn't get leaked ?
  - icefo 6y ago
    That server is at least 10 years old and quite custom. For example when an email is received the address is queried to an ldap and if it's a mailing list the email is sent to all the recipients of that mailing list which can be other mailing lists ! This goes on until an user address is found and then the email is delivered to the cyrus server. Each user can also have many alternative addresses that map to an unique mailbox Those mailings lists are used a lot internally and managed with internally developed tools. This make migrating to an external service difficult. That's the classic example of some critical piece of infrastructure that nobody dare to touch because it works... until it doesn't anymore. That's why we don't use external services to receive email. I'll consider using mailJet if setting up exim & related stuff correctly to send email in 2020 proves too painful.
pgrote 6y ago
Running an email server became a real pain about 4 years ago. Until then, we could run our own exchange server, control everything and it was awesome. Then came the email security products from the large companies. It became a self fulfilling prophecy. Our email would trigger a rule in big co's third party security app, which would then report to a centralized rule breaker clearing house automatically. Clients couldn't receive our emails, we would dive in, submit appeal to clearing house, get clear and a week later do it all again. Everything was configured properly on our end. We passed all the online validation engines. IPs were our own personally owned block and pristine. It became too much. Switched to office365 and all problems magically disappeared. Sent emails to same big cos with third party email security and haven't had a single issue.
- colordrops 6y ago
  Is the implication that this is an intentional practice to drive out small players?
  - jandrese 6y ago
    I don't think it is an attempt to drive out small players, it's a natural consequnce of domains being almost free to stand up. They just don't provide any level of assurance that the owner isn't a spammer. So the blackhole systems treat them as "probably spammer" and fire up a block at the first whiff of malfeasance. It's harder to sign up for a Gmail account than it is to register a domain and get some hosting. And Google has their own protections against sending large amounts of mail built-in. The system is as it is now because spammers will abuse it otherwise.
    - throwaway2048 6y ago
      gmail is effectively block by default these days, unless Google has some reason to not block your mail, it will be spamboxed.
      - bachmeier 6y ago
rsync 6y ago
Ugh. I have been running my own UNIX mailserver since 1997. I have consistently and loudly evangelized this practice, especially here on HN, and pushed back against the notion that it was too difficult or too time-consuming to implement and maintain one's own mailserver. But ... Having just recently made the major transition from relying solely on clean IP space and proper DNS records (which, in the last 18 months, has become increasingly untenable[1]) I have changed my mind. I will continue to run my own mailserver, for activist, ideological reasons, but I must now agree that it is too difficult and complicated - and fragile. In no particular order, some of the things that I find overly complex and disturbing are: - The DKIM implementations are very, very over-engineered. I understand, in principle, why we want DKIM to be a pluggable component that can be used as a general building block in various implementations, blah blah blah, but there is no reason that DKIM can't just be a feature, with a corresponding blob of config lines, in sendmail or OpenSMTPd or whatever. Having to pkg-install, and track, and maintain, whatever DKIM implementation you choose, along with all of its various dependencies, etc., is a big truckload of complexity and fragility that I just can't see ever appreciating. And the irony ? A popular DKIM implementation is to use the bundled DKIM functionality built into rspamd (!) ... so it ended up being just a feature anyway. - The whole SSL component ... I appreciate this, I understand this, and for many use-cases I just don't care. I don't currently need encrypted email and if someone makes a decision to disallow sending to addresses/domains that don't support it, fine. Of course, if all of gmail decides not to, now I have another giant truckload of complexity and fragility that I need to implement and maintain. My mailserver should be an extremely tight, stripped down host with as few packages and daemons possible[2]. Instead, I am now installing a big list of packages just so that letsencrypt can fire up a temporary web server to clicky click make my certificates[3]. - Now my mailserver is running a database (Redis) which is required to run rspamd, which is required to implement DKIM, without which I cannot send mail to yahoo.com addresses. True story. So much complexity and fragility ... [1] About 18 months ago yahoo.com stopped, with no errors or bounces, delivering mail from an 11 year clean IP with proper DNS/MX entries. I implemented DKIM and the problem is solved. [2] My mailserver is a FreeBSD jail and previously ran only sendmail - and nothing else. Now I am running OpenSMTPd, rspamd, redis ... and once in a while I am running a webserver to communicate with letsencrypt. [3] Speaking of letsencrypt ... requesting, generating and implementing SSL certs (even for use on the web) is simply creating, and trading, blocks of plain old ASCII text. I haven't looked into this, but is there an "expert mode" somewhere in letsencrypt where I don't have to run webserver instances and ... I can just run openssl command lines and cut and paste blobs of ASCII ?
niftylettuce 6y ago
I made Forward Email after running into many issues with self-hosted solutions (and realized thousands of others have the same issue due to the high-demand after launch back in 2017). It's a great alternative to having to set up your own small mail server. https://forwardemail.net https://forwardemail.net P.S. I'm coming out with a fully-fledged email service built on top of my R&D with Forward Email.
- CraftThatBlock 6y ago
  Any cheap paid services for this? I'm not comfortable with routing my emails through a free third-party.
  - dejj 6y ago
    Then how are you comfortable routing it through a cheap service instead? Asked more precisely: how do you determine reliability from price?
    - namibj 6y ago
      It's more that you tend to get some legal implications from selling a service vs. having some website with no payment on the net.
  - niftylettuce 6y ago
    We do offer paid plans with enhanced features. And we're completely open-source, store zero logs, and store zero metadata too. All source code is on GitHub as well. I formerly worked at DuckDuckGo, and take privacy very seriously. Privacy Policy: https://forwardemail.net/privacy https://forwardemail.net/privacy Terms: https://forwardemail.net/terms https://forwardemail.net/terms You can delete your account at any time.
  - 627467 6y ago
    Forwardemail.net seems to have paid plans. Reading through the about and FAQ they also seem to be driven by decent principles. https://forwardemail.net/en/faq#how-is-it-free https://forwardemail.net/en/faq#how-is-it-free
Cyph0n 6y ago
I went through the journey of configuring Postfix for a small side project I was working on. The idea was to accept emails sent to a unique, per-user address and forward all attachments to the user’s cloud storage account. I looked into using mail services like SES and Mailgun, but given the average expected email size and the fact that I would primarily receiving mail, the additional cost didn’t make sense. Along the way, I ended up learning quite a bit about Postfix filters, virtual domains, SPF, and DKIM. I also learned how quickly spammers can detect open relays. Apparently, while copy-pasting a config from somewhere, I had a slightly incorrect 172.x subnet listed as part of “mynetworks”. As a result, a spammer with a machine belonging to this subnet was able to use my server as an open relay. Long story short, it took me quite a while to root cause the issue and get my domain off the blacklists!
tgsovlerkhgsel 6y ago
There really needs to be a) a docker container (or other standardized, easy to deploy technology) with a "mail server in a box implementing best practices" b) a test for the stuff that Docker can't cover like DNS config, ideally with scripts to set it up on the most popular providers and clear copy&paste instructions for the rest. People shouldn't need to fully understand DKIM, SPF, etc. - they should just be able to click a button, copy&paste generated DNS records, and be up and running. Otherwise, commercial providers are just the only option that makes operational sense for most cases, _even if you know how to do it right_, because doing everything right manually step by step is still to much work to be worth it in many cases. None of this seems _hard_, it's just tedious.
- pbhjpbhj 6y ago
  Agreed. A few years back when moving my personal domain from a failing provider I considered running my own email server on Digital Ocean; they had some great tutorials on setting up various parts and I followed a HN recommended tutorial. I was already happy setting up LAMP with a nginx reverse proxy, how hard could it be - I got it setup, and realised two things: Keeping everything running without getting compromised was going to take way too long compared to the benefit. Two, Google and MS wouldn't accept my email - not even to myself when I whitelisted it - it's a crap shoot. Email has serious economies of scale and keeping such a critical system operational, on a small scale, was going to be some deal of stress. So I just went with my hosting providers Plesk-fronted setup, which matches your para.3 description.
- gsich 6y ago
  >People shouldn't need to fully understand DKIM, SPF, etc. - they should just be able to click a button, copy&paste generated DNS records, and be up and running. I oppose this. People should know what they are configuring. This applies to all stuff related to mailservers. You probably learn more by setting it up the "hard" way (without containers) so that in case of failure you at least have a little insight in how the components work.
  - im3w1l 6y ago
    There are many more people who want to drive a car, than people who want to learn the minutiae of how it works. If it breaks, then you get someone else to fix it / get a better product is all they need to know.
cm2187 6y ago
Point 14 is a very good one. If any account gets compromised, you might as well request a new IP as your previous IP will be in the dog house for a long time. Almost more important than monitoring failed logons, monitor successful logons, for instance by country. If it is a small mail server, chances are your users are in a handful of countries. If you suddenly see a successful logon from vietnam or brazil, you probably want to be notified ASAP.
davidbrennerjr 6y ago
Eh, I didn't read it...so my thoughts. From experience the prob with DNSSEC is latency in verifying PKI of the record. Usually what happens is the timeout per resolver has to be greater than 15 seconds under no load in a private network which is very inconvenient in comparsion to plain DNS that has a default of 5 seconds.
thoraway1010 6y ago
The volume of spam is mind boggling. When we first implemented DMARC / DKIM etc our legit emails were 3% or so of all outbound mail per the reporting we got back! We have a somewhat trusted domain. Some mail delivery systems start to notice that your domain name is being used as part of a lot of spam / bogus emails, so you can have perfect IP history / no spam and STILL start triggering some random filters (no big players but smaller protection product filters). So the game must be absolutely never ending for everyone and the inbox a valuable target - especially now that unsolicited phone calls really do seem to get ignored these days - I feel like spammers killed the golden goose on phone calls and the telcos let them. I will say DKIM / DMARC is working well, except google (which we now use for outbound) gives us transient SPF errors even though we are 100% using their IPs. Not sure why that is (ie, SPF failure on an IP that should clear)
- dorfsmay 6y ago
  For your last point, check https://toolbox.googleapps.com/apps/checkmx/check https://toolbox.googleapps.com/apps/checkmx/check. Chances are, google doesn't like your DNS server answers to its ANY requests, regardless for them complying to the RFCs.
- Geezus_42 6y ago
  As someone who works in email security I can tell you that it is a never ending cat and mouse game with a lot of false positives as to who is a mouse. Even our customers that have good domain rep, SPF, DKIM, DAMRC, etc still randomly get blocked by filters and large email providers. Part of the problem I see it those large providers and some of the filtering systems send back NDRs that basically provide no information as to why the message was blocked so trying to get your legitimate mail flow working again once someone has flagged you can take days to months as you randomly adjust all the knobs trying to figure out why they are blocking you. I'm sure they would say providing more informant NDRs would make it easier for the spammers to work around the filters. I don't know what the answer is but the current system sucks, it is a huge pita and a dumpster fire for the most part but it's what we have to work with. I'm sure someone will come up with another scheme like DKIM that will become another check box on the list of things you have to set up and once agian few will understand how to implement correctly.
s_gourichon 6y ago
TL;DR: Cool that this thread exists because people will mention (please do!): * Docker-based e-mail setup, with comments about their experience, like https://github.com/tomav/docker-mailserver https://github.com/tomav/docker-mailserver (which incidentally IIRC was recently looking for a new maintainer). * ...ideally that would have one host running SMTP facing the big bad public Internet and therefore minimalistic, and a separate more isolated host that runs the IMAP server. (My current setup does that with, looking for hints but perhaps I have more to share than to learn this time.) -- https://foxcpp.dev/maddy/ https://foxcpp.dev/maddy/ mentioned in this conversation may simplify things but with IMAP and SMTP inside one process you'll get a hard time doing this separation, not a good point for security. * perhaps advanced mail routing like user-selectable option to automatically create mailboxes, per mailing-list and envelope address, although technically we could start another conversation about this. (My current setup does that, looking for hints but perhaps I have more to share than to learn this time. One pain point is many senders generate uninformative List-Id or even change them at each message, even Mozilla for example, making things messy.) My current setup runs Postfix for SMTP on one host, doing all the rejection of spam and forwards approved messages via LMTP to another machine holds the private mailboxes contents to be served via IMAP by Dovecot. Plus custom recipient delimiter instead of the traditional "+". And all point above except being Dockerized.
muppetman 6y ago
Make sure you add yourself to dnswl.org if you're running a mailsever. Lots of people who use Spamassassian will check it (by default, it's part of the base install) and if you put yourself in there you will score yourself a few negative figures to help get your mail delivered. Of course DMARC, DKIM and SPF are key as well. I've run my own mailserver for the last 20 years and properly signing messages and being sure not to accept/forward spam (i use rspamd) has helped me be able to have a stable personal server all those years. The only places I have trouble with are Microsoft, because it seems if you don't send a mail every 24 hours your reputation gets "reset" and it can be hard to deliver into them.
coronadisaster 6y ago
Gmail still sends good emails to spambox even today... For example, a realestate agent sent documents with really personal info in them and it still got qualified as spam
- Someone1234 6y ago
  Without more information it is hard to say if that is a mistake. Sometimes when businesses send sensitive documents they use a third party "secure" service, if the realtor's domain isn't configured correctly (common) to allow that third party to masquerade, it would correctly be flagged as spam. Plus I get actual spam (and even generated blackmail) with personal information in it (inc. old passwords), farmed from one of a dozen service break-ins over the last 20+ years (thanks Adobe, amongst others).
  - throwaway2048 6y ago
    If the user wanted the email, its a mistake if it went into the spam box, period. Trust me when i say you can have a 100% flawless email setup in every technical respect (the full works from this article and more) and still end up spam boxed by Gmail approximately 100% of the time. Conversely, people that Gmail feels are important wont get spamboxed for minor technical reasons either...
- icedchai 6y ago
  Email / domain reputation is very complex. Their domain may have been flagged due to excessive marketing emails, so no matter what they send they are heavily weighted towards spam. I worked at a SaaS company that had about 50 sales reps. They were sending emails day and night, eventually getting the main corporate domain flagged. The product's transactional emails (signup, confirmation, etc.) would be flagged as spam, until we moved them to their own domain.
inopinatus 6y ago
Not a bad list of advice. I'll add a few things for infrastructure: * Setup ARC (RFC8617) for all mailing lists and forwarding. * Sign up for the Google & Microsoft postmaster tools. They're not great, but it's better than nothing. * Implement RFC8058 / RFC2369 unsubscribe mechanisms. * Use an outbound suppression list to ensure that unsubscribe requests or unwanted mail complaints have lasting effect. * Article touches on DMARC reporting, for which I suggest Postmark's free monitoring tool at https://dmarc.postmarkapp.com/ https://dmarc.postmarkapp.com/ * From time to time run your domain name(s) and IP address(es) through a panel of reputation services. Check out any red flags (some of them matter, some don't). And for content: * Re-evaluate email templates. Many out there are hot garbage, practically inviting users to hit the junk button, which just hurts sender reputation. Ensure there's always a plaintext part. Don't use tracking/opening pixels. Write validating HTML. Minimise size. Don't bury the unsubscribe link, put it at the top, and make sure it clearly works without hostile barriers like "are you sure" pages or sign-in. * In the very first paragraph, and without scrolling on a mobile, we must explain, honestly and concisely, why we are contacting someone, and why they might want to read the rest, because we have no entitlement to someone's attention. * Run all mailers through a spam scoring tool as part of CI/CD, because a high score is a bug. In addition, Google's sender guidelines are well worth reading, they're more than just self-serving gatekeeping: https://support.google.com/mail/answer/81126?hl=en https://support.google.com/mail/answer/81126?hl=en
shirro 6y ago
Running a personal mail server in these times is a little bit complicated but well within the capabilities of anyone who manages servers for a living or even advanced hobbyists. That doesn't mean it is a sensible thing for most people to spend their time on. The alternatives are both very good and very affordable. I do it because this isn't any harder than programming or administering other server technologies and the only way to keep skills current is to actively work on real systems. Many things on that list are not necessary to get emails accepted by the big players. There are many email servers with bad or outdated configurations that still work. To keep emails flowing to outlook/hotmail/live etc you need to sign up to SNDS and their junk mail reporting program and if you haven't sent emails there regularly enough to maintain a reputation expect to be blocked for no reason. If that happens there are ways to escalate the issue. The basics you must do is secure your email server. Don't relay email. Have all senders authenticated over secure connection. Web email forms are a bad idea. You don't want to have to recover from being on everyone's block list. If you do any sort of mass mailing you would probably be better off moving it to a specialised email marketing platform. I handle some application related emails myself and you have to be prepared to check delivery problems and follow up. The ones I do are B2B, opt-in, relatively low volume, small number of business email systems who can whitelist and are expecting emails which is just about manageable. The bare minimum to be a good email sender is to have your mail name, hostname and reverse ip all match up and to advertise SPF. I would recommend also signing with DKIM, advertising a DMARC record as these are very widely checked and not that hard. They generally will give a slight negative spam score although the downside is you get the occasional business that doesn't know how to configure their mail system to work with third party filter (eg mimecast). They may reject emails according to your dmarc policy for correctly recognising that your email has been intercepted and tampered with but usually those sort of mistakes get discovered and fixed. If you use an intermediary like this to validate your emails you have to turn off validation of these things on your own server. It probably won't be long until MTA-STS is as widely supported by big emailers and used as a reputation signal so it doesn't hurt to set up a sensible cipher suite and a certificate on your server even if you don't care if emails are sent over tls or not. These addons are mainly a hassle because they all rely on dealing with lots of separate services - mail servers, certificate authorities, web servers, dns servers but this is because smtp didn't anticipate the abuses of the modern world. I have dnssec and DANE as well but it is just for my own amusement like trying to collect all the moons in Mario Odyssey. No big email providers even look at it as far as I know.
WarOnPrivacy 6y ago
For a long time, the high-risk crap-spewing networks tended to be the same bad actors (eg: OVH, Digital Ocean, Psychz). This year though, most of my dodgy SMTP traffic is coming 3 new problem children - Gmail(spam), Amazon(spam, other) and Microsoft Azure(other). "Other" here is anything besides spam, like dictionary attacks, transactionless connections or service probes.
urda 6y ago
I'm not brave enough to run my own mail server, it's just such a critical service to me.
blickentwapft 6y ago
I’m pretty technical but the one thing I’ve sworn off is DIY email servers. It’s not worth the pain.
TonyTrapp 6y ago
Microsoft (Live/Hotmail/Outlook/...) is the single worst offender against small mail servers. I have run a mail server for the last ten years spam-free and vulnerability-free. It has never been on any spam blacklists, delivery to GMail is just fine. I have implemented everything from SPF to DKIM to correct rDNS. Yet, my IP address predictably ends up on Microsoft's "blacklist" every two months or so because of "spam behaviour or user complaints". I am rather sure that neither is true. Every time I have to fill in their appeal form, get unlocked immediately but have to do the same dance again two months later. They have every right to reject a mail server but at least they shouldn't lie about the reason. My server's IP address gets blacklisted again and again even during months I don't send any mail to Microsoft addresses.
- mekster 6y ago
  Maybe they have a rule that unless you keep sending clean emails to them periodically, you get blacklisted.
Tepix 6y ago
Sovereign ( https://github.com/sovereign/sovereign https://github.com/sovereign/sovereign ) takes away most of the work of setting up an email server. The project needs volunteers!
signal11 6y ago
I set up a personal mail server (SMTP+IMAP, no webmail) a couple months ago for personal/family use —- no sense in paying about $60 per user per year for GSuite or 365. (Microsoft actually offer a custom domain email solution for families that doesn’t charge per user provided you host your domain with GoDaddy, but it has some limitations, eg with respect to aliases, so that wasn’t quite right for me.) I did so with some trepidation, having been told by multiple people that I shouldn’t. Maybe I got lucky and had an IP with good reputation, but I was able to send mail to Gmail, Outlook, Yahoo and specific email lists after a few rounds of configuration. The only problem I had was with Apple’s iCloud email, but engaging with ProofPoint soon fixed that. Overall it wasn’t too bad an experience. I’d class myself as beginner level in that I’ve done this for myself ages ago (but never in a professional capacity). Just leaving this anecdote out here as a small counterpoint to all the horror stories out there. If you’ve got an IP address with a poor reputation I can imagine it’d be a much more difficult exercise.
- bluehatbrit 6y ago
  Would you mind sharing if you're using a cloud provider or something hosted from home/co-located?
  - signal11 6y ago
    It's a small droplet on Digital Ocean.
mark_l_watson 6y ago
Very interesting article, but hosting my own email is a fantasy for me that I will probably never spend the time setting up and maintaining. ProtonMail does everything I need in an email service, with FastMail and Hey (which I only tried for a week) being excellent runners-up. Sorry for going off topic but everyone should host their own web site(s) and stop putting their content on other people's platforms.

401 points by ebcase 6y ago

I personally use zoho.com(free plan) for email to and from my domain. What is the use of self-hosting a email server over using Zoho or Gmail?

acidburnNSA 6y ago
I like hosting mine for a few reasons: - It's a hobby and I enjoy it - It like fiddling around with advanced features, like wildcard addresses, server side filtering (via dovecot-sieve), and other fun stuff. For instance, a friend once was mulling on facebook on detecting tone of email and rejecting ones that were too mean. I made a simple python script that attempted this and had her try it out. It was funny. (See previous point about hobby) - I figure that being on your own server you're at least slightly less likely to be a victim of a big internal platform hack like twitter's recent one, or state-run ones. - It feels slightly badass. I don't recommend it to people who don't want to do it for fun.

elric 6y ago

Not sure if I want to carry on host other people's email anymore, it's become a huge pain. It's bad enough that Google and MS frequently refuse to deliver mail; without any apparent reason or any humans to discuss the problem with. But now that zimbra is dead (no new open source versions), email hosting is pretty much back to where it was in the 00s.

juskrey 6y ago

Mailinabox handles all of that automatically and nicely. You just need to do some manual DNS work and check your IP against blacklists on http://multirbl.valli.org/ http://multirbl.valli.org/ - and mostly done.

jeffbee 6y ago
I think part of the point of the post and the PDF to which it links is that Gmail and the other big operators don't use the blacklists, so for deliverability to the majority of recipients it makes no difference whether you are or are not on those lists.
- jeffbee 6y ago
  Example of how the blacklists contain essentially no information: every one of Gmail's outbound VIPs is in a SORBS blacklist. Every one I checked is in at least 20 other blacklists. So if you just don't want to get mail from hundreds of millions of people, subscribe to IP blacklists.
  - johnklos 6y ago
    That's not true, unless "VIP" means something that I don't understand. I use SORBS on a rather busy mail server with thousands of messages sent to and from Gmail daily. Aside from very occasional SORBS rejections, which happen more often with Outlook.com and Yahoo, this simply isn't the case.
    - jeffbee 6y ago
      See for yourself: http://multirbl.valli.org/lookup/209.85.128.65.html http://multirbl.valli.org/lookup/209.85.128.65.html
      - e12e 6y ago
        Seems to be mostly not listed?
        jeffbee 6y ago
        Sure but blacklists are binary. That’s why they don’t work.
        6y ago

davemtl 6y ago

With all the fixes and standards just to send a cute cat GIFs to your friends on Gmail is such a pain in the butt for most self hosters. If you're doing it, great! Keep it up! All the better for you. But for me, using a e-mail hosting provider with a good reputation is preferable, they've already done all the hard work (my preference is FastMail, but it's not everyone's cup of tea). I've been in the IT field for almost 20 years, I've done the self hosting e-mail thing. In my early teens and 20s, I didn't have anyone to please or take care of, so keeping up was easy with new e-mail standards as they rolled out, 15-20 years later, I'm divorced, work full time and have two kids, I simply don't have the time to self-host anymore. I'd rather give somebody a few $ a month to take care of that for me.

hkt 6y ago
I've been so irritated with self hosting I've gone the other way: I'm in the process of setting up an email hosting co-op, with the aim of paying for managed service from a small hosting provider for most things but paying no more than I do now. That way it won't only fall to me to deal with deliverability issues etc.
- saimiam 6y ago
  See my sibling comment to yours! I think we are both working towards the same goals. Nice to meet you.
saimiam 6y ago
> I'd rather give somebody a few $ a month to take care of that for me I'll do it though i'm not FastMail! I'm working on a SES backed email client so SES takes care of deliverability (assuming the user does the basic DKIM, MX, TXT set ups) so it will work out much cheaper to send and receive mails than a FastMail/GSuite type solution. Plus you'll get unlimited disposable emails for free, a shared inbox (if you want to share an inbox with your family to help stay on top of every one's schedules, for exam), and while the client I have built is pretty rudimentary, i'll be adding bells and whistles to make it better. Github @ https://github.com/saiorama/ses-email-client https://github.com/saiorama/ses-email-client Landing page to submit a request @ https://shared-inbox.landen.co/ https://shared-inbox.landen.co/ Note to others - I'm open to collaborating to take back email from the big guys. Hit me up via Github.

jks 6y ago

Addendum: if you host a mailing list server such as Mailman, you'll need to do one of (1) don't change the content or DKIM-signed headers at all (e.g. by adding Subject tags or unsubscribe instructions); (2) rewrite the From: header so that you don't look like you are impersonating the individual senders; (3) use ARC [https://en.wikipedia.org/wiki/Authenticated_Received_Chain https://en.wikipedia.org/wiki/Authenticated_Received_Chain] signatures. I don't know if Gmail and the other big providers actually honor ARC headers yet, but if they ever do, that would seem to be the cleanest solution if you need to add unsubscribe instructions or similar banners.

guerby 6y ago

I wonder if there's an ansible playbook implementing all that?

ireflect 6y ago
Sovereign does most of it: github.com/sovereign/sovereign I was an early contributor to Sovereign and used it for many years, but I found that we kept adding features to it and it got too large for my liking. Now I use docker-mailserver: https://github.com/tomav/docker-mailserver https://github.com/tomav/docker-mailserver
- gertrunde 6y ago
  I looked at Sovereign a while back, and came to the same conclusion, and ended up going with mail-in-a-box, which is possibly a little too far the other way, but at least very low maintenance.

evilsetg 6y ago

Thank you! I recently managed to get a static IP and was looking forward to set my own mail server to be independent of the big providers. Now I have been searching for something exactly like this, a recent, concise overview of the hoops I have to jump through to be allowed to send to the big guys.

saimiam 6y ago
I wrote one as well @ https://medium.com/@ssr233/get-disposable-emails-for-your-domain-in-minutes-8c5c8830fe7b https://medium.com/@ssr233/get-disposable-emails-for-your-do... I'm also trying to build my own email client for SES @ https://github.com/saiorama/ses-email-client https://github.com/saiorama/ses-email-client. Happy to collaborate in case you want to work on this issue.

icefo 6y ago

Tangentially related but I'm having some issues related to email beeing refused by Gmail. In my organization we have ~10 linux VM and they are all configured to send email to an exim MTA. For example monitoring stuff. The system mail name is set to vmName.srv.domain.name (domain.name is replaced by our actual domain and this resolve to a local address) so it's quite common to receive mail from root@vmName.srv.domain.name. This cause no issue when it's delivered to the local Cyrus server but when it's transfered to a Gmail address for some reason Gmail reject the email because the headers are bad. I am considering rewriting the address to user.vmName.srv@domain.name in exim when receiving email from local VMs and I am wondering if this could be a bad idea. I'm going to redo this mail server from scratch soon because it has been left untouched for something like 6 years. Some software was compiled on it with shared libraries which then were updated but the software was not recompiled. It's a dumpster fire and I'm actually surprised most of our emails seem to go through. I would also add to the article that's it's a good idea to have some rate limit on outbound email, outbound email spam checks and to check that the user sending email has the right to use that address.

johnchristopher 6y ago
Quick question (as someone who doesn't manage email): why not using an external mail provider (like mailjet) ? Privacy concerns and/or internal mails that shouldn't get leaked ?
- icefo 6y ago
  That server is at least 10 years old and quite custom. For example when an email is received the address is queried to an ldap and if it's a mailing list the email is sent to all the recipients of that mailing list which can be other mailing lists ! This goes on until an user address is found and then the email is delivered to the cyrus server. Each user can also have many alternative addresses that map to an unique mailbox Those mailings lists are used a lot internally and managed with internally developed tools. This make migrating to an external service difficult. That's the classic example of some critical piece of infrastructure that nobody dare to touch because it works... until it doesn't anymore. That's why we don't use external services to receive email. I'll consider using mailJet if setting up exim & related stuff correctly to send email in 2020 proves too painful.

pgrote 6y ago

Running an email server became a real pain about 4 years ago. Until then, we could run our own exchange server, control everything and it was awesome. Then came the email security products from the large companies. It became a self fulfilling prophecy. Our email would trigger a rule in big co's third party security app, which would then report to a centralized rule breaker clearing house automatically. Clients couldn't receive our emails, we would dive in, submit appeal to clearing house, get clear and a week later do it all again. Everything was configured properly on our end. We passed all the online validation engines. IPs were our own personally owned block and pristine. It became too much. Switched to office365 and all problems magically disappeared. Sent emails to same big cos with third party email security and haven't had a single issue.

colordrops 6y ago
Is the implication that this is an intentional practice to drive out small players?
- jandrese 6y ago
  I don't think it is an attempt to drive out small players, it's a natural consequnce of domains being almost free to stand up. They just don't provide any level of assurance that the owner isn't a spammer. So the blackhole systems treat them as "probably spammer" and fire up a block at the first whiff of malfeasance. It's harder to sign up for a Gmail account than it is to register a domain and get some hosting. And Google has their own protections against sending large amounts of mail built-in. The system is as it is now because spammers will abuse it otherwise.
  - throwaway2048 6y ago
    gmail is effectively block by default these days, unless Google has some reason to not block your mail, it will be spamboxed.
    - bachmeier 6y ago

rsync 6y ago

Ugh. I have been running my own UNIX mailserver since 1997. I have consistently and loudly evangelized this practice, especially here on HN, and pushed back against the notion that it was too difficult or too time-consuming to implement and maintain one's own mailserver. But ... Having just recently made the major transition from relying solely on clean IP space and proper DNS records (which, in the last 18 months, has become increasingly untenable[1]) I have changed my mind. I will continue to run my own mailserver, for activist, ideological reasons, but I must now agree that it is too difficult and complicated - and fragile. In no particular order, some of the things that I find overly complex and disturbing are: - The DKIM implementations are very, very over-engineered. I understand, in principle, why we want DKIM to be a pluggable component that can be used as a general building block in various implementations, blah blah blah, but there is no reason that DKIM can't just be a feature, with a corresponding blob of config lines, in sendmail or OpenSMTPd or whatever. Having to pkg-install, and track, and maintain, whatever DKIM implementation you choose, along with all of its various dependencies, etc., is a big truckload of complexity and fragility that I just can't see ever appreciating. And the irony ? A popular DKIM implementation is to use the bundled DKIM functionality built into rspamd (!) ... so it ended up being just a feature anyway. - The whole SSL component ... I appreciate this, I understand this, and for many use-cases I just don't care. I don't currently need encrypted email and if someone makes a decision to disallow sending to addresses/domains that don't support it, fine. Of course, if all of gmail decides not to, now I have another giant truckload of complexity and fragility that I need to implement and maintain. My mailserver should be an extremely tight, stripped down host with as few packages and daemons possible[2]. Instead, I am now installing a big list of packages just so that letsencrypt can fire up a temporary web server to clicky click make my certificates[3]. - Now my mailserver is running a database (Redis) which is required to run rspamd, which is required to implement DKIM, without which I cannot send mail to yahoo.com addresses. True story. So much complexity and fragility ... [1] About 18 months ago yahoo.com stopped, with no errors or bounces, delivering mail from an 11 year clean IP with proper DNS/MX entries. I implemented DKIM and the problem is solved. [2] My mailserver is a FreeBSD jail and previously ran only sendmail - and nothing else. Now I am running OpenSMTPd, rspamd, redis ... and once in a while I am running a webserver to communicate with letsencrypt. [3] Speaking of letsencrypt ... requesting, generating and implementing SSL certs (even for use on the web) is simply creating, and trading, blocks of plain old ASCII text. I haven't looked into this, but is there an "expert mode" somewhere in letsencrypt where I don't have to run webserver instances and ... I can just run openssl command lines and cut and paste blobs of ASCII ?

niftylettuce 6y ago

I made Forward Email after running into many issues with self-hosted solutions (and realized thousands of others have the same issue due to the high-demand after launch back in 2017). It's a great alternative to having to set up your own small mail server. https://forwardemail.net https://forwardemail.net P.S. I'm coming out with a fully-fledged email service built on top of my R&D with Forward Email.

CraftThatBlock 6y ago
Any cheap paid services for this? I'm not comfortable with routing my emails through a free third-party.
- dejj 6y ago
  Then how are you comfortable routing it through a cheap service instead? Asked more precisely: how do you determine reliability from price?
  - namibj 6y ago
    It's more that you tend to get some legal implications from selling a service vs. having some website with no payment on the net.
- niftylettuce 6y ago
  We do offer paid plans with enhanced features. And we're completely open-source, store zero logs, and store zero metadata too. All source code is on GitHub as well. I formerly worked at DuckDuckGo, and take privacy very seriously. Privacy Policy: https://forwardemail.net/privacy https://forwardemail.net/privacy Terms: https://forwardemail.net/terms https://forwardemail.net/terms You can delete your account at any time.
- 627467 6y ago
  Forwardemail.net seems to have paid plans. Reading through the about and FAQ they also seem to be driven by decent principles. https://forwardemail.net/en/faq#how-is-it-free https://forwardemail.net/en/faq#how-is-it-free

Cyph0n 6y ago

I went through the journey of configuring Postfix for a small side project I was working on. The idea was to accept emails sent to a unique, per-user address and forward all attachments to the user’s cloud storage account. I looked into using mail services like SES and Mailgun, but given the average expected email size and the fact that I would primarily receiving mail, the additional cost didn’t make sense. Along the way, I ended up learning quite a bit about Postfix filters, virtual domains, SPF, and DKIM. I also learned how quickly spammers can detect open relays. Apparently, while copy-pasting a config from somewhere, I had a slightly incorrect 172.x subnet listed as part of “mynetworks”. As a result, a spammer with a machine belonging to this subnet was able to use my server as an open relay. Long story short, it took me quite a while to root cause the issue and get my domain off the blacklists!

tgsovlerkhgsel 6y ago

There really needs to be a) a docker container (or other standardized, easy to deploy technology) with a "mail server in a box implementing best practices" b) a test for the stuff that Docker can't cover like DNS config, ideally with scripts to set it up on the most popular providers and clear copy&paste instructions for the rest. People shouldn't need to fully understand DKIM, SPF, etc. - they should just be able to click a button, copy&paste generated DNS records, and be up and running. Otherwise, commercial providers are just the only option that makes operational sense for most cases, _even if you know how to do it right_, because doing everything right manually step by step is still to much work to be worth it in many cases. None of this seems _hard_, it's just tedious.

pbhjpbhj 6y ago
Agreed. A few years back when moving my personal domain from a failing provider I considered running my own email server on Digital Ocean; they had some great tutorials on setting up various parts and I followed a HN recommended tutorial. I was already happy setting up LAMP with a nginx reverse proxy, how hard could it be - I got it setup, and realised two things: Keeping everything running without getting compromised was going to take way too long compared to the benefit. Two, Google and MS wouldn't accept my email - not even to myself when I whitelisted it - it's a crap shoot. Email has serious economies of scale and keeping such a critical system operational, on a small scale, was going to be some deal of stress. So I just went with my hosting providers Plesk-fronted setup, which matches your para.3 description.
gsich 6y ago
>People shouldn't need to fully understand DKIM, SPF, etc. - they should just be able to click a button, copy&paste generated DNS records, and be up and running. I oppose this. People should know what they are configuring. This applies to all stuff related to mailservers. You probably learn more by setting it up the "hard" way (without containers) so that in case of failure you at least have a little insight in how the components work.
- im3w1l 6y ago
  There are many more people who want to drive a car, than people who want to learn the minutiae of how it works. If it breaks, then you get someone else to fix it / get a better product is all they need to know.

cm2187 6y ago

Point 14 is a very good one. If any account gets compromised, you might as well request a new IP as your previous IP will be in the dog house for a long time. Almost more important than monitoring failed logons, monitor successful logons, for instance by country. If it is a small mail server, chances are your users are in a handful of countries. If you suddenly see a successful logon from vietnam or brazil, you probably want to be notified ASAP.

davidbrennerjr 6y ago

Eh, I didn't read it...so my thoughts. From experience the prob with DNSSEC is latency in verifying PKI of the record. Usually what happens is the timeout per resolver has to be greater than 15 seconds under no load in a private network which is very inconvenient in comparsion to plain DNS that has a default of 5 seconds.

thoraway1010 6y ago

The volume of spam is mind boggling. When we first implemented DMARC / DKIM etc our legit emails were 3% or so of all outbound mail per the reporting we got back! We have a somewhat trusted domain. Some mail delivery systems start to notice that your domain name is being used as part of a lot of spam / bogus emails, so you can have perfect IP history / no spam and STILL start triggering some random filters (no big players but smaller protection product filters). So the game must be absolutely never ending for everyone and the inbox a valuable target - especially now that unsolicited phone calls really do seem to get ignored these days - I feel like spammers killed the golden goose on phone calls and the telcos let them. I will say DKIM / DMARC is working well, except google (which we now use for outbound) gives us transient SPF errors even though we are 100% using their IPs. Not sure why that is (ie, SPF failure on an IP that should clear)

dorfsmay 6y ago
For your last point, check https://toolbox.googleapps.com/apps/checkmx/check https://toolbox.googleapps.com/apps/checkmx/check. Chances are, google doesn't like your DNS server answers to its ANY requests, regardless for them complying to the RFCs.
Geezus_42 6y ago
As someone who works in email security I can tell you that it is a never ending cat and mouse game with a lot of false positives as to who is a mouse. Even our customers that have good domain rep, SPF, DKIM, DAMRC, etc still randomly get blocked by filters and large email providers. Part of the problem I see it those large providers and some of the filtering systems send back NDRs that basically provide no information as to why the message was blocked so trying to get your legitimate mail flow working again once someone has flagged you can take days to months as you randomly adjust all the knobs trying to figure out why they are blocking you. I'm sure they would say providing more informant NDRs would make it easier for the spammers to work around the filters. I don't know what the answer is but the current system sucks, it is a huge pita and a dumpster fire for the most part but it's what we have to work with. I'm sure someone will come up with another scheme like DKIM that will become another check box on the list of things you have to set up and once agian few will understand how to implement correctly.

s_gourichon 6y ago

TL;DR: Cool that this thread exists because people will mention (please do!): * Docker-based e-mail setup, with comments about their experience, like https://github.com/tomav/docker-mailserver https://github.com/tomav/docker-mailserver (which incidentally IIRC was recently looking for a new maintainer). * ...ideally that would have one host running SMTP facing the big bad public Internet and therefore minimalistic, and a separate more isolated host that runs the IMAP server. (My current setup does that with, looking for hints but perhaps I have more to share than to learn this time.) -- https://foxcpp.dev/maddy/ https://foxcpp.dev/maddy/ mentioned in this conversation may simplify things but with IMAP and SMTP inside one process you'll get a hard time doing this separation, not a good point for security. * perhaps advanced mail routing like user-selectable option to automatically create mailboxes, per mailing-list and envelope address, although technically we could start another conversation about this. (My current setup does that, looking for hints but perhaps I have more to share than to learn this time. One pain point is many senders generate uninformative List-Id or even change them at each message, even Mozilla for example, making things messy.) My current setup runs Postfix for SMTP on one host, doing all the rejection of spam and forwards approved messages via LMTP to another machine holds the private mailboxes contents to be served via IMAP by Dovecot. Plus custom recipient delimiter instead of the traditional "+". And all point above except being Dockerized.

muppetman 6y ago

Make sure you add yourself to dnswl.org if you're running a mailsever. Lots of people who use Spamassassian will check it (by default, it's part of the base install) and if you put yourself in there you will score yourself a few negative figures to help get your mail delivered. Of course DMARC, DKIM and SPF are key as well. I've run my own mailserver for the last 20 years and properly signing messages and being sure not to accept/forward spam (i use rspamd) has helped me be able to have a stable personal server all those years. The only places I have trouble with are Microsoft, because it seems if you don't send a mail every 24 hours your reputation gets "reset" and it can be hard to deliver into them.

coronadisaster 6y ago

Gmail still sends good emails to spambox even today... For example, a realestate agent sent documents with really personal info in them and it still got qualified as spam

Someone1234 6y ago
Without more information it is hard to say if that is a mistake. Sometimes when businesses send sensitive documents they use a third party "secure" service, if the realtor's domain isn't configured correctly (common) to allow that third party to masquerade, it would correctly be flagged as spam. Plus I get actual spam (and even generated blackmail) with personal information in it (inc. old passwords), farmed from one of a dozen service break-ins over the last 20+ years (thanks Adobe, amongst others).
- throwaway2048 6y ago
  If the user wanted the email, its a mistake if it went into the spam box, period. Trust me when i say you can have a 100% flawless email setup in every technical respect (the full works from this article and more) and still end up spam boxed by Gmail approximately 100% of the time. Conversely, people that Gmail feels are important wont get spamboxed for minor technical reasons either...
icedchai 6y ago
Email / domain reputation is very complex. Their domain may have been flagged due to excessive marketing emails, so no matter what they send they are heavily weighted towards spam. I worked at a SaaS company that had about 50 sales reps. They were sending emails day and night, eventually getting the main corporate domain flagged. The product's transactional emails (signup, confirmation, etc.) would be flagged as spam, until we moved them to their own domain.

inopinatus 6y ago

Not a bad list of advice. I'll add a few things for infrastructure: * Setup ARC (RFC8617) for all mailing lists and forwarding. * Sign up for the Google & Microsoft postmaster tools. They're not great, but it's better than nothing. * Implement RFC8058 / RFC2369 unsubscribe mechanisms. * Use an outbound suppression list to ensure that unsubscribe requests or unwanted mail complaints have lasting effect. * Article touches on DMARC reporting, for which I suggest Postmark's free monitoring tool at https://dmarc.postmarkapp.com/ https://dmarc.postmarkapp.com/ * From time to time run your domain name(s) and IP address(es) through a panel of reputation services. Check out any red flags (some of them matter, some don't). And for content: * Re-evaluate email templates. Many out there are hot garbage, practically inviting users to hit the junk button, which just hurts sender reputation. Ensure there's always a plaintext part. Don't use tracking/opening pixels. Write validating HTML. Minimise size. Don't bury the unsubscribe link, put it at the top, and make sure it clearly works without hostile barriers like "are you sure" pages or sign-in. * In the very first paragraph, and without scrolling on a mobile, we must explain, honestly and concisely, why we are contacting someone, and why they might want to read the rest, because we have no entitlement to someone's attention. * Run all mailers through a spam scoring tool as part of CI/CD, because a high score is a bug. In addition, Google's sender guidelines are well worth reading, they're more than just self-serving gatekeeping: https://support.google.com/mail/answer/81126?hl=en https://support.google.com/mail/answer/81126?hl=en

shirro 6y ago

Running a personal mail server in these times is a little bit complicated but well within the capabilities of anyone who manages servers for a living or even advanced hobbyists. That doesn't mean it is a sensible thing for most people to spend their time on. The alternatives are both very good and very affordable. I do it because this isn't any harder than programming or administering other server technologies and the only way to keep skills current is to actively work on real systems. Many things on that list are not necessary to get emails accepted by the big players. There are many email servers with bad or outdated configurations that still work. To keep emails flowing to outlook/hotmail/live etc you need to sign up to SNDS and their junk mail reporting program and if you haven't sent emails there regularly enough to maintain a reputation expect to be blocked for no reason. If that happens there are ways to escalate the issue. The basics you must do is secure your email server. Don't relay email. Have all senders authenticated over secure connection. Web email forms are a bad idea. You don't want to have to recover from being on everyone's block list. If you do any sort of mass mailing you would probably be better off moving it to a specialised email marketing platform. I handle some application related emails myself and you have to be prepared to check delivery problems and follow up. The ones I do are B2B, opt-in, relatively low volume, small number of business email systems who can whitelist and are expecting emails which is just about manageable. The bare minimum to be a good email sender is to have your mail name, hostname and reverse ip all match up and to advertise SPF. I would recommend also signing with DKIM, advertising a DMARC record as these are very widely checked and not that hard. They generally will give a slight negative spam score although the downside is you get the occasional business that doesn't know how to configure their mail system to work with third party filter (eg mimecast). They may reject emails according to your dmarc policy for correctly recognising that your email has been intercepted and tampered with but usually those sort of mistakes get discovered and fixed. If you use an intermediary like this to validate your emails you have to turn off validation of these things on your own server. It probably won't be long until MTA-STS is as widely supported by big emailers and used as a reputation signal so it doesn't hurt to set up a sensible cipher suite and a certificate on your server even if you don't care if emails are sent over tls or not. These addons are mainly a hassle because they all rely on dealing with lots of separate services - mail servers, certificate authorities, web servers, dns servers but this is because smtp didn't anticipate the abuses of the modern world. I have dnssec and DANE as well but it is just for my own amusement like trying to collect all the moons in Mario Odyssey. No big email providers even look at it as far as I know.

WarOnPrivacy 6y ago

For a long time, the high-risk crap-spewing networks tended to be the same bad actors (eg: OVH, Digital Ocean, Psychz). This year though, most of my dodgy SMTP traffic is coming 3 new problem children - Gmail(spam), Amazon(spam, other) and Microsoft Azure(other). "Other" here is anything besides spam, like dictionary attacks, transactionless connections or service probes.

urda 6y ago

I'm not brave enough to run my own mail server, it's just such a critical service to me.

blickentwapft 6y ago

I’m pretty technical but the one thing I’ve sworn off is DIY email servers. It’s not worth the pain.

TonyTrapp 6y ago

Microsoft (Live/Hotmail/Outlook/...) is the single worst offender against small mail servers. I have run a mail server for the last ten years spam-free and vulnerability-free. It has never been on any spam blacklists, delivery to GMail is just fine. I have implemented everything from SPF to DKIM to correct rDNS. Yet, my IP address predictably ends up on Microsoft's "blacklist" every two months or so because of "spam behaviour or user complaints". I am rather sure that neither is true. Every time I have to fill in their appeal form, get unlocked immediately but have to do the same dance again two months later. They have every right to reject a mail server but at least they shouldn't lie about the reason. My server's IP address gets blacklisted again and again even during months I don't send any mail to Microsoft addresses.

mekster 6y ago
Maybe they have a rule that unless you keep sending clean emails to them periodically, you get blacklisted.

Tepix 6y ago

Sovereign ( https://github.com/sovereign/sovereign https://github.com/sovereign/sovereign ) takes away most of the work of setting up an email server. The project needs volunteers!

signal11 6y ago

I set up a personal mail server (SMTP+IMAP, no webmail) a couple months ago for personal/family use —- no sense in paying about $60 per user per year for GSuite or 365. (Microsoft actually offer a custom domain email solution for families that doesn’t charge per user provided you host your domain with GoDaddy, but it has some limitations, eg with respect to aliases, so that wasn’t quite right for me.) I did so with some trepidation, having been told by multiple people that I shouldn’t. Maybe I got lucky and had an IP with good reputation, but I was able to send mail to Gmail, Outlook, Yahoo and specific email lists after a few rounds of configuration. The only problem I had was with Apple’s iCloud email, but engaging with ProofPoint soon fixed that. Overall it wasn’t too bad an experience. I’d class myself as beginner level in that I’ve done this for myself ages ago (but never in a professional capacity). Just leaving this anecdote out here as a small counterpoint to all the horror stories out there. If you’ve got an IP address with a poor reputation I can imagine it’d be a much more difficult exercise.

bluehatbrit 6y ago
Would you mind sharing if you're using a cloud provider or something hosted from home/co-located?
- signal11 6y ago
  It's a small droplet on Digital Ocean.

mark_l_watson 6y ago

Very interesting article, but hosting my own email is a fantasy for me that I will probably never spend the time setting up and maintaining. ProtonMail does everything I need in an email service, with FastMail and Hey (which I only tried for a week) being excellent runners-up. Sorry for going off topic but everyone should host their own web site(s) and stop putting their content on other people's platforms.