7 ms·
Why it’s so hard to innovate in the e-mail space (2014)
- xs 11y agoI'm trying to build an email platform too, yes it's hard. But I thought it was only because I'm a beginner and 1 man team. Some of the reasons why I think it's hard: * I don't think security was a big deal to many of the mail applications when they were being built. I tried and gave up so many times trying to find a strong hashing algorithm that both dovecot and ruby supported. There was practically no documentation on this and it felt like I was venturing into territory never before seen. * Parsing emails to be display correctly seems impossible. Sometimes emails have "<br>" in them and sometimes they have "\n". But what if the "\n" doesn't mean newline but it's just what someone wrote in the email? They come in a variety of mime types and formats. It sometimes seems impossible to do this. * Not everyone uses the RFC standards! I thought the RFC said that a subject can only be 78 chars long. Yet I get emails all day long that go way beyond this which cause major problems in my code. AM I THE ONLY ONE AROUND HERE THAT CARES ABOUT RFCS?
- feld 11y agoLook at http://archiveopteryx.org/badmail/ http://archiveopteryx.org/badmail/ for your storage backend. Strict RFC implementation, email normalization. Bad emails won't kill your email client 30 years from now.
- timClicks 11y agoOn your last point, email subjects are created by end users. I don't think I've ever had a mail client complain that my subject was too long and refer me to the RFC. Pretty much every application developer will prefer complying with their users' preferences rather than specs.
- zAy0LfpBZLC8mAC 11y ago> Sometimes emails have "<br>" in them and sometimes they have "\n". One of those should have an HTML MIME type, one should have a plaintext MIME type. The decoding is specified in detail in RFCs and W3C recommendations, please follow those rather than try to implement this using try and error. > Not everyone uses the RFC standards! That is unfortunately true, ... > I thought the RFC said that a subject can only be 78 chars long. ... and one of the major reasons why is because people think but don't read. There exist all kinds of crazy myths about the content of RFCs, which is how all those broken implementations arise, this seems to be one of them--but feel free to point out which RFC says this where, in case I really have missed it.
- CM30 11y agoNice article. But I think there are a few other things they missed out that make building an email client so difficult: 1. Consistency. Okay, this is always going to be a problem, since rendering anything more than plain text in emails is a minefield that makes the bad old days of Internet Explorer vs Netscape Navigator look like a utopia. But people still expect their Amazon/eBay/Google/whatever account/confirmation emails to look decent, so you have to figure out a basic HTML parser as well. And then try and make it somewhat work with people who are designing their messages with stuff like Outlook's Microsoft Word engine in mind. 2. Spam. I'm surprised this wasn't really mentioned in the article, but it's arguably the biggest issue any email provider, client developer or email server software maker has to deal with. You have to make sure your software can figure out what messages are unwanted, then either send them to the spam folder or delete them. Most startups don't have to figure that out, since their systems are closed and spam prevention can be done on sign up. Not so much if you're building an email client where any Tom, Dick or Harry can send messages to anyone with no real way to verify if they're human or not. So yeah, few other issues.
- xs 11y agoUnless you're building an MTA from scratch, I feel like the spam problem is solved by addons the MTA has, like SpamAssassin for Postfix.
- feld 11y agoIf I ever get a chance to revamp an email system I will probably use rspamd instead. Those "add-ons" are painful. https://rspamd.com https://rspamd.com https://rspamd.com/rspamd-slides.pdf https://rspamd.com/rspamd-slides.pdf
- baudehlo 11y agoCheck out Haraka, built by an old SpamAssassin developer (ie me). It's all that and has custom anti-spam features like the Karma plugin too. It's in place at some large installs (eg craigslist, who went from 20 postfix servers to just 7 Haraka servers).
- vonklaus 11y agoThere are tons of reasons why this is so hard. Leaving aside all of those there are 2 that I see as the biggest: * A 100x improvement over current email is needed for a switch. * It has to be 100x cheaper than the current product. While you can't multiply by 0 and I meant that facetiously, users won't pay for the software or only niche consumers. Which is why email providers that get paid focus on things like security, collaboration and and backup. These products are essentially 90% other things and 10% email. examples: * dropbox * slack * telegram * silent circle * sms obviously I am not going to name everything in the space. I can't think of a company that has email at it's core. maybe 'hushmail' or some small encrypted mail providers but what profitable business is at it's core an email provider
- AlexB138 11y agoMailChimp? Constant Contact? I may be wrong, I'm not terribly familiar with either, but I believe they're largely focused on email.
- bradlys 11y agoMailchimp and such aren't for everyday emailing between you and me. They're for emailing large lists of users one email a day or week or whatever and using tracking information based off referral links users click on in the email.
- wodenokoto 11y agoWhy do people blog about what their company is building without so much as mentioning the name of their company or product or even a hyperlink? In the author profile at the very bottom of the page, in small letters, there is a URL (but not a link) to https://frontapp.com/ https://frontapp.com/ and the email client the author works on is apparently called Front. But why should her readers ever care about that?
- mathouc 11y agoYou're right. When I wrote this article I was more interested in replying to Des Traynor's article than promoting our product. I've added a link (even if I'm not sure people care)
- horsecaptin 11y agoFront is a great product. It lets us manage emails and SMS support together!
- mikhailt 11y agoIt will be easier to build a new email spec that's more feasible/modern with security built in mind than it is to build a sustainable email client that tries to comply with various email services (GMail is the worst offender for not being consistent with IMAP spec). Mailmate is my fav OS X client that focues more on power user features but it has a boring classic interface which doesn't bother me at all. Despite having integration support with SMIME/PGP, Markdown reply, powerful search features, and so on that makes me productive, the only thing I hear when I recommend this to other people is that its UI is ugly. :|
- akvadrako 11y agoIt's not as ugly as Thunderbird or a web-app. At least it follows most of the OS X conventions. Anyway the existence of Mailmate shows how the premise in the article is totally wrong - it's not hard to create a decent modern email client - it already exists and only takes one developer. A better title would be: Why is it so hard for users to use a good email client?
- Animats 11y agoIt's hard to make money in that space, yes. I've considered merging mail and messaging, so that email protocols are used to do the things people do with instant messaging, Telegram and such. The first step is to get away from queuing mail. A first step is a new kind of mail forwarder, one for machines that don't have local mailboxes (which is most non-mail servers). It acts as an SMTP server and client. When an email comes in, and the sender reaches the end of the DATA section, the forwarder checks its forwarding list to see where this mail is supposed to go. It opens a SMTP connection to the destination and sends the email. The incoming SMTP connection is held open while this is in progress. Any problems are reported back to the sender immediately with an SMTP status code. There are no bounce emails, no queuing, and no possibility of joe-jobs. All mail is handled immediately or rejected immediately. (This only applies to single-address messages, and only to "To" addresses. Everything else is bulk.) The next step is an IMAP server which has the same immediate orientation. When it gets an incoming email, it sends a push notification to any connected clients, while holding the incoming connection open. If at least one client is responding, a SMTP success status is returned. Otherwise, you get some status such as 251 (User Not Local, Will Forward.) Then, a threaded email client. It would help to have the convention that an email with subject but no text, text but no subject, or text the same as subject is displayed like an instant message. All of these can be done independently, and are backwards-compatible. If you have all of them, email is equivalent to instant messaging - you can tell if it was delivered, it goes through immediately, and it works like a conversation. Of course, there's no big-money startup in this.
- baudehlo 11y agoYou can do the first part with Haraka. For what it's worth. I started work on an imap server but decided there was little point as nobody needs a highly custom imap server.
- zombition 11y agoI tried some similar things when making Zombition, but I found that blurring the line between email and instant message mainly results in a more complicated UI and more details for end users to remember. For example, with instant messages you generally have some status information about whether or not the other user is online, whether or not they may have read whatever you just sent them, and whether or not they're typing a reply. If you want to add that into a email interface, how do you indicate to your end users that another user will never have associated status information because their account is on a server that is incompatible with the method you proposed? Plus the behavioral differences between email and instant message didn't blend well: was I going to wait in a mixed email/instant message conversation for a response, or was I going to move on to the other conversations in my inbox and send a reply or archive each so that I could move on to doing other things? And if I wanted to temporarily leave an instant-message-like conversation so that I could deal with some more email-like conversations, would the person I was chatting with know whether or not I would return? Eventually I stepped back to look at what additional value a user gained by blurring instant message and email in this way. Fundamentally they're both asynchronous text-based communication, and the things you can actually do with them aren't strikingly different. The biggest difference is in the expectations that a user has for each. Anyway I still wanted to "innovate in the email space", so I decided to go a more extreme route, and I made my own pull-based protocol (superficial similarities to IM2000 - http://cr.yp.to/im2000.html http://cr.yp.to/im2000.html) that retains backwards compatibility with email while adding abilities like stateful plugins (JavaScript apps/widgets) that can share data through the new protocol, editing messages after they're sent (SMTP recipients receive the new version after the update), and inline replies to parts of messages to enable nested conversations. If you're interested, I posted it earlier this afternoon here (https://news.ycombinator.com/item?id=10693535 https://news.ycombinator.com/item?id=10693535).
- lassejansen 11y agoAs the article mentions Unibox: If somebody is interested in giving the beta version of Unibox for iOS a try, send me an email: lasse.jansen@eightloops.com. Unibox groups emails by person. It's different, but it really helps you staying organized. https://www.uniboxapp.com https://www.uniboxapp.com
- J_Darnley 11y agoBecause. It. Doesn't. Need. It. Email isn't broken.
- luxpir 11y agoNot necessarily broken, but certainly outdated. What do you think about spam, encryption, attachment sizes? Imagine being able to publish your email address online without worrying about spam; even in your profile you've been forced to obfuscate your own a little. Or to know that every message you send can only be read by the recipient and nobody in-between - without having to explain PGP to non-techs. Or being able to send a 50-100MB file without resorting to third party or public hosting - any of those features would be a bonus, in my view. Not broken, but can be improved, don't you think?
- J_Darnley 11y agoYou make a good point. Why would I obfuscate like that? My email is out there in git logs, mailing list logs, lists stolen from other websites, lists bought from other websites. I can't see that encryption will ever work end-to-end as we want. People are too stupid to learn most things. Technical solutions rely on big corporations that we trust to do the Right Thing(TM). The Right Thing(TM) is impossible from US companies and probably impossible from European companies. Attachments. I don't remember the last time I sent a large one. As far as I know most of the size limitations are artificially implemented by companies that don't want to spend that much on storage. Other limitations such as Gmail's block on executables is because people are stupid and will run everything they get sent. People will never not be stupid so we can never have nice things.
- hackuser 11y agoWhy are messages separated by transport? That is, why are messages sent to/from me via one transport (e.g., SMTP) read and composed in one application, and those sent via another transport (e.g., SMS, Twitter) read and composed in another? Why, as a user, do I give %#@%! what the underlying transport is? I think the focus on a particular technology (SMTP) rather than a service (messaging) is why email clients don't advance. Bizarrely, three other transports often are integrated into email clients: NNTP, voicemail, and fax. I can only guess that it's because of a completely arbitrary reason: They are older. Probably chat clients should also be integrated. I should be able to send my contacts an instant or time-shifted message.
- aetherson 11y agoBecause the different transports have vastly different abilities and limitations that are a source of frustration if you attempt to obfuscate the differences between them and aren't served by a unified UI?
- hackuser 11y agoCould you be more specific? For example, comparing SMS and SMTP, the two most prominent transports?
- zombition 11y agoSMS is used to send tiny chunks of text, and SMTP can be used to send message content in HTML format and rich text in addition to plaintext. I haven't worked much with rich text format, but HTML format for sure would be relatively impractical via SMS (XML adds a lot of characters, so you would have a lot of little chunks to send if you wanted to send HTML over SMS).
- hackuser 11y agoI agree with that. I don't mean that I want to send my email messages visa SMS. I mean that I want all my messages sent/received and read in one messaging client, whether they are transported via SMS, SMTP, etc. If the HTML message should go via SMTP, I expect my software to deal with that. I want the content; the transport is an obscure detail for developers to worry about (speaking as a typical end user).
- rrggrr 11y agoMixmax. Unrollme. Sanemail. Zapier mail parser. There is more innovation taking olace than many realize.
- tsunamifury 11y agoThis isn't thats complicated: Because email is dying and is already mostly optimized.
- JustSomeNobody 11y agoEmail is most certainly not dying.
- tsunamifury 11y agoYes it most certainly is on a secular decline. It is no longer growing at rates that are keeping up with the online user base. New internet users are no longer even setting up email accounts. Email is dying. Its not dead, but it is no longer the go-forward internet communication platform.
- abeing 11y ago> New internet users are no longer even setting up email accounts. What percentage of new internet users do not have any email accounts? I haven't been able to find any reliable data.
- oh_sigh 11y agoSeems like a strange claim, considering many sites that new internet users use will not let you sign up without an email account
- abeing 11y agoI agree. I'm skeptical and would like to see what data might support that claim. I've not been able to find any myself.
- tsunamifury 11y ago1.2 Billion Whatsapp users with phone number only 800 Million Messengers users as phone number only 20+ Apps with 100m + users as phone number only. Together these PN based winners outstrip the combined usage of almost all web apps. I mean you are all downvoting me -- but its kind of funny how out of the loop the thinking is here.
- deleted 11y ago[deleted]
- merih 11y agoSuch a great timing for this post, as Dropbox announced that they are shutting down Mailbox: https://blogs.dropbox.com/dropbox/2015/12/saying-goodbye-to-carousel-and-mailbox/ https://blogs.dropbox.com/dropbox/2015/12/saying-goodbye-to-...
- 11thEarlOfMar 11y agoE-mail as a paradigm has been optimized about as far as it can be. We need a new paradigm. It shouldn't look like e-mail. Google Wave was an attempt, but the chemistry wasn't right. Someone one will find the right mix...
- grayclhn 11y agoIsn't this taking a somewhat narrow definition of "the e-mail space"? Slack, Twitter, Facebook, Snapchat, etc. all seem like successful innovations that lots of people use to communicate instead of email.
- tmail21 11y agoAt a very high level, email is primarily used for discussions. With this view, successful innovation in the e-mail space requires innovation in the nature of discussions themselves. Tinkering around with better usability or feature improvements will not be sufficient to overcome the massive inertia of 'traditional e-mail'. We are a startup that just launched (https://tmail21.com https://tmail21.com) that aims to rethink the discussion itself. In our view, one of the major problems of e-mail (and chat for that matter) is that the discussions are not goal-oriented. So, discussions/threads can meander around with no outcome or accountability. So, we enhance the notion of discussions to be goal-oriented. Now, once one has the notion of goal-oriented discussions, a natural next step is to evolve a goal-oriented discussion into a Lean Process (which is basically a goal-oriented discussion with more structure) . Examples of a Lean Process might be a Blog Article process, a Product Deployment Process, a Feature Design Process, an Issue Escalation Process etc. We've done all this in an email-LIKE interface. I guess we'll leave it to others to decide whether this constitutes innovation in the e-mail space :)
- untog 11y agoAt a very high level, email is primarily used for discussions. I guess it depends on what you mean by "primarily" but my inbox is overwhelmingly notifications and announcements, not discussions. There's remarkably little of my inbox dedicated to actually talking to people.
- tmail21 11y agoYou are right. There is a mix of notifications and discussions with the mix varying between the two based on the nature of user's communication. In more collaborative scenarios, it may be 80%-20% discussion-notification. In other scenarios the ratio may be reversed. On the pure notification front, we have friendly tracking numbers (so that notifications can be referred to from other places like spreadsheets, chat, apps, voice etc.). These tracking numbers look something like 124-1234-1234. We also support Certified Mail, Certified Forwards, Certified Read Acks etc. Having said this, we currently support outbound notifications (from TMail to TMail/EMail). On the inbound side we support TMail to TMail. We do not currently support EMail to TMail notifications although we may add this if we see sufficient demand.
- unsignedint 11y agoThe biggest problem is that E-mail demography tends to be broader than many of the other services. Some of E-mail users are very conservative and specific about how they handle their E-mail. (Some of which are very risky behavior, which I actually tried to correct by explaining why that's bad idea, but ultmately given up...) There are demography between novice and expert. Novices tend to not care about changes so much; some even won't notice changes, as long as features are somehow accessible. Experts may have some particular tastes, but they'd be quick to find solutions to change things around when they have to. But people in between tend to be very vocal about any changes and they can't (or not willing) to find and accept "solutions" -- they will be the first one to complain when UI elements like buttons shift around the screen. Unfortunately, as universal E-mail gets, a lot of E-mail users fall under category. (Though, in different ways, "experts" can be very particular about how they handle things can be very stubborn when there's no good reason presented to change -- but it's a bit different context.) I can't even imagine how tough it will be to migrate some of people I know to anything other than what they are used to.