9 ms·

Fastmail breaks UI in production

34 points by blux 1y ago

markstos 1y ago
Fastmail is great-- solid-- for about 10 years for me. There's a lot that I prefer over Google Mail.
detritus 1y ago
I'm glad I came here to check to see if anyone else had the same issue as I've had! As soon as I realised that both my webmail and my phone app were buggered, it was probably not just a Me Problem.
- antongribok 1y ago
  Having the same problems. It started out as not being able to search, but the situation is quickly deteriorating and now I'm unable to open pretty much any email message. Some content seems to briefly show up and then it quickly disappears and after that, it's as if cache has been invalidated and you can't get back into it.
  - detritus 1y ago
    Luckily for me the Drafts folder's showing, so I was able to send (well, I assume it's sent) the Single Most Important Email I need to send today, which I'd spent about half an hour getting right just as the interface imploded...
    - toomuchtodo 1y ago
      While the web UX is borked, mobile app is not impacted if you can fallback to that until they fix the web.
      - mistertrotsky 1y ago
        This doesn't actually work. The mobile app is just a wrapper around the web interface.
        aaviator42 1y ago
        I thought apple didn't allow this? Edit: apparently "hybrid apps" using webviews are allowed as long as they're not "thin wrappers" for websites and provide meaningful functionality. See also: the capacitor framework.
        deleted 1y ago
        [deleted]
antongribok 1y ago
I opened a support case at 9:04 AM EDT. So far, no response yet.
- SV_BubbleTime 1y ago
  Maybe they don’t realize how important you are. This is a failure in their VIP program, imo they should expedite that over a bug that effects all users.
  - ocdtrekkie 1y ago
    For what it's worth the CEO responded to my support ticket once and I felt like a VIP.
    - SV_BubbleTime 1y ago
      Ha, cool. I imagine an an age of AI and LLMs, a lot of people are going to feel special at different points despite the cost of that going to nearly zero.
csomar 1y ago
It didn't break anything for me. However, I am failing to understand the point of this design "refreshment". There is nothing new out there, just minor UI changes with no purpose?
- jdbernard 1y ago
  Same. I don't hate it, but I do hate that things changed out from underneath me without any notice, opt-in period, or ability to go back.
  - RandomBacon 1y ago
    In a few days, when the frenzy dies down, I will make a feature request ticket asking them to do this next time. Please do the same.
- garciansmith 1y ago
  It makes access to the calendar, contacts, etc., one less click, which I think is nice. Would have preferred all that was set to buttons at the top instead of a thin side bar though.
  - csomar 1y ago
    I think that's how it was before? They just added the icons on the left side but you always had the menu on top?
sidcool 1y ago
I don't understand the urge to do public shaming instead of first reaching out to support. And the OP is so rude.
- jasonvorhe 1y ago
  I don't know. I'm always confused when social media accounts ask to report issues over different channels when it's so much easier to just reply to customer and ask for additional (non PII) information there. In this case it's unlikely that this was the only or even first report about this issue. So why place the initial burden on a paying customer? If your social media is just a marketing channel, at least make that clear so i don't even bother reporting issues that route.
  - yesfitz 1y ago
    Companies make it clear how to report issues by having a separate support system. If you clicked a "Support" link and it took you to Twitter, I could better understand your confusion. As for using social media to take issue reports: What will you do when you need PII or have to reassign the issue or reassign part of the issue and those people need to be able to contact the user? "Why place the initial burden on a paying customer?" Because it creates a better service for everyone to have a known way of doing things.
  - bjtitus 1y ago
    > If your social media is just a marketing channel, at least make that clear so i don't even bother reporting issues that route Is this snippet from their Twitter bio clear? "If you need assistance, please submit a ticket: http://fastmail.com/support http://fastmail.com/support"
    - jasonvorhe 1y ago
      Actually, yes. This is exactly what I would've asked for. I didn't see that.
- koakuma-chan 1y ago
ocdtrekkie 1y ago
The design refresh is actually one of the few I don't hate, I got it yesterday and it is decent. But the blank emails thing started today and this... is a problem.
SomeoneOnTheWeb 1y ago
What the hell is that with OP being so rude? And following FastMail's reply > Hello Andrew! Can you please contact our support team so we can look into this for you? fastmail.com/support They say: > Don't have time. Consider my tweet the bug report. Sorry but this asshole behavior. Bugs happen. No need to do public shaming and being rude to the company for that.
- baal80spam 1y ago
  Last time I checked fastmail was a paid service. OP has every right to expect it to be working.
  - SoftTalker 1y ago
    Sure but Fastmail's failure doesn't give people license to be assholes. There is far too much assholery in the world. It's never OK.
- dilap 1y ago
  I disgree. Other people were saying it's broken too. Reasonable company behavior here would be 1 tweet "we'll look into it", and then either "we reproduced and are working on a fix" or "looks good to us please contact support so we can investigate your particular issue". But there's no reason to initially make users jump through hoops.
  - input_sh 1y ago
    It's in their status page and it's tagged as a partial outage: https://fastmailstatus.com/cme1fq7ej002dh0iu6z8pey4f https://fastmailstatus.com/cme1fq7ej002dh0iu6z8pey4f Personally I don't really use their web interface, but I tried it now and it all works just fine (on both prod and beta).
    - ocdtrekkie 1y ago
      When I experienced this issue it was not on their status page, so I assume it wasn't for the original tooter either.
susanthenerd 1y ago
The bug (at least for me) isn't present on the beta version (beta.fastmail.com)
adityaathalye 1y ago
:hugops: "Things Happen", in production, to the best of us (and FM's pretty damned good at their job). Pretty sure someone's pager duty has been going off like mad. A little over half an hour ago, the mail UI broke for me on Android, and then I panicked and went to desktop web and it broke there too. Also on different networks. As far as I can tell, stuff from their CDN is 404-ing, and a JSON api POST request appears to be going in infinite loop with 200 OKs. The webmail piece seems to be borked... Calendar, Files, Notes etc. are at least rendering.
- adityaathalye 1y ago
  Ok, I just checked and things seem to be rendering again. I just replied to an email via the webmail UI. Back to business as usual.
- SoftTalker 1y ago
  Been working fine for me all morning. I don't use the reader pane UI option though.
pupppet 1y ago
"Breaks UI in production". The most dramatic way to say there was a bug.
brongondwana 1y ago
There are a few different threads on this and now that things are in a stable place I'm going to cross post this to all of them! The larger context is that we're making a major change to how we create IDs for email and mailboxes over the JMAP protocol. The old IDs are a UUID for mailboxId and the first 25 chars of the sha1 of the message for the emailId, prefixed by an 'M'. The new IDs are the createdmodseq for the mailbox prefixed by a 'P' (these are pretty short for most users) and a reverse counter of nanoseconds of the message internaldate (delivery time) for the emailId. This gives good storage density for offline and good data locality in databases for the email listings. You can see all the code for that in a handful of merge requests in the public cyrus-imapd repository on github at https://github.com/cyrusimap/cyrus-imapd/ https://github.com/cyrusimap/cyrus-imapd/ Over the past few weeks, I've been helping out with the last bits of code modification, largely the changes on https://github.com/cyrusimap/cyrus-imapd/pull/5539 https://github.com/cyrusimap/cyrus-imapd/pull/5539 if you're interested. This morning we rolled out a build which we'd tested extensively on our staging and staff servers, but missed that for older v19 mailboxes which hadn't been upgraded to v20, the code to check if messages belonged in a thread incorrectly marked them all as missing. This made MOST emails appear missing for most customers, clearly a very bad situation. We immediately rolled back, but in the hurry missed that an unrelated change to correct subject matching for some languages (Japanese users had reported the issue, but possibly others as well) had changed the thread version, so new threads then had failed reads (making some, though many fewer, messages appear blank in the UI). There were about 50 million attempts to read those values over 15,000 users, because our UI was keeping on retrying thinking it was just a temporary synchronisation issue because the previous request told it there was a Thread to fetch data for. Ouch. https://github.com/cyrusimap/cyrus-imapd/pull/5527 https://github.com/cyrusimap/cyrus-imapd/pull/5527 contains those changes. Anyway, since the only difference between the old and new records was normalisation of subjects, I wrote a tiny patch to let the old code read the newer records and just deployed that, which made all the emails re-appear for everyone again. This is the one bit of code from all this which isn't in a public repo, but it's two lines of: if (version == 2) version = 1; Meanwhile, the real bug is fixed https://github.com/cyrusimap/cyrus-imapd/pull/5553 https://github.com/cyrusimap/cyrus-imapd/pull/5553 And a test has been written to prove it: https://github.com/cyrusimap/cyrus-imapd/pull/5554 https://github.com/cyrusimap/cyrus-imapd/pull/5554 But we'll wait until Monday to upgrade again, when we have fresh eyes available to watch that it's OK. ... P.S. this is almost entirely unrelated to the UI changes. The underlying reason we're doing these changes IS related to UI changes, it's there to make offline mode use storage more efficiently on your device because the IDs are smaller and provide better data locality, but the timing is purely coincidental. The Cyrus changes have been done almost exclusively by the team in the USA and the UI changes by the team in Australia, and our deploy timelines were not synchronised.

34 points by blux 1y ago

Fastmail is great-- solid-- for about 10 years for me. There's a lot that I prefer over Google Mail.

I'm glad I came here to check to see if anyone else had the same issue as I've had! As soon as I realised that both my webmail and my phone app were buggered, it was probably not just a Me Problem.

antongribok 1y ago
Having the same problems. It started out as not being able to search, but the situation is quickly deteriorating and now I'm unable to open pretty much any email message. Some content seems to briefly show up and then it quickly disappears and after that, it's as if cache has been invalidated and you can't get back into it.
- detritus 1y ago
  Luckily for me the Drafts folder's showing, so I was able to send (well, I assume it's sent) the Single Most Important Email I need to send today, which I'd spent about half an hour getting right just as the interface imploded...
  - toomuchtodo 1y ago
    While the web UX is borked, mobile app is not impacted if you can fallback to that until they fix the web.
    - mistertrotsky 1y ago
      This doesn't actually work. The mobile app is just a wrapper around the web interface.
      - aaviator42 1y ago
        I thought apple didn't allow this? Edit: apparently "hybrid apps" using webviews are allowed as long as they're not "thin wrappers" for websites and provide meaningful functionality. See also: the capacitor framework.
      - deleted 1y ago
        [deleted]

antongribok 1y ago

I opened a support case at 9:04 AM EDT. So far, no response yet.

SV_BubbleTime 1y ago
Maybe they don’t realize how important you are. This is a failure in their VIP program, imo they should expedite that over a bug that effects all users.
- ocdtrekkie 1y ago
  For what it's worth the CEO responded to my support ticket once and I felt like a VIP.
  - SV_BubbleTime 1y ago
    Ha, cool. I imagine an an age of AI and LLMs, a lot of people are going to feel special at different points despite the cost of that going to nearly zero.

csomar 1y ago

It didn't break anything for me. However, I am failing to understand the point of this design "refreshment". There is nothing new out there, just minor UI changes with no purpose?

jdbernard 1y ago
Same. I don't hate it, but I do hate that things changed out from underneath me without any notice, opt-in period, or ability to go back.
- RandomBacon 1y ago
  In a few days, when the frenzy dies down, I will make a feature request ticket asking them to do this next time. Please do the same.
garciansmith 1y ago
It makes access to the calendar, contacts, etc., one less click, which I think is nice. Would have preferred all that was set to buttons at the top instead of a thin side bar though.
- csomar 1y ago
  I think that's how it was before? They just added the icons on the left side but you always had the menu on top?

sidcool 1y ago

I don't understand the urge to do public shaming instead of first reaching out to support. And the OP is so rude.

jasonvorhe 1y ago
I don't know. I'm always confused when social media accounts ask to report issues over different channels when it's so much easier to just reply to customer and ask for additional (non PII) information there. In this case it's unlikely that this was the only or even first report about this issue. So why place the initial burden on a paying customer? If your social media is just a marketing channel, at least make that clear so i don't even bother reporting issues that route.
- yesfitz 1y ago
  Companies make it clear how to report issues by having a separate support system. If you clicked a "Support" link and it took you to Twitter, I could better understand your confusion. As for using social media to take issue reports: What will you do when you need PII or have to reassign the issue or reassign part of the issue and those people need to be able to contact the user? "Why place the initial burden on a paying customer?" Because it creates a better service for everyone to have a known way of doing things.
- bjtitus 1y ago
  > If your social media is just a marketing channel, at least make that clear so i don't even bother reporting issues that route Is this snippet from their Twitter bio clear? "If you need assistance, please submit a ticket: http://fastmail.com/support http://fastmail.com/support"
  - jasonvorhe 1y ago
    Actually, yes. This is exactly what I would've asked for. I didn't see that.
koakuma-chan 1y ago

ocdtrekkie 1y ago

The design refresh is actually one of the few I don't hate, I got it yesterday and it is decent. But the blank emails thing started today and this... is a problem.

SomeoneOnTheWeb 1y ago

What the hell is that with OP being so rude? And following FastMail's reply > Hello Andrew! Can you please contact our support team so we can look into this for you? fastmail.com/support They say: > Don't have time. Consider my tweet the bug report. Sorry but this asshole behavior. Bugs happen. No need to do public shaming and being rude to the company for that.

baal80spam 1y ago
Last time I checked fastmail was a paid service. OP has every right to expect it to be working.
- SoftTalker 1y ago
  Sure but Fastmail's failure doesn't give people license to be assholes. There is far too much assholery in the world. It's never OK.
dilap 1y ago
I disgree. Other people were saying it's broken too. Reasonable company behavior here would be 1 tweet "we'll look into it", and then either "we reproduced and are working on a fix" or "looks good to us please contact support so we can investigate your particular issue". But there's no reason to initially make users jump through hoops.
- input_sh 1y ago
  It's in their status page and it's tagged as a partial outage: https://fastmailstatus.com/cme1fq7ej002dh0iu6z8pey4f https://fastmailstatus.com/cme1fq7ej002dh0iu6z8pey4f Personally I don't really use their web interface, but I tried it now and it all works just fine (on both prod and beta).
  - ocdtrekkie 1y ago
    When I experienced this issue it was not on their status page, so I assume it wasn't for the original tooter either.

susanthenerd 1y ago

The bug (at least for me) isn't present on the beta version (beta.fastmail.com)

adityaathalye 1y ago

:hugops: "Things Happen", in production, to the best of us (and FM's pretty damned good at their job). Pretty sure someone's pager duty has been going off like mad. A little over half an hour ago, the mail UI broke for me on Android, and then I panicked and went to desktop web and it broke there too. Also on different networks. As far as I can tell, stuff from their CDN is 404-ing, and a JSON api POST request appears to be going in infinite loop with 200 OKs. The webmail piece seems to be borked... Calendar, Files, Notes etc. are at least rendering.

adityaathalye 1y ago
Ok, I just checked and things seem to be rendering again. I just replied to an email via the webmail UI. Back to business as usual.
SoftTalker 1y ago
Been working fine for me all morning. I don't use the reader pane UI option though.

pupppet 1y ago

"Breaks UI in production". The most dramatic way to say there was a bug.

brongondwana 1y ago

There are a few different threads on this and now that things are in a stable place I'm going to cross post this to all of them! The larger context is that we're making a major change to how we create IDs for email and mailboxes over the JMAP protocol. The old IDs are a UUID for mailboxId and the first 25 chars of the sha1 of the message for the emailId, prefixed by an 'M'. The new IDs are the createdmodseq for the mailbox prefixed by a 'P' (these are pretty short for most users) and a reverse counter of nanoseconds of the message internaldate (delivery time) for the emailId. This gives good storage density for offline and good data locality in databases for the email listings. You can see all the code for that in a handful of merge requests in the public cyrus-imapd repository on github at https://github.com/cyrusimap/cyrus-imapd/ https://github.com/cyrusimap/cyrus-imapd/ Over the past few weeks, I've been helping out with the last bits of code modification, largely the changes on https://github.com/cyrusimap/cyrus-imapd/pull/5539 https://github.com/cyrusimap/cyrus-imapd/pull/5539 if you're interested. This morning we rolled out a build which we'd tested extensively on our staging and staff servers, but missed that for older v19 mailboxes which hadn't been upgraded to v20, the code to check if messages belonged in a thread incorrectly marked them all as missing. This made MOST emails appear missing for most customers, clearly a very bad situation. We immediately rolled back, but in the hurry missed that an unrelated change to correct subject matching for some languages (Japanese users had reported the issue, but possibly others as well) had changed the thread version, so new threads then had failed reads (making some, though many fewer, messages appear blank in the UI). There were about 50 million attempts to read those values over 15,000 users, because our UI was keeping on retrying thinking it was just a temporary synchronisation issue because the previous request told it there was a Thread to fetch data for. Ouch. https://github.com/cyrusimap/cyrus-imapd/pull/5527 https://github.com/cyrusimap/cyrus-imapd/pull/5527 contains those changes. Anyway, since the only difference between the old and new records was normalisation of subjects, I wrote a tiny patch to let the old code read the newer records and just deployed that, which made all the emails re-appear for everyone again. This is the one bit of code from all this which isn't in a public repo, but it's two lines of: if (version == 2) version = 1; Meanwhile, the real bug is fixed https://github.com/cyrusimap/cyrus-imapd/pull/5553 https://github.com/cyrusimap/cyrus-imapd/pull/5553 And a test has been written to prove it: https://github.com/cyrusimap/cyrus-imapd/pull/5554 https://github.com/cyrusimap/cyrus-imapd/pull/5554 But we'll wait until Monday to upgrade again, when we have fresh eyes available to watch that it's OK. ... P.S. this is almost entirely unrelated to the UI changes. The underlying reason we're doing these changes IS related to UI changes, it's there to make offline mode use storage more efficiently on your device because the IDs are smaller and provide better data locality, but the timing is purely coincidental. The Cyrus changes have been done almost exclusively by the team in the USA and the UI changes by the team in Australia, and our deploy timelines were not synchronised.