7 ms·

Root cause analysis: significantly elevated error rates on 2019‑07‑10

203 points by gr2020 7y ago

chance_state 7y ago
This reads like the marketing/PR teams wrote much of it. Compare to the Cloudflare post-mortem from today: https://blog.cloudflare.com/details-of-the-cloudflare-outage-on-july-2-2019/ https://blog.cloudflare.com/details-of-the-cloudflare-outage...
- draw_down 7y ago
  Nobody from marketing writes this stuff, come on
- buildawesome 7y ago
  Out of curiosity, how would you have preferred to see a shard unable to accept writes? I think in both post-mortems, you would see comparable graphs - usage and then a drop in usage. I think it's easier to document a failed regex versus "here's our cluster architecture that we've been using for 3 months". Also, does your company's engineering decisions change based on other companies' post-mortems?
- dps 7y ago
  I'm Stripe's CTO and wrote a good deal of the RCA (with the help of others, including a lot of the engineers who responded to the incident). If you've any specific feedback on how to make this more useful, I'd love to hear it.
  - patio11 7y ago
    I work at Stripe, on the marketing team, and assisted a bit here. My last major engineering work was writing the backend to a stock exchange. If anyone on HN knows anyone who has the sort of interesting life story where they both know what can cause a cluster election to fail and like writing about that sort of thing, we would eagerly like to make their acquaintance.
    - luizfelberti 7y ago
      Maybe Kyle Kingsbury (aka @aphyr) is the person you are looking for? https://jepsen.io/services#consulting https://jepsen.io/services#consulting
      - wbronitsky 7y ago
        Kyle used to work at Stripe and left. I don’t think he would come back unfortunately. That guy is absolutely amazing, especially with regards to distributes DBs and writing about them
  - Ocha 7y ago
    For starters maybe provide more details beside the vague information of some feature of some database didnt work as expected. Imagine you are giving this to your employees (especially new ones) to learn something. How much actual useful knowledge is being shared here to learn from?
  - chacham15 7y ago
    Unexpected things are bound to happen. But, one thing that stuck out to me is that you dont seem to have a safe way to test changes (which would have prevented the second failure). Are there no other environments to test changes on? Is there no way to incrementally roll-out? Is there not another environment which can step in in place of a failing one while you investigate? These seem like fairly common industry practices which help you deal with unexpected failures, but I dont see a mention of if/why these practices failed and if/how that is being remediated.
  - davidw 7y ago
    I don't think either one is particularly "useful" to me as a consumer of the business, other than knowing that "we have top people working on it right now" and there's a plan in place to try and avoid future problems. What's fun for a software person is that there's a lot of interesting digressions and stuff to learn in the Cloudflare one. The whole explanation of the regexp at the end is something that no one cares about from the business side, but is an interesting read in and of itself. It's worth noting that yours came out a bit more than a week faster than theirs, which jgrahamc clearly spent a lot of time writing. No idea if anyone cares about the speed with which these things are released...
  - jabart 7y ago
    It would be great that in these types of situations if the CC Tokens validity period is extended, or at least known as the documentation states it is short. For our app if the tokens were valid longer, we could write this up as a non-event and retry when things were better.
  - hibikir 7y ago
    Hi Dave, you probably won't remember me (we only spent about 2 months together in Stripe), but I bet Mr Larson remembers. The first question is who is this written for: It lacks the detail I would write for the incident review meeting audience, while lacking a simpler story for the non technical. As it is at the time I read it, I don't think it aims any audience very well. I understand that the level of detail of the internal report might be excessive for the internal report, but if technical readers are the target, some more details would have helped. For example the monitoring details that Will described in another thread are a key missing detail that, if anything, would make Stripe look better, as problems like that happen all the time. I bet there are more details that are equally useful that would be in an internal report that would not reveal delicate information. In general, the only reason I could follow the document well is that I remember how the Stripe storage system worked last year, and I could handwave a year worth of changes. Since this part of the Stripe infrastructure is relatively unique, it's difficult to understand from the outside, and looks as if it doesn't have enough information. In particular, the remediations say very little that is understandable from the outside: Most of the text could apply to pretty much any incident on a storage of queuing subsystem I was ever a part of: More alerts, an extra chart in an ever growing dashboard, some circuit breakers to deal with the specific failure shape... It's all real, but without details, it says very little. I understand why you might not want to divulge that level of detail though. If we want fewer details, then the article could cut all kinds of low-information sections, and instead focus more on the response, and the things that will be changed in the future. The most interesting bit about this is the quick version rollback, which, in retrospect, might not have been the right call. A more detailed view of the alternatives, And why the actions that ultimately led to the second incident were made would be enlightening, and would humanize the piece. Thank you for not just providing a public root cause analysis, but coming here to discuss it in HN.
- Havoc 7y ago
  >This reads like the marketing/PR teams wrote much of it. The remediation part is quite cautious/generic but overall it seems like a good faith effort by someone constrained by corporate rules.
gr2020 7y ago
Anybody know what database they’re using?
- conroy 7y ago
  MongoDB is the primary data store used at Stripe.
  - a13n 7y ago
    Really speaks volumes about how mature MongoDB has become considering how solid Stripe's reliability is.
    - ahuang 7y ago
      Not always ;). As someone with experience managing mongo at scale, this really speaks volumes to the amount of effort needed to make it not do the wrong thing. And even then, there are unknown unknowns like this that can pop up at any time.
    - londons_explore 7y ago
      MongoDB is a really scary database to use at scale. It doesn't shard nicely. Failovers have rather nasty semantics that can cause nasty bugs in client side code. Performance cliffs abound. If your datastore is anything over 1TB, I'd be using postgres, or if you can manage it something bigtable-like.
deleted 7y ago
[deleted]
debt 7y ago
"We identified that our rolled-back election protocol interacted poorly with a recently-introduced configuration setting to trigger the second period of degradation." Damn what a mess. Sounds like y'all are rolling out way to many changes too quickly with little to no time for integration testing. It's a somewhat amateur move to assume you can just arbitrarily rollback without consequence, without testing etc. One solution I don't see mentioned, don't upgrade to minor versions ever. And create a dependency matrix so if you do rollback, you rollback all the other things that depend on the thing you're rolling back as well.
- cetico 7y ago
  Yes this was very surprising. The system was working fine after the cluster restart. There was no need for an emergency rollback. Doing a large rollback based on a hunch seems like an overreaction. It's totally normal for engineers to commit these errors. That's fine. The detail that's missing in this PM is what kind of operational culture, procedures and automation is in place to reduce operator errors. Did the engineer making this decision have access to other team members to review their plan of action? I believe that a group (2-3) of experienced engineers sharing information in real-time and coordinating the response could have reacted better. Of course, I wasn't there so I could be completely off.
  - debt 7y ago
    "That's fine." idk the suits have a very different viewpoint; 30 minutes of downtime for a large financial system isn't fine. it can be very costly.
    - uponcoffee 7y ago
      I think the GP means that as far as incidents occurring, so far as care is (or was) taken to prevent them and learn from them, then that's all one can really reasonably ask for. The first incident falls under that heading and 'is fine' in a 'life happens' sense. The following incident comes across as reckless and avoidable as there should have been procedures to safely test the rollback (and perhaps there were, but a perfect storm allowed it fail in prod). Lacking details about how the second incident came to be or how they will be prevented going forward places the second incident as 'not fine'. This information is what the GP comment is asking for. Compare this PM with Cloudflare's PM, where they detail how they tested rules, what safeguards were in place, how the incident came to be, and how they intend to prevent similar incidents; the impression given here is that they will put up more fire alarms and fire extinguishers but do little fire prevention.
laCour 7y ago
"[Four days prior to the incident] Two nodes became stalled for yet-to-be-determined reasons." How did they not catch this? It's super surprising to me that they wouldn't have monitors for this.
- ashelmire 7y ago
  If you can think of every possible failure and create monitoring and reporting for it before it happens, then you're the best dev on the planet.
  - sithlord 7y ago
    And also have the greatest bosses on the history of earth giving you unlimited time to do this.
  - raverbashing 7y ago
    And then filtering for a lot of crap and false alarms the tools and supporting infrastructure throws I kinda lost count of how many times Nagios barfed itself and reported an error while the application was fine
- gtirloni 7y ago
  In this environment: Stripe splits data by kind into different database clusters and by quantity into different shards. Each cluster has many shards, and each shard has multiple redundant nodes. having a few nodes down is perfectly acceptable. I guess they would have had an alert if the number of down nodes exceeded some threshold.
  - runevault 7y ago
    this case that doesn't sound like it was the issue, it was the lack of promotion of new master due to the bug in the system in terms of shard promotion.
- deleted 7y ago
  [deleted]
jacquesm 7y ago
Why don't they call 'significantly elevated error rates' an 'outage' instead?
- dmlittle 7y ago
  My guess is that it's because not everything was down so it wasn't a total outage. From the post mortem: > Stripe splits data by kind into different database clusters and by quantity into different shards. So in theory any request that didn't interact with the problematic database should have been OK (I don't know if the offending DB was in the critical path of _every_ request).
- NikolaeVarius 7y ago
  Because "A substantial majority of API requests during these windows failed. " implying that there was not a complete outage. I don't understand why people demand the usage of incorrect language.
  - teraflop 7y ago
    In my mind, a "degradation" would be if some fraction of requests were randomly failing, but they would be likely to eventually succeed if retried. Or if the service itself was essentially accessible, but some non-essential functionality was not working correctly. On the other hand, if for a significant number of users the site was completely unusable for some period of time, then I think it's fair to use the word "outage". (Even if it's not a complete outage affecting all users.) I don't know whether other people would interpret these terms the same way I do, nor do I think there's enough information in this blog post to determine for sure which label is more accurate for this particular incident. So personally, I'm not going to be too picky about the wording.
  - ComputerGuru 7y ago
    > Because "A substantial majority of API requests during these windows failed. " implying that there was not a complete outage. The fact that you needed to qualify “outage” with “complete” clearly means the word on its own is not incorrect for cases where a system was “only” mostly unavailable rather than completely so. > I don't understand why people demand the usage of incorrect language. The irony.
ssalazars 7y ago
[2019-07-10 20:13 UTC] During our investigation into the root cause of the first event, we identified a code path likely causing the bug in a new minor version of the database’s election protocol. [2019-07-10 20:42 UTC] We rolled back to a previous minor version of the election protocol and monitored the rollout. There's a 20 minute gap between investigation and "rollback". Why did they rollback if the service was back to normal? How can they decide, and document the change within 20 minutes? Are they using CMs to document changes in production? Were there enough engineers involved in the decision? Clearly all variables were not considered. To me, this demonstrates poor Operational Excellence values. Your first goal is to mitigate the problem. Then, you need to analyze, understand, and document the root cause. Rolling-back was a poor decision, imo.
- sb8244 7y ago
  I think this is a good point. Don't rollback if you don't know why your new code is giving you problems. You may fix things with the rollback, or you may put yourself in a worse situation where the forward/backwards compatibility has a bug in it. The issue may even be coincidental to the new code. However, it's hard to say whether this is a poor decision unless we know that they didn't analyze the path and determine that it would most likely be fine. If they did do that, then it's just a mistake and those happen. 20 minutes is enough time to make that call for the team that built it.
  - greenleafjacob 7y ago
    If rollbacks are not safe then you have a change management problem. If you have a good CM system, you should have a timeline of changes that you can correlate against incidents. Most incidents are caused by changes, so you can narrow down most incidents to a handful of changes. Then the question is, if you have a handful of changes that you could roll back, and rollbacks are risk free, then does it make sense to delay rolling back any particular change until the root cause is understood?
    - Silhouette 7y ago
      It's not always as simple as that. What if the problem was that something in a change didn't behave as specified and wound up writing important data in an incorrect but retrievable format? Rolling back might not recognise that data properly and could end up either modifying it further so the true data could no longer be retrieved or causing data loss elsewhere as a consequence.
vjagrawal1984 7y ago
In the face of so many outages from big companies, I wonder how Visa/MasterCard is so resilient. Is it because they are over the curve and don't make "any" changes to their system. As opposed to other companies, we are still maturing?
- Thaxll 7y ago
  They're also much simpler and the system behind payment solution didn't changed that much in the last 10 years.
  - londons_explore 7y ago
    They are also miles behind on features customers want... For example: * My credit card statement should have links to the merchant, the address, a list of the things I bought, a link to the returns process, etc. * Why can't my statement also have the total number of calories I've purchased in the last month, or grams of carbon in fuel I've put in the truck? * Why can't I use my mastercard to pay another mastercard user directly? * Why hasn't mastercard produced a '2 factor' for card payments rather than forcing every bank to implement their own? * Why can't I buy a dual Mastercard/Visa/Other card, which works with merchants who are picky and will only accept one or the other? * Why are we still issuing bits of plastic in the digital age anyway? * Why don't the cards have a microusb plug on one edge, or NFC to plug into a phone or computer to log in, to act as an identity card, to authenticate or make payments, or anything else other companies issue smartcards for? * Why don't mastercard work with mobile providers to issue cards that you can spend your pay-as-you-go balance with, turning a mobile provider into a bank. It seems mastercards business is 'stuck', and there are opportunities to innovate all around them, but they won't.
    - kortilla 7y ago
      >Why are we still issuing bits of plastic in the digital age anyway? Phones die. If you don’t care, I suggest you look into Apple Pay or something similar. You’ll find many merchants that you won’t be able to pay.
luminati 7y ago
Since both companies' root cause analysis are currently trending on HN, it's pretty apparent that Stripe's engineering culture has a long ways to go catch up with Cloudflare's.
segmondy 7y ago
As I mentioned early, " human error often, configuration changes often, new changes often. " https://news.ycombinator.com/item?id=20406116 https://news.ycombinator.com/item?id=20406116
zby 7y ago
So the article identifies a software bug and a software/config bug as the root cause. That sounds a bit shallow for such a high visibility case - I was expecting something like the https://en.wikipedia.org/wiki/5_Whys https://en.wikipedia.org/wiki/5_Whys method with subplots on why the bugs where not caught in testing. By the way I only clicked on it because I was hoping it would be an occasion to use the methods from http://bayes.cs.ucla.edu/WHY/ http://bayes.cs.ucla.edu/WHY/ - alas no - it was too shallow for that.
- zbentley 7y ago
  It is likely that this RCA was shallow because it was intended for everyone--including non-technical users, who (at least in my experience) tend to misinterpret or get confused by deep technical or systemic failure analysis. It would be excellent if Stripe published a truly technical RCA, perhaps for distribution via their tech blog, so that folks like us could get a more complete understanding and what-not-to-do lesson (if the failing systems were based on non-proprietary technologies, that is).
  - throwawaydba 7y ago
    From reading the RCA, this should be the trinity of mysql + orchestrator + vitess. If stripe can't get it right, there is no chance for the others.
mual 7y ago
Is this Stripe's first public RCA? Looking through their tweets, there do not appear to be other RCAs for the same "elevated error rates". It seems hard to conclude much from one RCA.

203 points by gr2020 7y ago

This reads like the marketing/PR teams wrote much of it. Compare to the Cloudflare post-mortem from today: https://blog.cloudflare.com/details-of-the-cloudflare-outage-on-july-2-2019/ https://blog.cloudflare.com/details-of-the-cloudflare-outage...

draw_down 7y ago
Nobody from marketing writes this stuff, come on
buildawesome 7y ago
Out of curiosity, how would you have preferred to see a shard unable to accept writes? I think in both post-mortems, you would see comparable graphs - usage and then a drop in usage. I think it's easier to document a failed regex versus "here's our cluster architecture that we've been using for 3 months". Also, does your company's engineering decisions change based on other companies' post-mortems?
dps 7y ago
I'm Stripe's CTO and wrote a good deal of the RCA (with the help of others, including a lot of the engineers who responded to the incident). If you've any specific feedback on how to make this more useful, I'd love to hear it.
- patio11 7y ago
  I work at Stripe, on the marketing team, and assisted a bit here. My last major engineering work was writing the backend to a stock exchange. If anyone on HN knows anyone who has the sort of interesting life story where they both know what can cause a cluster election to fail and like writing about that sort of thing, we would eagerly like to make their acquaintance.
  - luizfelberti 7y ago
    Maybe Kyle Kingsbury (aka @aphyr) is the person you are looking for? https://jepsen.io/services#consulting https://jepsen.io/services#consulting
    - wbronitsky 7y ago
      Kyle used to work at Stripe and left. I don’t think he would come back unfortunately. That guy is absolutely amazing, especially with regards to distributes DBs and writing about them
- Ocha 7y ago
  For starters maybe provide more details beside the vague information of some feature of some database didnt work as expected. Imagine you are giving this to your employees (especially new ones) to learn something. How much actual useful knowledge is being shared here to learn from?
- chacham15 7y ago
  Unexpected things are bound to happen. But, one thing that stuck out to me is that you dont seem to have a safe way to test changes (which would have prevented the second failure). Are there no other environments to test changes on? Is there no way to incrementally roll-out? Is there not another environment which can step in in place of a failing one while you investigate? These seem like fairly common industry practices which help you deal with unexpected failures, but I dont see a mention of if/why these practices failed and if/how that is being remediated.
- davidw 7y ago
  I don't think either one is particularly "useful" to me as a consumer of the business, other than knowing that "we have top people working on it right now" and there's a plan in place to try and avoid future problems. What's fun for a software person is that there's a lot of interesting digressions and stuff to learn in the Cloudflare one. The whole explanation of the regexp at the end is something that no one cares about from the business side, but is an interesting read in and of itself. It's worth noting that yours came out a bit more than a week faster than theirs, which jgrahamc clearly spent a lot of time writing. No idea if anyone cares about the speed with which these things are released...
- jabart 7y ago
  It would be great that in these types of situations if the CC Tokens validity period is extended, or at least known as the documentation states it is short. For our app if the tokens were valid longer, we could write this up as a non-event and retry when things were better.
- hibikir 7y ago
  Hi Dave, you probably won't remember me (we only spent about 2 months together in Stripe), but I bet Mr Larson remembers. The first question is who is this written for: It lacks the detail I would write for the incident review meeting audience, while lacking a simpler story for the non technical. As it is at the time I read it, I don't think it aims any audience very well. I understand that the level of detail of the internal report might be excessive for the internal report, but if technical readers are the target, some more details would have helped. For example the monitoring details that Will described in another thread are a key missing detail that, if anything, would make Stripe look better, as problems like that happen all the time. I bet there are more details that are equally useful that would be in an internal report that would not reveal delicate information. In general, the only reason I could follow the document well is that I remember how the Stripe storage system worked last year, and I could handwave a year worth of changes. Since this part of the Stripe infrastructure is relatively unique, it's difficult to understand from the outside, and looks as if it doesn't have enough information. In particular, the remediations say very little that is understandable from the outside: Most of the text could apply to pretty much any incident on a storage of queuing subsystem I was ever a part of: More alerts, an extra chart in an ever growing dashboard, some circuit breakers to deal with the specific failure shape... It's all real, but without details, it says very little. I understand why you might not want to divulge that level of detail though. If we want fewer details, then the article could cut all kinds of low-information sections, and instead focus more on the response, and the things that will be changed in the future. The most interesting bit about this is the quick version rollback, which, in retrospect, might not have been the right call. A more detailed view of the alternatives, And why the actions that ultimately led to the second incident were made would be enlightening, and would humanize the piece. Thank you for not just providing a public root cause analysis, but coming here to discuss it in HN.
Havoc 7y ago
>This reads like the marketing/PR teams wrote much of it. The remediation part is quite cautious/generic but overall it seems like a good faith effort by someone constrained by corporate rules.

gr2020 7y ago

Anybody know what database they’re using?

conroy 7y ago
MongoDB is the primary data store used at Stripe.
- a13n 7y ago
  Really speaks volumes about how mature MongoDB has become considering how solid Stripe's reliability is.
  - ahuang 7y ago
    Not always ;). As someone with experience managing mongo at scale, this really speaks volumes to the amount of effort needed to make it not do the wrong thing. And even then, there are unknown unknowns like this that can pop up at any time.
  - londons_explore 7y ago
    MongoDB is a really scary database to use at scale. It doesn't shard nicely. Failovers have rather nasty semantics that can cause nasty bugs in client side code. Performance cliffs abound. If your datastore is anything over 1TB, I'd be using postgres, or if you can manage it something bigtable-like.

deleted 7y ago

[deleted]

debt 7y ago

"We identified that our rolled-back election protocol interacted poorly with a recently-introduced configuration setting to trigger the second period of degradation." Damn what a mess. Sounds like y'all are rolling out way to many changes too quickly with little to no time for integration testing. It's a somewhat amateur move to assume you can just arbitrarily rollback without consequence, without testing etc. One solution I don't see mentioned, don't upgrade to minor versions ever. And create a dependency matrix so if you do rollback, you rollback all the other things that depend on the thing you're rolling back as well.

cetico 7y ago
Yes this was very surprising. The system was working fine after the cluster restart. There was no need for an emergency rollback. Doing a large rollback based on a hunch seems like an overreaction. It's totally normal for engineers to commit these errors. That's fine. The detail that's missing in this PM is what kind of operational culture, procedures and automation is in place to reduce operator errors. Did the engineer making this decision have access to other team members to review their plan of action? I believe that a group (2-3) of experienced engineers sharing information in real-time and coordinating the response could have reacted better. Of course, I wasn't there so I could be completely off.
- debt 7y ago
  "That's fine." idk the suits have a very different viewpoint; 30 minutes of downtime for a large financial system isn't fine. it can be very costly.
  - uponcoffee 7y ago
    I think the GP means that as far as incidents occurring, so far as care is (or was) taken to prevent them and learn from them, then that's all one can really reasonably ask for. The first incident falls under that heading and 'is fine' in a 'life happens' sense. The following incident comes across as reckless and avoidable as there should have been procedures to safely test the rollback (and perhaps there were, but a perfect storm allowed it fail in prod). Lacking details about how the second incident came to be or how they will be prevented going forward places the second incident as 'not fine'. This information is what the GP comment is asking for. Compare this PM with Cloudflare's PM, where they detail how they tested rules, what safeguards were in place, how the incident came to be, and how they intend to prevent similar incidents; the impression given here is that they will put up more fire alarms and fire extinguishers but do little fire prevention.

laCour 7y ago

"[Four days prior to the incident] Two nodes became stalled for yet-to-be-determined reasons." How did they not catch this? It's super surprising to me that they wouldn't have monitors for this.

ashelmire 7y ago
If you can think of every possible failure and create monitoring and reporting for it before it happens, then you're the best dev on the planet.
- sithlord 7y ago
  And also have the greatest bosses on the history of earth giving you unlimited time to do this.
- raverbashing 7y ago
  And then filtering for a lot of crap and false alarms the tools and supporting infrastructure throws I kinda lost count of how many times Nagios barfed itself and reported an error while the application was fine
gtirloni 7y ago
In this environment: Stripe splits data by kind into different database clusters and by quantity into different shards. Each cluster has many shards, and each shard has multiple redundant nodes. having a few nodes down is perfectly acceptable. I guess they would have had an alert if the number of down nodes exceeded some threshold.
- runevault 7y ago
  this case that doesn't sound like it was the issue, it was the lack of promotion of new master due to the bug in the system in terms of shard promotion.
deleted 7y ago
[deleted]

jacquesm 7y ago

Why don't they call 'significantly elevated error rates' an 'outage' instead?

dmlittle 7y ago
My guess is that it's because not everything was down so it wasn't a total outage. From the post mortem: > Stripe splits data by kind into different database clusters and by quantity into different shards. So in theory any request that didn't interact with the problematic database should have been OK (I don't know if the offending DB was in the critical path of _every_ request).
NikolaeVarius 7y ago
Because "A substantial majority of API requests during these windows failed. " implying that there was not a complete outage. I don't understand why people demand the usage of incorrect language.
- teraflop 7y ago
  In my mind, a "degradation" would be if some fraction of requests were randomly failing, but they would be likely to eventually succeed if retried. Or if the service itself was essentially accessible, but some non-essential functionality was not working correctly. On the other hand, if for a significant number of users the site was completely unusable for some period of time, then I think it's fair to use the word "outage". (Even if it's not a complete outage affecting all users.) I don't know whether other people would interpret these terms the same way I do, nor do I think there's enough information in this blog post to determine for sure which label is more accurate for this particular incident. So personally, I'm not going to be too picky about the wording.
- ComputerGuru 7y ago
  > Because "A substantial majority of API requests during these windows failed. " implying that there was not a complete outage. The fact that you needed to qualify “outage” with “complete” clearly means the word on its own is not incorrect for cases where a system was “only” mostly unavailable rather than completely so. > I don't understand why people demand the usage of incorrect language. The irony.

ssalazars 7y ago

[2019-07-10 20:13 UTC] During our investigation into the root cause of the first event, we identified a code path likely causing the bug in a new minor version of the database’s election protocol. [2019-07-10 20:42 UTC] We rolled back to a previous minor version of the election protocol and monitored the rollout. There's a 20 minute gap between investigation and "rollback". Why did they rollback if the service was back to normal? How can they decide, and document the change within 20 minutes? Are they using CMs to document changes in production? Were there enough engineers involved in the decision? Clearly all variables were not considered. To me, this demonstrates poor Operational Excellence values. Your first goal is to mitigate the problem. Then, you need to analyze, understand, and document the root cause. Rolling-back was a poor decision, imo.

sb8244 7y ago
I think this is a good point. Don't rollback if you don't know why your new code is giving you problems. You may fix things with the rollback, or you may put yourself in a worse situation where the forward/backwards compatibility has a bug in it. The issue may even be coincidental to the new code. However, it's hard to say whether this is a poor decision unless we know that they didn't analyze the path and determine that it would most likely be fine. If they did do that, then it's just a mistake and those happen. 20 minutes is enough time to make that call for the team that built it.
- greenleafjacob 7y ago
  If rollbacks are not safe then you have a change management problem. If you have a good CM system, you should have a timeline of changes that you can correlate against incidents. Most incidents are caused by changes, so you can narrow down most incidents to a handful of changes. Then the question is, if you have a handful of changes that you could roll back, and rollbacks are risk free, then does it make sense to delay rolling back any particular change until the root cause is understood?
  - Silhouette 7y ago
    It's not always as simple as that. What if the problem was that something in a change didn't behave as specified and wound up writing important data in an incorrect but retrievable format? Rolling back might not recognise that data properly and could end up either modifying it further so the true data could no longer be retrieved or causing data loss elsewhere as a consequence.

vjagrawal1984 7y ago

In the face of so many outages from big companies, I wonder how Visa/MasterCard is so resilient. Is it because they are over the curve and don't make "any" changes to their system. As opposed to other companies, we are still maturing?

Thaxll 7y ago
They're also much simpler and the system behind payment solution didn't changed that much in the last 10 years.
- londons_explore 7y ago
  They are also miles behind on features customers want... For example: * My credit card statement should have links to the merchant, the address, a list of the things I bought, a link to the returns process, etc. * Why can't my statement also have the total number of calories I've purchased in the last month, or grams of carbon in fuel I've put in the truck? * Why can't I use my mastercard to pay another mastercard user directly? * Why hasn't mastercard produced a '2 factor' for card payments rather than forcing every bank to implement their own? * Why can't I buy a dual Mastercard/Visa/Other card, which works with merchants who are picky and will only accept one or the other? * Why are we still issuing bits of plastic in the digital age anyway? * Why don't the cards have a microusb plug on one edge, or NFC to plug into a phone or computer to log in, to act as an identity card, to authenticate or make payments, or anything else other companies issue smartcards for? * Why don't mastercard work with mobile providers to issue cards that you can spend your pay-as-you-go balance with, turning a mobile provider into a bank. It seems mastercards business is 'stuck', and there are opportunities to innovate all around them, but they won't.
  - kortilla 7y ago
    >Why are we still issuing bits of plastic in the digital age anyway? Phones die. If you don’t care, I suggest you look into Apple Pay or something similar. You’ll find many merchants that you won’t be able to pay.

luminati 7y ago

Since both companies' root cause analysis are currently trending on HN, it's pretty apparent that Stripe's engineering culture has a long ways to go catch up with Cloudflare's.

segmondy 7y ago

As I mentioned early, " human error often, configuration changes often, new changes often. " https://news.ycombinator.com/item?id=20406116 https://news.ycombinator.com/item?id=20406116

zby 7y ago

So the article identifies a software bug and a software/config bug as the root cause. That sounds a bit shallow for such a high visibility case - I was expecting something like the https://en.wikipedia.org/wiki/5_Whys https://en.wikipedia.org/wiki/5_Whys method with subplots on why the bugs where not caught in testing. By the way I only clicked on it because I was hoping it would be an occasion to use the methods from http://bayes.cs.ucla.edu/WHY/ http://bayes.cs.ucla.edu/WHY/ - alas no - it was too shallow for that.

zbentley 7y ago
It is likely that this RCA was shallow because it was intended for everyone--including non-technical users, who (at least in my experience) tend to misinterpret or get confused by deep technical or systemic failure analysis. It would be excellent if Stripe published a truly technical RCA, perhaps for distribution via their tech blog, so that folks like us could get a more complete understanding and what-not-to-do lesson (if the failing systems were based on non-proprietary technologies, that is).
- throwawaydba 7y ago
  From reading the RCA, this should be the trinity of mysql + orchestrator + vitess. If stripe can't get it right, there is no chance for the others.

mual 7y ago

Is this Stripe's first public RCA? Looking through their tweets, there do not appear to be other RCAs for the same "elevated error rates". It seems hard to conclude much from one RCA.