8 ms·

UK bank fined £49M over IT system meltdown

139 points by shaman1 4y ago

heurisko 4y ago
2008: "The UK-based IT department of the fifth largest bank continues to dwindle as more jobs go overseas... This round of cuts, starting in June and lasting 12 months, involves up to 250 permanent IT roles and 200 contractors from the bank's technical delivery division, responsible for software development and design." [1] 2018: "Timeline of trouble: how the TSB IT meltdown unfolded". [2] It's probably more complicated than that, but perhaps not much more complicated. [1] https://www.itpro.co.uk/197982/lloyds-tsb-cuts-more-uk-it-jobs https://www.itpro.co.uk/197982/lloyds-tsb-cuts-more-uk-it-jo... [2] https://www.theguardian.com/business/2018/jun/06/timeline-of-trouble-how-the-tsb-it-meltdown-unfolded https://www.theguardian.com/business/2018/jun/06/timeline-of...
- makomk 4y ago
  That probably has very little to do with it. The immediate cause of all the problems was that Lloyds TSB was forcibly split up in order to try and increase competition and the Lloyds half kept the IT department, and when the TSB half tried to move over to the existing IT platform of their new parent company everything broke.
  - lmm 4y ago
    True as far as it goes, but "everything broke" as a predictable result of poor decisions they made, such as moving everyone over in one go.
    - sofixa 4y ago
      > such as moving everyone over in one go. It's the core banking system of a big bank. Handling consistent state between the old and new systems while progressively migrating customers would probably have been extremely complex. They also get to have a maintenance window (few people will complain if they get warned their bank and all transactions won't work for 2 hours in the middle of the night on Monday). A "big bang" migration makes more sense, if everything is properly prepared and tested, which it wasn't.
      - JackFr 4y ago
        > A "big bang" migration makes more sense, if everything is properly prepared and tested, which it wasn't. That sounds like ‘A “big bang” migration makes more sense, if it works.’ Your rollback plan should never be an afterthought. Your rollback plan should be designed like you expect to use it. If your rollback plan is “Burn the ships” and fix-forward, you shouldn’t be working at a bank.
        flerchin 4y ago
        Definitely been the plan at a variety of places I've worked. Including a bank.
      - PeterisP 4y ago
        I've been a part of multiple "big bang" migrations in banking (generally scheduled to the coincidences of local banking holidays next to a weekend, so you can afford multiple days of semi-downtime) and all of them had explicit multiple testing gates for potential rollback during the migration, where after pretty much all the stuff is on the new system, the board convenes and after looking at the difficulties (there inevitably are some unexpected difficulties) make the decision whether they "accept" the switch to the new system or postpone the switch. Part of your preparation and testing is the rollback of a partial migration - if you're irreversibly committed to the "big bang" before you know its outcome, then your preparation and testing has failed.
    - kitd 4y ago
      Note that "they" in this case is the new parent company. IIRC, they were a fairly new bank, heavily reliant on technology. They had the tech but not the customers. TSB were the opposite. The parent thought their tech, which up to that point was only dealing with 100,000s of records, could deal with billions with little change. They were spectacularly wrong and it showed in all testing. But their management pushed ahead to go live anyway. Their actions are not far from criminal negligence IMHO.
      - deleted 4y ago
        [deleted]
      - nerdawson 4y ago
        TSB’s parent company is Sabadell which is a massive Spanish corporation. I was a Solbank (one of their brands) customer and can safely say tech isn’t their strong point. Awful UX and a pain to deal with.
  - 908B64B197 4y ago
    It's funny how there's always a cheap offshored bodyshop involved in these stories yet it's never their fault. Cue in the 9$/hour indian coders working for Boeing. Here's the thing I've learned over the years: Never touch offshored code. Always go for a complete re-write. Don't add features to it, don't refactor it, don't extend it. Just re-write. In my experience it's the best approach. I know guys who made it their whole business to go and completely re-write projects from scratch after offshoring efforts failed.
- varispeed 4y ago
  There was also a heavy enforcement of IR35 in the banking sector, so that substantially reduced the access to talent pool. There was also a tightening of posted worker regulations, so that banks couldn't ship workers from overseas as a source of cheap talented workforce.
kmlx 4y ago
i remember this. for at least a week people couldn't access their money. it was chaos. the bank lost lots of money and customers due to this botched transfer.
- rwmj 4y ago
  People were seeing random balances in their accounts, or seeing other people's accounts, according to some news reports at the time: https://www.bbc.co.uk/news/uk-43860449 https://www.bbc.co.uk/news/uk-43860449
- jesusthatsgreat 4y ago
  Not your keys, not your coins.
  - maxehmookau 4y ago
    Nobody actually lost any money. The "coins" were still mine (I was affected) and heavy banking regulation in the UK probably reduced the harm to effectively zero for the vast majority of retail customers. I received compensation for the harm caused and got all of my money back.
orf 4y ago
The actual report by the FCA: https://www.fca.org.uk/publication/final-notices/tsb-bank-plc-2022.pdf https://www.fca.org.uk/publication/final-notices/tsb-bank-pl...
- shaman1 4y ago
  Quite thorough report, some points that stand out from the summary: >SABIS was TSB’s principal outsourced provider >SABIS relied extensively on 85 third parties (TSB’s fourth parties) to deliver the systems required for the migration and the operation of the platform, which required it to act as a service aggregator. It amazes me the sheer complexity of a retail bank software system and I suspect most of it is due to legacy systems, legal requirements and lack of regular spring cleaning.
sherr 4y ago
Not my main bank so this did not affect me badly but their online (web based) banking portal is still glitchy and not very good.
- MaxBarraclough 4y ago
  Even if it's not your main account, it could be bad if your account details were leaked.
petepete 4y ago
Hopefully Virgin Money will get one too. They broke their Android app earlier this year and since they make you verify web logins using the app I was unable to access any of my business accounts for ~3 weeks. If something really urgent had come up I could have done what I needed via telephone banking or in a branch, but it was a huge pain in the arse because of a single point of failure. Just let me use a Yubikey as my second factor damnit.
- insomniacity 4y ago
  +1 on the Yubikey. I'm pretty good at moving my savings around and getting the best interest rate possible - the side effect is a ton of accounts, which means I'm drowning in 'secure memorable passcode key PINs' and my SMS inbox is full of SMS 2FA codes, and I'm wondering what it would take to get a bank to offer Webauthn/FIDO. How about a website where we pledged to open an account and deposit £X into savings, or switch current account, if they offered Webauthn/FIDO?
  - rlpb 4y ago
    I'd love FIDO for online banking auth. But AIUI, there's some EU regulation that requires 2FA, but that 2FA must also verify some other data (like the recipient of a transfer, amount being transferred and suchlike). I don't remember the details, but unfortunately that rules out FIDO for 2FA to make transactions. For initial authentication it would work, but it would have to be yet another system on top of the 2FA they have to use for transaction validation.
    - insomniacity 4y ago
      That makes sense, thanks for the info.
arpinum 4y ago
The 250+ page analysis of the incident was an excellent insight into how large IT projects fail: https://www.tsb.co.uk/news-releases/slaughter-and-may/slaughter-and-may-report.pdf https://www.tsb.co.uk/news-releases/slaughter-and-may/slaugh... money quote: > This situation has all the hallmarks of business management strong-arming the IT organization into an unrealistic timeline. When business leaders push for overly-aggressive timelines, or regulators ask for multiple competing risk frameworks and excessive after-the-fact incident reporting, this all puts a strain on the delivery organization’s ability to untangle the complexity before ‘go live’.
- zeristor 4y ago
  The report is by Slaughter & May, one of the more delightful company names in the City of London. My understanding was that they’re a law firm, perhaps they’ve also branched into IT consultancy?
  - Nextgrid 4y ago
    > they’re a law firm, perhaps they’ve also branched into IT consultancy? And that's exactly how such IT disasters begin.
  - graderjs 4y ago
    No this kind of thing is called an audit and law firms are typically involved.
  - joshuaissac 4y ago
    Law firms are often hired to conduct independent reviews when things go wrong or when allegations of wrongdoing are made, e.g. RBS in 2013,[1] RICS in 2018,[2] Baker McKenzie in 2018[3], and UNICEF in 2020.[4][5] 1. https://www.natwestgroup.com/news-and-insights/feature-content/our-updates/2011-2020/clifford-chance-review.html https://www.natwestgroup.com/news-and-insights/feature-conte... 2. https://www.rics.org/uk/about-rics/corporate-governance/independent-external-review-information-centre/ https://www.rics.org/uk/about-rics/corporate-governance/inde... 3. https://www.legalbusiness.co.uk/blogs/metoo-latest-bakers-appoints-simmons-to-review-handling-of-complaint-against-partner-accused-of-sexual-assault/ https://www.legalbusiness.co.uk/blogs/metoo-latest-bakers-ap... 4. https://www.unicef.org.uk/press-releases/unicef-uk-confirms-launch-of-independent-review/ https://www.unicef.org.uk/press-releases/unicef-uk-confirms-... 5. https://www.civilsociety.co.uk/news/unicef-appoints-different-law-firm-for-review-into-bullying-allegations.html https://www.civilsociety.co.uk/news/unicef-appoints-differen...
VincentEvans 4y ago
I sneer at the emphasis on “1.4 billion records!” in the article as if it’s a lot. At a recent place of employment I created and was responsible for a database that had about that many records and in actuality was a single 2tb postgres db and completely unremarkable. I never claimed to have worked with big data.
- teleforce 4y ago
  Just wondering if the migration disaster at this scale can be avoided using modern cluster and orchestration technology like Kubernetes?
  - jhugo 4y ago
    Without even having got around to reading the whole report yet, I can promise you that a f*ckup on this scale cannot be avoided solely through technology decisions. The problem was (is always) with the people and the structures they were working in.
  - usr1106 4y ago
    No technology can compensate poor planning and technical incompetence. From all I read that was the root cause of the problem. So the same people and processes using Kubernetes: No. (Of course this is just speculation. I have no insider knowledge.)
  - JackFr 4y ago
    To the contrary, switching to a new technology is a favorite reductive excuse of poor management. They choose one early technical decision and try to hang all the failure on that. As a sibling comment states a screw up of this magnitude is never simply a technology issue — it requires bad management at many levels.
  - ilyt 4y ago
    k8s does absolutely jack shit when it comes to data migration so not really. Still need to write all the procedures, test it, then do it on live system again. It might make prototyping easier (...or harder) but that's about it
jesusthatsgreat 4y ago
Sounds like classic A type personalities with zero technical chops deciding how long a technical project should take to further their own agenda: > The Migration Programme experienced delays from the outset and fell behind the IMP timings. While progress had been made, on 20 September 2017 the firm decided that the Migration Programme would have to be re-planned. However, nine days after it had resolved to re-plan, and before it had concluded its re-planning exercise, TSB publicly announced it would now migrate in Q1 2018.
azalemeth 4y ago
I have a bank account with TSB and got compensation as a result of this mix-up. Some rather personal experiences of the fiasco: – Rather pointlessly, the website changed from being mostly static to entirely written in a very JS-heavy, "dynamic" way. I still can't use it in my normal browser (FF) with its extensions because it relies heavily upon CORS requests and referrer information that my somewhat privacy-paranoid extensions block. – This was introduced at the time of the switchover, and until that point the IT system used looked identical between Lloyds, TSB and Halifax / BOS systems (I have accounts with some of those) – The online browser-based system was telemetry and JS heavy, replacing a far leaner page – I was unable to log in during the time of the fiasco, mostly due to 403 errors or timeouts. Often the page would just hang as an async request wasn't answered. – Once I did manage to log in, I was amazed to see another person's account details (!!!), replete with (their) name and statement. – I was unable to use online banking to pay bills or check my balance – I could see someone else's account in detail but was too honest to do anything with that knowledge. I can't remember if my card stopped working but I was effectively forced to make other arrangements for quite an extended period of time.
- jbkkd 4y ago
  Honest question - why do you still have an account with them?
  - greggarious 4y ago
    Why don't people get divorced or break up? Abusive relationships can trap you, personal or business.
  - PeterisP 4y ago
    In general, banks compete on other attributes - a small difference in a mortgage interest rate is a lot of money and makes up for a HUGE difference in the quality of internet services; and whenever loan market becomes tight (and thus it's not attractive to refinance to another bank) people are pretty much locked in.
    - 4y ago
jayceedenton 4y ago
I will always remember this incident as the time when the UK general public were exposed, en masse, to Spring error messages. The confusion caused by ordering a member of the general public not to request a bean from a bean factory in a destroy method implementation still makes me laugh, even now.
- jw1224 4y ago
  “Just want to see my balance and these guys @tsb think I'm robbing a bean factory with a bomb, jesus” Brilliant. Reminisce with a screenshot here: https://twitter.com/thejackthomson_/status/988564354512687104 https://twitter.com/thejackthomson_/status/98856435451268710...
- Nextgrid 4y ago
  The fact that those messages were visible to external users is a major problem and sign of incompetence.
- ilyt 4y ago
  I remember when we were telling devs to stop returning java 503's with stacktraces to the user. Devs fixed it by returning 200's with stack traces. And as page was ESI stitched together on Varnish, when they fucked up there wasn't just a stacktrace, but a bunch of different ones in various parts of the page.
  - ywei3410 4y ago
    I really feel this is an infra problem rather than a dev problem though; the reverse-proxy should strip 5xx response bodies before the egress no?
    - ilyt 4y ago
      Eh, depends, for some apps 503 is legit response that should be returned to customer/app, in other cases it's app being badly designed. We did that few times when it made sense but in most it does not. There is no error code for "down for maintenance" so 503 was also used sometimes for that purpose (although we recommended devs to just fail healthchecks so loadbalancer displays its own error page in those cases) The other problem (let's just say devs were not... that great with architecture) was that they were getting exceptions in the logs without attached URL and other metadata so they kinda wanted to get that exception directly on the webpage. I'll paint you a picture of how shoddy some stuff was: they were using a templating language to generate JSONs (and had many bugs in it too) for years, instead of just natively encoding some JSON in Java, coz of party shoddy architecture, partly years-long war between frontend and backed dept. They migrated to Git in... 2018 I think ? No CI/CD of any sort till recently. End result (of them insisting that they will send the exception to frontend to be seen) was actually exceptions being signed and encrypted blob wrapped with a bunch of JS that gathers all the errors (incl. errors that JS on site might've done), adds all the metadata it sees from "browser perspective" and sends it to monitoring endpoint where that is shoved into ES cluster. Sort of retarded version of distributed tracing that is now in vogue... done somewhere in 2013. But it did catch a bunch of bugs that were "only" showing to users in browser
UltraViolence 4y ago
It's not just the CEO who should've been fired. The COO, CTO and CIO also should've left the building with a cardboard box in their hands. This is a shameless indulgence in incompetence and recklessness. They didn't even bother to test large swaths of the transitional data or have a fallback plan if things went wrong. Most likely their customers will now simply leave and the bank will be shut down.
deleted 4y ago
[deleted]
mellosouls 4y ago
Discussed variously at the time, eg. https://news.ycombinator.com/item?id=16910947 https://news.ycombinator.com/item?id=16910947 Others https://hn.algolia.com/?dateEnd=1600300800&dateRange=custom&dateStart=1515110400&page=0&prefix=false&query=tsb&sort=byPopularity&type=story https://hn.algolia.com/?dateEnd=1600300800&dateRange=custom&...
JCM9 4y ago
I remember when this all happened. Would be interesting if it was the result of some really interesting technical bug that nobody could have foresaw followed by a fascinating effort to save the migration. In fact it was all quite boring and simply the result of the sheer incompetence of the bank’s leadership in running bank IT.

139 points by shaman1 4y ago

2008: "The UK-based IT department of the fifth largest bank continues to dwindle as more jobs go overseas... This round of cuts, starting in June and lasting 12 months, involves up to 250 permanent IT roles and 200 contractors from the bank's technical delivery division, responsible for software development and design." [1] 2018: "Timeline of trouble: how the TSB IT meltdown unfolded". [2] It's probably more complicated than that, but perhaps not much more complicated. [1] https://www.itpro.co.uk/197982/lloyds-tsb-cuts-more-uk-it-jobs https://www.itpro.co.uk/197982/lloyds-tsb-cuts-more-uk-it-jo... [2] https://www.theguardian.com/business/2018/jun/06/timeline-of-trouble-how-the-tsb-it-meltdown-unfolded https://www.theguardian.com/business/2018/jun/06/timeline-of...

makomk 4y ago
That probably has very little to do with it. The immediate cause of all the problems was that Lloyds TSB was forcibly split up in order to try and increase competition and the Lloyds half kept the IT department, and when the TSB half tried to move over to the existing IT platform of their new parent company everything broke.
- lmm 4y ago
  True as far as it goes, but "everything broke" as a predictable result of poor decisions they made, such as moving everyone over in one go.
  - sofixa 4y ago
    > such as moving everyone over in one go. It's the core banking system of a big bank. Handling consistent state between the old and new systems while progressively migrating customers would probably have been extremely complex. They also get to have a maintenance window (few people will complain if they get warned their bank and all transactions won't work for 2 hours in the middle of the night on Monday). A "big bang" migration makes more sense, if everything is properly prepared and tested, which it wasn't.
    - JackFr 4y ago
      > A "big bang" migration makes more sense, if everything is properly prepared and tested, which it wasn't. That sounds like ‘A “big bang” migration makes more sense, if it works.’ Your rollback plan should never be an afterthought. Your rollback plan should be designed like you expect to use it. If your rollback plan is “Burn the ships” and fix-forward, you shouldn’t be working at a bank.
      - flerchin 4y ago
        Definitely been the plan at a variety of places I've worked. Including a bank.
    - PeterisP 4y ago
      I've been a part of multiple "big bang" migrations in banking (generally scheduled to the coincidences of local banking holidays next to a weekend, so you can afford multiple days of semi-downtime) and all of them had explicit multiple testing gates for potential rollback during the migration, where after pretty much all the stuff is on the new system, the board convenes and after looking at the difficulties (there inevitably are some unexpected difficulties) make the decision whether they "accept" the switch to the new system or postpone the switch. Part of your preparation and testing is the rollback of a partial migration - if you're irreversibly committed to the "big bang" before you know its outcome, then your preparation and testing has failed.
  - kitd 4y ago
    Note that "they" in this case is the new parent company. IIRC, they were a fairly new bank, heavily reliant on technology. They had the tech but not the customers. TSB were the opposite. The parent thought their tech, which up to that point was only dealing with 100,000s of records, could deal with billions with little change. They were spectacularly wrong and it showed in all testing. But their management pushed ahead to go live anyway. Their actions are not far from criminal negligence IMHO.
    - deleted 4y ago
      [deleted]
    - nerdawson 4y ago
      TSB’s parent company is Sabadell which is a massive Spanish corporation. I was a Solbank (one of their brands) customer and can safely say tech isn’t their strong point. Awful UX and a pain to deal with.
- 908B64B197 4y ago
  It's funny how there's always a cheap offshored bodyshop involved in these stories yet it's never their fault. Cue in the 9$/hour indian coders working for Boeing. Here's the thing I've learned over the years: Never touch offshored code. Always go for a complete re-write. Don't add features to it, don't refactor it, don't extend it. Just re-write. In my experience it's the best approach. I know guys who made it their whole business to go and completely re-write projects from scratch after offshoring efforts failed.
varispeed 4y ago
There was also a heavy enforcement of IR35 in the banking sector, so that substantially reduced the access to talent pool. There was also a tightening of posted worker regulations, so that banks couldn't ship workers from overseas as a source of cheap talented workforce.

kmlx 4y ago

i remember this. for at least a week people couldn't access their money. it was chaos. the bank lost lots of money and customers due to this botched transfer.

rwmj 4y ago
People were seeing random balances in their accounts, or seeing other people's accounts, according to some news reports at the time: https://www.bbc.co.uk/news/uk-43860449 https://www.bbc.co.uk/news/uk-43860449
jesusthatsgreat 4y ago
Not your keys, not your coins.
- maxehmookau 4y ago
  Nobody actually lost any money. The "coins" were still mine (I was affected) and heavy banking regulation in the UK probably reduced the harm to effectively zero for the vast majority of retail customers. I received compensation for the harm caused and got all of my money back.

orf 4y ago

The actual report by the FCA: https://www.fca.org.uk/publication/final-notices/tsb-bank-plc-2022.pdf https://www.fca.org.uk/publication/final-notices/tsb-bank-pl...

shaman1 4y ago
Quite thorough report, some points that stand out from the summary: >SABIS was TSB’s principal outsourced provider >SABIS relied extensively on 85 third parties (TSB’s fourth parties) to deliver the systems required for the migration and the operation of the platform, which required it to act as a service aggregator. It amazes me the sheer complexity of a retail bank software system and I suspect most of it is due to legacy systems, legal requirements and lack of regular spring cleaning.

sherr 4y ago

Not my main bank so this did not affect me badly but their online (web based) banking portal is still glitchy and not very good.

MaxBarraclough 4y ago
Even if it's not your main account, it could be bad if your account details were leaked.

petepete 4y ago

Hopefully Virgin Money will get one too. They broke their Android app earlier this year and since they make you verify web logins using the app I was unable to access any of my business accounts for ~3 weeks. If something really urgent had come up I could have done what I needed via telephone banking or in a branch, but it was a huge pain in the arse because of a single point of failure. Just let me use a Yubikey as my second factor damnit.

insomniacity 4y ago
+1 on the Yubikey. I'm pretty good at moving my savings around and getting the best interest rate possible - the side effect is a ton of accounts, which means I'm drowning in 'secure memorable passcode key PINs' and my SMS inbox is full of SMS 2FA codes, and I'm wondering what it would take to get a bank to offer Webauthn/FIDO. How about a website where we pledged to open an account and deposit £X into savings, or switch current account, if they offered Webauthn/FIDO?
- rlpb 4y ago
  I'd love FIDO for online banking auth. But AIUI, there's some EU regulation that requires 2FA, but that 2FA must also verify some other data (like the recipient of a transfer, amount being transferred and suchlike). I don't remember the details, but unfortunately that rules out FIDO for 2FA to make transactions. For initial authentication it would work, but it would have to be yet another system on top of the 2FA they have to use for transaction validation.
  - insomniacity 4y ago
    That makes sense, thanks for the info.

arpinum 4y ago

The 250+ page analysis of the incident was an excellent insight into how large IT projects fail: https://www.tsb.co.uk/news-releases/slaughter-and-may/slaughter-and-may-report.pdf https://www.tsb.co.uk/news-releases/slaughter-and-may/slaugh... money quote: > This situation has all the hallmarks of business management strong-arming the IT organization into an unrealistic timeline. When business leaders push for overly-aggressive timelines, or regulators ask for multiple competing risk frameworks and excessive after-the-fact incident reporting, this all puts a strain on the delivery organization’s ability to untangle the complexity before ‘go live’.

zeristor 4y ago
The report is by Slaughter & May, one of the more delightful company names in the City of London. My understanding was that they’re a law firm, perhaps they’ve also branched into IT consultancy?
- Nextgrid 4y ago
  > they’re a law firm, perhaps they’ve also branched into IT consultancy? And that's exactly how such IT disasters begin.
- graderjs 4y ago
  No this kind of thing is called an audit and law firms are typically involved.
- joshuaissac 4y ago
  Law firms are often hired to conduct independent reviews when things go wrong or when allegations of wrongdoing are made, e.g. RBS in 2013,[1] RICS in 2018,[2] Baker McKenzie in 2018[3], and UNICEF in 2020.[4][5] 1. https://www.natwestgroup.com/news-and-insights/feature-content/our-updates/2011-2020/clifford-chance-review.html https://www.natwestgroup.com/news-and-insights/feature-conte... 2. https://www.rics.org/uk/about-rics/corporate-governance/independent-external-review-information-centre/ https://www.rics.org/uk/about-rics/corporate-governance/inde... 3. https://www.legalbusiness.co.uk/blogs/metoo-latest-bakers-appoints-simmons-to-review-handling-of-complaint-against-partner-accused-of-sexual-assault/ https://www.legalbusiness.co.uk/blogs/metoo-latest-bakers-ap... 4. https://www.unicef.org.uk/press-releases/unicef-uk-confirms-launch-of-independent-review/ https://www.unicef.org.uk/press-releases/unicef-uk-confirms-... 5. https://www.civilsociety.co.uk/news/unicef-appoints-different-law-firm-for-review-into-bullying-allegations.html https://www.civilsociety.co.uk/news/unicef-appoints-differen...

VincentEvans 4y ago

I sneer at the emphasis on “1.4 billion records!” in the article as if it’s a lot. At a recent place of employment I created and was responsible for a database that had about that many records and in actuality was a single 2tb postgres db and completely unremarkable. I never claimed to have worked with big data.

teleforce 4y ago
Just wondering if the migration disaster at this scale can be avoided using modern cluster and orchestration technology like Kubernetes?
- jhugo 4y ago
  Without even having got around to reading the whole report yet, I can promise you that a f*ckup on this scale cannot be avoided solely through technology decisions. The problem was (is always) with the people and the structures they were working in.
- usr1106 4y ago
  No technology can compensate poor planning and technical incompetence. From all I read that was the root cause of the problem. So the same people and processes using Kubernetes: No. (Of course this is just speculation. I have no insider knowledge.)
- JackFr 4y ago
  To the contrary, switching to a new technology is a favorite reductive excuse of poor management. They choose one early technical decision and try to hang all the failure on that. As a sibling comment states a screw up of this magnitude is never simply a technology issue — it requires bad management at many levels.
- ilyt 4y ago
  k8s does absolutely jack shit when it comes to data migration so not really. Still need to write all the procedures, test it, then do it on live system again. It might make prototyping easier (...or harder) but that's about it

jesusthatsgreat 4y ago

Sounds like classic A type personalities with zero technical chops deciding how long a technical project should take to further their own agenda: > The Migration Programme experienced delays from the outset and fell behind the IMP timings. While progress had been made, on 20 September 2017 the firm decided that the Migration Programme would have to be re-planned. However, nine days after it had resolved to re-plan, and before it had concluded its re-planning exercise, TSB publicly announced it would now migrate in Q1 2018.

azalemeth 4y ago

I have a bank account with TSB and got compensation as a result of this mix-up. Some rather personal experiences of the fiasco: – Rather pointlessly, the website changed from being mostly static to entirely written in a very JS-heavy, "dynamic" way. I still can't use it in my normal browser (FF) with its extensions because it relies heavily upon CORS requests and referrer information that my somewhat privacy-paranoid extensions block. – This was introduced at the time of the switchover, and until that point the IT system used looked identical between Lloyds, TSB and Halifax / BOS systems (I have accounts with some of those) – The online browser-based system was telemetry and JS heavy, replacing a far leaner page – I was unable to log in during the time of the fiasco, mostly due to 403 errors or timeouts. Often the page would just hang as an async request wasn't answered. – Once I did manage to log in, I was amazed to see another person's account details (!!!), replete with (their) name and statement. – I was unable to use online banking to pay bills or check my balance – I could see someone else's account in detail but was too honest to do anything with that knowledge. I can't remember if my card stopped working but I was effectively forced to make other arrangements for quite an extended period of time.

jbkkd 4y ago
Honest question - why do you still have an account with them?
- greggarious 4y ago
  Why don't people get divorced or break up? Abusive relationships can trap you, personal or business.
- PeterisP 4y ago
  In general, banks compete on other attributes - a small difference in a mortgage interest rate is a lot of money and makes up for a HUGE difference in the quality of internet services; and whenever loan market becomes tight (and thus it's not attractive to refinance to another bank) people are pretty much locked in.
  - 4y ago

jayceedenton 4y ago

I will always remember this incident as the time when the UK general public were exposed, en masse, to Spring error messages. The confusion caused by ordering a member of the general public not to request a bean from a bean factory in a destroy method implementation still makes me laugh, even now.

jw1224 4y ago
“Just want to see my balance and these guys @tsb think I'm robbing a bean factory with a bomb, jesus” Brilliant. Reminisce with a screenshot here: https://twitter.com/thejackthomson_/status/988564354512687104 https://twitter.com/thejackthomson_/status/98856435451268710...
Nextgrid 4y ago
The fact that those messages were visible to external users is a major problem and sign of incompetence.
ilyt 4y ago
I remember when we were telling devs to stop returning java 503's with stacktraces to the user. Devs fixed it by returning 200's with stack traces. And as page was ESI stitched together on Varnish, when they fucked up there wasn't just a stacktrace, but a bunch of different ones in various parts of the page.
- ywei3410 4y ago
  I really feel this is an infra problem rather than a dev problem though; the reverse-proxy should strip 5xx response bodies before the egress no?
  - ilyt 4y ago
    Eh, depends, for some apps 503 is legit response that should be returned to customer/app, in other cases it's app being badly designed. We did that few times when it made sense but in most it does not. There is no error code for "down for maintenance" so 503 was also used sometimes for that purpose (although we recommended devs to just fail healthchecks so loadbalancer displays its own error page in those cases) The other problem (let's just say devs were not... that great with architecture) was that they were getting exceptions in the logs without attached URL and other metadata so they kinda wanted to get that exception directly on the webpage. I'll paint you a picture of how shoddy some stuff was: they were using a templating language to generate JSONs (and had many bugs in it too) for years, instead of just natively encoding some JSON in Java, coz of party shoddy architecture, partly years-long war between frontend and backed dept. They migrated to Git in... 2018 I think ? No CI/CD of any sort till recently. End result (of them insisting that they will send the exception to frontend to be seen) was actually exceptions being signed and encrypted blob wrapped with a bunch of JS that gathers all the errors (incl. errors that JS on site might've done), adds all the metadata it sees from "browser perspective" and sends it to monitoring endpoint where that is shoved into ES cluster. Sort of retarded version of distributed tracing that is now in vogue... done somewhere in 2013. But it did catch a bunch of bugs that were "only" showing to users in browser

UltraViolence 4y ago

It's not just the CEO who should've been fired. The COO, CTO and CIO also should've left the building with a cardboard box in their hands. This is a shameless indulgence in incompetence and recklessness. They didn't even bother to test large swaths of the transitional data or have a fallback plan if things went wrong. Most likely their customers will now simply leave and the bank will be shut down.

deleted 4y ago

[deleted]

mellosouls 4y ago

Discussed variously at the time, eg. https://news.ycombinator.com/item?id=16910947 https://news.ycombinator.com/item?id=16910947 Others https://hn.algolia.com/?dateEnd=1600300800&dateRange=custom&dateStart=1515110400&page=0&prefix=false&query=tsb&sort=byPopularity&type=story https://hn.algolia.com/?dateEnd=1600300800&dateRange=custom&...

JCM9 4y ago

I remember when this all happened. Would be interesting if it was the result of some really interesting technical bug that nobody could have foresaw followed by a fascinating effort to save the migration. In fact it was all quite boring and simply the result of the sheer incompetence of the bank’s leadership in running bank IT.