13 ms·

Due to failure in the IT system, it is not possible to run any trains today

651 points by ar0 4y ago

rbreaves 4y ago
No - due to failure with management your IT failed.
samspenc 4y ago
I do hope they have a more detailed RCA at some point, right now this is the only helpful paragraph in the article about what happened: > The IT failure occurred at the end of the morning. It affected the system that generates up-to-date schedules for trains and staff. This system is important for safe and scheduled operations: if there is an incident somewhere, the system adjusts itself accordingly. This was not possible due to the failure.
- qiskit 4y ago
  I was expecting more as well. Not much information to glean from a barebones press release. If I had to guess, it probably was an issue with scheduling/timetable software rather than anything to do with the trains, rails, etc. Nothing exceedingly seriously or difficult to correct.
  - XorNot 4y ago
    Keep in mind that train schedules control what trains are on what tracks when, so while their are literally fail-safes in the system, this is not exactly far removed from being "safety critical".
    - TylerE 4y ago
      That hasn’t been true in Europe for over a hundred years, and exceedingly rare in the US these days.
      - benjaminwootton 4y ago
        Isn’t this exactly what the press release states? “ if there is an incident somewhere, the system adjusts itself accordingly. This was not possible due to the failure.”
        TylerE 4y ago
        No, it’s talking about crew scheduling/allocation , not signaling. The signaling system is totally fail safe. Even if that system totally went out, drivers are required to know where all the signals are, and to treat an unlit signal as a red.
        throwawayboise 4y ago
        What if the failure lit all the signals in green?
        TylerE 4y ago
        It can’t, they’re not wired like that. They’re engineered so any fault shows either red or nothing.
      - innocenat 4y ago
        The train schedule system data aren't fed into the Automatic Route Setting system in Europe and US?
        TylerE 4y ago
        Sure, but failing that everything will just stop, not crash into other like GP implied
bmitc 4y ago
More and more we are seeing "the computer says no" in our lives. We've replaced human activity and decision making with computerized versions, which are nothing more than the automation of stored human knowledge and routines.
- sofixa 4y ago
  A much more efficient automation. For instance, running high speed rail without computer signalling would be a huge risk there's no sense in taking ( due to the speeds involved, by the time you see anything of issue like a train or damage in the tracks in front, it's too late to stop).
  - bmitc 4y ago
    I think that's a bit of a leap from what I was saying. I didn't imply that all software and computation is bad. Obviously, a human cannot live handle any control system that needs even remotely fast operation. The point is that due to scaling issues, we're removing humans from systems where humans excel or could serve as backups in the event of failure (and actually vice versa). Scale creates problems that are partially solved via software systems, which only handle a subset of the existing variety and new variety introduced by scaling. This makes the systems leaky with regards to unhandled variety and incredibly rigid (i.e., not robust) against exceptions, exceptions that could often be readily be handled by humans. By software being a snapshot of some small group of humans' knowledge of a system, it is completely unable to respond to live operational exceptions. Stafford Beer's work is interesting background on this type of thing. > A much more efficient automation. There are many, many examples where that is both not true and true. For the not true cases, just see literally any customer service interaction with a large company in the past half decade. I suppose this could be arguable though if one asks the question "efficient for who?".
    - tsimionescu 4y ago
      > There are many, many examples where that is both not true and true. For the not true cases, just see literally any customer service interaction with a large company in the past half decade. I suppose this could be arguable though if one asks the question "efficient for who?". Another example I love, that is even less ambiguous, is restaurants (traditional, not fast food) that require waiters to take orders on touchscreen PoS systems, instead of taking the orders on a notepad and inputting them later. The touch screens are incredibly slower on a bigger party ordering appetizers, a main course, drinks and desert - it takes forever to navigate back and forth and find the right item, and usually has to be done again and again for each diner. By comparison, a notepad allows the waiter to write down exactly what they hear, often just making small extra notes to indicate duplicate items. Of course, this particular one could be improved by arranging the menus more naturally for someone taking orders, and several other conveniences (such as always keeping the current order on screen, to easily tap to add "oh, and another beer for me", "I'll have the same desert as her" etc).
dt3ft 4y ago
There is a swiss company based in Zürich that develops this kind of software. I believe it was a c#, wpf application, the last time I spoke with them. The schedules that their software produces is calculated and set for several years ahead. That baffled me! The tracks and infrastructure is booked up so far ahead! I think they mentioned that the german railway uses their software, nut only the sbb.
- vanviegen 4y ago
  From what I understand, the failing system is a bit different, in that its primary purpose is to deal with outages and other unexpected events.
gridspy 4y ago
tl;dr : Restarting train schedule mid-day is too hard and unsafe. So we're sending them back to depot instead.
- amelius 4y ago
  Why "unsafe"? Isn't safety supposed to be handled at a different abstraction level (in hardware)?
  - coder543 4y ago
    Defense in depth? With this safety feature inoperable, the margin of safety is reduced. Other systems that might normally provide safety critical redundancy could be providing the sole measure of safety, with no other redundancy available in case one of those fails. “Unsafe” is always defined based on context.
    - throwaway0a5e 4y ago
      What's the point of having redundancy if you can't use it to avoid an impact to service? Edit: and to be clear, we're not talking about holding down a resetable circuit breaker to avoid being late, we're talking about the rail network of an entire nation being inoperable for a day.
      - coder543 4y ago
        The point is to still have it when you didn’t know you needed it. That’s the purpose of all redundant safety systems. The redundancy isn’t there to keep the service operating; it’s to keep people from dying. You can never know if the primary safety system is functioning perfectly, so you need other systems to be there to step in when the primary fails unexpectedly. If you detect the primary system has failed, isn’t it reasonable that you should stop operation as quickly and safely as possible, and be thankful nothing bad happened while you lacked redundancy? Any SPoF could be fatal for hundreds of people.
        4y ago
user_7832 4y ago
I was in the station for a different reason when I heard the announcement. The people entering and hearing the announcement were unsurprisingly very crestfallen, but it appears that twitter helped organize a lot of travel, with people posting where they were/going and picking up other travellers. Also probably worth noting is that trains here are used fairly frequently, however unfortunately this isn't the first time trains are stopping - for example snow is a common reason for delayed/reduced service. (Snow isn't very common here in the NLs)
- mushyhammer 4y ago
  > with people posting where they were/going and picking up other travellers. That’s ludicrous, I wish people helping other people this way was more common.
  - wolverine876 4y ago
    Imagine if we did that all the time! An online ride board.
    - mark-r 4y ago
      ... which would stop working if the computer went down.
      - shapefrog 4y ago
        You could even install rails that go along regular routes, and put on scheduled carridges that people could all jump into at designated places along the route. The computer could generate a schedule and post it publicly so people knew when they could hop on.
        wolverine876 4y ago
        You really don't see a difference between a public rideshare and public mass transit?
xiphias2 4y ago
That software bug made a lot of Hungarians miss elections today in Netherlands :(
- csomar 4y ago
  So you are saying it's a plot to change the election's outcome? edit: It was a joke. :/
  - xiphias2 4y ago
    I don't think so. Anyways the same dictatorship (Viktor Orban) got full control again of the country.
    - mardifoufs 4y ago
      Were the elections illegitimate ? I'm a complete outsider to hungarian politics so I'm not sure if this is hyperbole... but talking about how a dictatorship has been reelected again doesn't make sense to me. Unless elections in Hungary are just theather, and results are just faked like in Syria for example.
      - Timwi 4y ago
        Outright dictatorships like North Korea are getting rarer. It's more common nowadays for de-facto dictatorships to maintain an appearance of democracy by actually holding elections but either having no viable candidates other than the incumbent (e.g. Russia), or by systematically banning or even murdering all opposition (e.g. Belarus). Hungary isn't quite that far gone yet. It still has elections that technically meet the requirements of EU membership. I don't know any more details though.
      - boomboomsubban 4y ago
        It's a relatively free election, there's a strong incumbent bias that undoubtedly helps him win but there's no evidence the entire thing is rigged. Dictators can be popular, the best example is Napoleon.
deleted 4y ago
[deleted]
miggol 4y ago
My curiosity is killing me as to what exactly went wrong. And as someone living in The Netherlands I'm also kind of mad at the apparent fragility of this huge chunk of our infrastructure. If this had happened on a work day, it would have been a real national emergency, rather than just a huge national inconvenience. I would appreciate one of those post-outage "what happened" reports like with the AWS and Facebook outages last year. But outside of IT I don't think anyone really expects those over here. And there might be national security considerations preventing such disclosure until any chance of a repeat has been engineered away, at which point everyone will have forgotten.
- jokoon 4y ago
  > And there might be national security considerations That especially. I wonder how expensive that thing is going to cost.
- tjr225 4y ago
  Physical infrastructure can be fragile/fail catastrophically, as well. I recall starting a job near Seattle, 40 miles from my home in Olympia, WA and a passenger train derailed onto the highway killing a few people and backing up traffic for days. That was not a pleasant first day! https://en.wikipedia.org/wiki/2017_Washington_train_derailment https://en.wikipedia.org/wiki/2017_Washington_train_derailme...
  - Wowfunhappy 4y ago
    I assume, however, that this didn't ground every single train in the state!
    - noizejoy 4y ago
      I believe most trains in the world are grounded to this day. (Mixed metaphors always make me smile.)
      - axiolite 4y ago
throw7 4y ago
Whatever happened, I actually kinda like the wording in this notice in that it did not have the word "glitch".
- qwertox 4y ago
  The short sentences were what got me.
spac 4y ago
I wonder if this is connected to a similar incident in Italy. https://news.yahoo.com/italys-state-railway-may-target-163417138.html https://news.yahoo.com/italys-state-railway-may-target-16341...
- mushyhammer 4y ago
  Considering the state of Italian railways IT, I wonder if the attack was accomplished by entering admin/admin in some login box.
  - denysvitali 4y ago
    I think it's more likely that they didn't have any password at all :)
JaimeThompson 4y ago
Maybe they outsourced running the rail system to the people that run the Texas power grid?
stuckindoors 4y ago
There was a massive southwest airlines IT failure compounded by weather issues over the weekend. This cancelled numerous flights but also snowballed into most of the southwest flights being delayed. We were lucky to return home. The news is only reporting weather delays yet for those at the airport this weekend...it was clearly larger. If it is true that Netherland's and Italy's transport infrastructure were also down I wonder if Russian supported hackers are to blame.
- wsc981 4y ago
  > If it is true that Netherland's and Italy's transport infrastructure were also down I wonder if Russian supported hackers are to blame. It's been Russian hackers so many times last few years (Hunter Biden's laptop, Trump elections, etc...), it's a sure bet this time it's the same. /s Nobody ever got fired for blaming Russians.
  - BlueTemplar 4y ago
    Their bad reputation is their own damn fault. See also : assassinations - maybe the death of the top polish government & military and Total's CEO in plane crashes were just freak accidents, but a nagging suspicion that Russia was involved will always remain...
- selimthegrim 4y ago
  Southwest has put something out about it so it's in the press/Twitter for sure.
rossdavidh 4y ago
As a developer, any day in which you don't see a national headline about how your f*#(up has just made a lot of other people sad and/or angry, is a day that could have been worse.
323 4y ago
In 2019 there was a power blackout in London. When power was restored, a number of new trains refused to start without a technician physically coming over with a diagnostics laptop to them because power fluctuated outside allowed bounds or something before the blackout.
- gridspy 4y ago
  It's likely the same power fault could occur in a single train in other circumstances. When an electric train cannot access the grid reliably it would become far more dangerous - no more regenerating brakes, no reliable acceleration ... Better to force the operators to fix the boom arm / whatever other cause of power glitches with fault codes like this. Even if they false-trip when there are grid level outages.
  - Symbiote 4y ago
    No, it was considered a design fault with these particular trains, and fixed soon afterwards. Being unable to accelerate or unable to use regenerative braking is not in any way a safety problem. Either situation can happen at any time. (Electric trains have banks of resistors used to dump the unwanted kinetic energy when it's not possible to return it to the power supply, and this was the normal method for decades before regenerative braking became common.)
    - iggldiggl 4y ago
      > Electric trains have banks of resistors used to dump the unwanted kinetic energy when it's not possible to return it to the power supply On AC-electrified railways (especially those with their own independent power supply network like Germany/Austria/Switzerland), resistors are actually rather uncommon on stock capable of regenerative braking, because the OHLE not being able to accept regenerated power is a relatively rare occurrence. Though of course in that case the conventional friction brakes are of course fully rated to bring the train to halt, only with a bit more wear and tear. DC-electrified railways are a different matter – transmission losses are higher, power supply sections are smaller and substations often not capable of returning any surplus power back into higher network layers of the power supply systems, so the OHLE/third rail not being able to accept the power generated by regenerative braking happens much more frequently and so trains continue carrying braking resistors as a fallback system.
mostertoaster 4y ago
Probably Putins fault. Conspiracy theories are so fun.
rdxm 4y ago
ncmncm 4y ago
It wasn't the IT failure that kept the trains from running. It was management failure, not ensuring they could run without the IT system.
- lucasyvas 4y ago
  Now we're getting somewhere!
- Tepix 4y ago
  That sounds unrealistic given the size of the problem.
  - Aeolun 4y ago
    They did it before the whole thing was automated. Surely they can do it again.
    - CyanBird 4y ago
      Sure, *but not on a single day without any warning what's ever*
      - Aeolun 4y ago
        I’m sure nobody raised this issue before. They just didn’t care “Don’t worry, the system will always work”. But here we are.
    - innocenat 4y ago
      The automation allowed running more trains than ever before using less personnel than ever before. You want the train company to keep 2x-3x personnel on hand just for days like this? Not to mentioned transition time.
      - ncmncm 4y ago
        So, run fewer trains. Fewer is better than none.
chevman 4y ago
It’s always DNS!
- comprev 4y ago
  Does Not Schedule? :-)
  - chevman 4y ago
    Where is the DNS test environment?????????? :)
deleted 4y ago
[deleted]
supernova87a 4y ago
While having algorithms and computers dictating every aspect of train movements, crew scheduling, resource allocation is efficient and in a sense optimal, I think there's something compelling to be said about having a system that can survive with human-rememberable patterns when such an optimizer fails. That it can fall back on (or have at the core) simple patterns that people can still operate without algorithmic intervention. I.e. the algorithmic tuning enhances at the edges, and isn't at its core some tangled non-interpretable system that someone can't operate without computer assistance. The idea of a protected set of core routes and staffing patterns that operates without having to be computer-determined? I used to be so impressed at European train stations that they had a single sheet that would give the entire day's arrivals and departures, down to which track. I assume that behind that were schedules that weren't so far from being hand-calculated with equivalent sheets that told people and equipment where to be at what times. I think we lose something when we jump so far forward that you cannot fall back gracefully without a system completely breaking down.
- pjmlp 4y ago
  There is an Isac Asimov book about this. A crew is stranded in space, because of a computer malfunction and no one on board remembers how they used to be programmed.
  - at-fates-hands 4y ago
    Is that "The Last Question"? https://en.wikipedia.org/wiki/The_Last_Question https://en.wikipedia.org/wiki/The_Last_Question
    - Tenoke 4y ago
      Are you posting it reflexively because 9 out of 10 times Asimov is mentioned that's the story people talk about? Because that's definitely not the plot of it.
      - andybak 4y ago
Agentlien 4y ago
This reminds me of the story one of my university teachers used to introduce himself. He was the most brilliant educator I've had, really passionate about making sure people could actually follow and so humble. The course was called "Digital and Computer Technology" and it was a first year course teaching you to build a CPU starting from logic gates and then program it with assembly. He said that when he had recently graduated and just moved from Norway to Sweden his first job here was as a technician for the local railway. One of his first assignments was a call where the computer controlling all the track switches had stopped working. Luckily there was a backup, but they needed him to fix it immediately. He arrived, took a look at the computer and its backup running next to it. He started out by measuring voltages on both, comparing to see what the difference was. After a while some men in suits came in and asked how things were going. He said it was all good, they said it wasn't because now all trains had stopped. Apparently, he had short circuited the backup. "And that's when I decided to go for a theoretical career!" My teacher happily concluded. The classroom was left in a stunned silence.
- Saser 4y ago
  Rolf Snedsböl? :-) Agreed that he was one of the best educators (if not _the_ best) I had
  - Agentlien 4y ago
    Yes, Rolf Snedsböl! I absolutely loved his classes. I bumped into him a few years ago when I was holding a recruitment thing for Ghost Games and it happened to be just after his lecture. I told him how much I enjoyed his courses.
    - RonaldOlzheim 4y ago
      Did he wrote a book? Do you have a link?
  - valvar 4y ago
    An absolute legend! My favourite of his jokes: "when I moved to Sweden from Norway, the average IQ went up in both countries".
reilly3000 4y ago
I think it’s worth mentioning that while the IT system couldn’t coordinate the trains safely, it was a management decision to stop them from running. Coordination can happen myriad ways. For example, trains could operate according to their normal schedule, and if one needed to deviate they could radio in, and an operator could let downstream trains know to proceed accordingly. Maybe simple cell-phone GPS pings could suffice. Perhaps the state of the system was in such chaos that such orchestration wasn’t possible with the staff on hand. That said, a truly resilient system should have multiple coordination channels to fall back upon.
- mnumber 4y ago
  Computers are great at ensuring your workforce cannot function without them. This is a small price to pay.
- throwaway48375 4y ago
  The cost of a single mishap is too high. It's just not worth it.
- gpvos 4y ago
  The drivers and conductors would like to end up at home at the end of the day. Even if you just run the normal schedule, there will always be disruptions, and there needs to be a system to adjust the planning so people and things end up at the right place at the end of the day. That was the system that failed. Have you any idea how frequent the trains in the Netherlands are and how complex the schedule? "Simply" running the existing schedule is an absolute impossibility. About the only thing they could maybe have done was to fall back on back-and-forth shuttle trains between major node stations. But that would only provide a very basic service, with incorrect train materiel being used for the normally expected number of passengers. It would be just enough to get passengers and personnel home. I would like to see something like a pen-and-paper schedule that they can fall back on, but they don't have it. (They did something like that at the first corona lockdowns, although with more planning involved.)
CaptainZapp 4y ago
Reminds me of when the entire Swiss train network shut down in 2005[0] This was, however, not due to a computer glitch, but problems in SBB's electrical grid, which lead to overload of some of the lines and then to a complete shutdown of the entire grid. Warnings were displayed on control consoles, but were drowned in 1000s of other meaningless warnings (there's a lesson for an UI designer here). I was in the ICE from Basel to Zurich at the time and after fuming for a few minutes I figured: "What can you do?" and moved to the bar wagon. There we had something of an impromptu party until there was no more power for the cash register and they stopped sales. In the end it was a (relatively) funny adventure for most and a huge embarrassment for SBB. I don't think they like to be reminded of that faithful day. [0] https://www.swissinfo.ch/eng/swiss-train-network-shuts-down/4578052 https://www.swissinfo.ch/eng/swiss-train-network-shuts-down/...
- Scoundreller 4y ago
  > There we had something of an impromptu party until there was no more power for the cash register and they stopped sales. Funny how the entire system has ground to a halt and the bartender is still worried that their cash register receipts and inventory won’t reconcile and shuts that part down too.
  - Tabular-Iceberg 4y ago
    I’m puzzled by the lack of resilience in retail sales. I thought “always be closing” was supposed to apply to all types of sales. A receipt pad and a credit card imprinter under the counter would be a very cheap backup. It would be slow, but at least they would be closing. It can even be considered a national security issue in the case of food, medicines and other vital services.
    - Freak_NL 4y ago
      This is Europe. Credit card imprinters are rare if they were ever used at all. I've never seen one. The most resilient back up (especially in the face of national crises) is cash of course, but a growing group of people have stopped carrying any cash at all, and a lot of people in power are doing their best to make sure a cashless future is nearer each day. I don't mind paying with a debit card; I just want to have a backup in place in an increasingly interconnected world where a glitch can take out any nationwide system.
yummybear 4y ago
Failed OTA update?
j0e1 4y ago
What jumps out at me after reading this notice is how it completely lacks any apology. I am not prescribing what it should say but merely observing that having lived in North America for sometime now, it seems hard for me to imagine for such a notice to not sound profusely apologetic here. Note: I do not consider words like 'unfortunately' and 'extremely unpleasant' apologetic. Also, I understand there are cultural differences between the Netherlands and the USA- no joke.
- mnumber 4y ago
  "Due to the enormous impact of the failure in the IT system, it is unfortunately not possible to run any trains today." I think the writer is more pointing the finger at IT than feeling apologetic. Imagine getting called an enormous failure by your adjacent department, that writer wants you to know who screwed up instead of apologize for their screw up. Seems more useful than an apology in my opinion.
- withinboredom 4y ago
  Why would they be sorry? They’re the ones who are missing millions? of Euros in revenue due to this issue while the passengers just had to take a bus or bike somewhere. They probably would tell you that if you had to bike, it’s healthier that way. A little bit of rain won’t make you melt. /s
- fwr 4y ago
  This is very consistent with customer service in The Netherlands - nothing is ever the fault of service providers, and you manage to convince them otherwise - you'll get nothing more than "unfortunately (...) not possible"
  - MonaroVXR 4y ago
    Customer service is done by another comapny. Source: It's me this time, NS customers service errrr....We help, it's called Webhelp.
- coolgoose 4y ago
  I don't mind the tone, I got tired of US style politically correctness just for the sake of saying it. How would "We are trully sorry for this incident" make it any better ? I do mind the fact that they don't pay the people their money back if they try to find alternatives to get to their destination. That's 100% unacceptable.
CannisterFlux 4y ago
Sounds similar to a problem that happened a couple of weeks ago in Madrid, although that was not the whole network it pretty much shut down the trains for the day. That was an IT failure too. https://www.20minutos.es/noticia/4973632/0/graves-retrasos-en-cercanias-por-una-averia-informatica-en-chamartin/ https://www.20minutos.es/noticia/4973632/0/graves-retrasos-e... (in Spanish)
- tuukkah 4y ago
  Wow, talk about a single point of failure when an issue in the systems of one station can stop all but one train line. Interestingly, it seems to be a station where tracks from six directions meet so a backup mode of trains simply passing through might be impossible: https://en.wikipedia.org/wiki/Madrid_Chamart%C3%ADn_railway_station https://en.wikipedia.org/wiki/Madrid_Chamart%C3%ADn_railway_...
comprev 4y ago
I was walking up to Amsterdam Centraal when I noticed floods of people all exiting the building... within minutes an Uber was booked to Schiphol before prices rocketed due to demand.
AndyMcConachie 4y ago
The fact that it happened on a Sunday suggests to me that this is the result of a service outage gone awry. Realize that the trains failing on a Sunday is really not a big deal. However, if the trains don't run for Monday morning rush hour that is a big deal. I bet they made a change Saturday night and something broke Sunday morning. It's now Monday morning here and the trains are back to normal in time for rush hour. So it's not really a big deal.
- regularfry 4y ago
  I have to say that I do find this framing which says "well, it didn't affect business, so no big deal" to be somewhat distasteful. It's completely dehumanising. Why run the trains at all at the weekend, if a weekend outage is no big deal?
  - tsimionescu 4y ago
    That's not the framing. For the majority of travelers, missing a weekend trip is an annoyance. In contrast, missing their work is a major headache, for them personally. It also cascades pretty badly if it happens on a lathe scale - meetings get pushed and overlap others, suplliers are not able to deliver, stalling production, financial operations get delayed, potentially affecting deadlines - it's a huge mess in everyone's life that won't just go away when trains come back. It's also as true for non-profits, state organizations and schools as it is for businesses.
lifty 4y ago
Most likely their Kubernetes cluster went down. (not a joke)
HNHatesUsers 4y ago
lqet 4y ago
Seems like trains are up and running again today as announced, with only little delays: https://travic.app/?z=9&x=628581.0&y=6814491.0 https://travic.app/?z=9&x=628581.0&y=6814491.0
- woudsma 4y ago
  I didn’t know about Travic, thanks!
qwezxcrty 4y ago
Based in my knowledge as a public transportation enthusiastist in China, I got really surprised to see a IT problem bring the entire system down. There should usually be some "more manual" backup mechanism that allows downgrade and operation of trains without computerized signaling system. By kinds of token signaling it will still be possible to run trains as long as the physical infrastructure is still there. The efficiency will be much lower and more prone to (potentialy deadly) errors, but no more trains today sounds unlikely.
- apexalpha 4y ago
  The Dutch rail network is top 3 busiest in the world. Many lines have 10+ trains in each direction, every hour. The usage of certain lines at a certain time is calculated in periods of 6 seconds! We're waaay beyond normal human coordination capacity.
NWoodsman 4y ago
This post is a fascinating use of grammatic passive voice. The hallmark is gratuitous use of "being" verbs, i.e. "is,was". Passive voice is a way to avoid using active voice, which is an implicit way to avoid responsibility for an error. "Due to the enormous impact of the failure in the IT system, it is unfortunately not possible to run any trains today." // => "Due to the enormous impact of the failure in the IT system, NS cannot run trains today." "Although the cause of the failure has now been resolved, the impact is considerable. To be able to start up reliably, systems must be updated and trains must be brought to the right place. That takes time. For our passengers, this is extremely unpleasant news." // => "Although we have fixed the cause of the failure, the failure impacted most of our customers. To fix this failure so that trains would start reliably, we needed to return all trains to a central depot, which took nearly 12 hours. We apologize for the extremely unpleasant interruption." "The expectation is that tomorrow morning the normal timetable can largely be resumed. The night trains can still run." // => We expect to resume minor operation with night trains first and then major operations tomorrow morning. "The IT failure occurred at the end of the morning. It affected the system that generates up-to-date schedules for trains and staff. This system is important for safe and scheduled operations: if there is an incident somewhere, the system adjusts itself accordingly. This was not possible due to the failure." // => The IT failure occured at 11am in the scheduling system, a major requirement for safe operation; if train operations are interrupted at one location, the system normally reroutes other trains to prevent collisions. This system failed. "The international trains are not affected by this failure. For information about the timetables of other transporters, passengers can consult the websites of these transporters." // => The failure did not affect international trains. If you have an inquiry about other transportation services, please consult those services. "The journey planner is updated." // => We updated the journey planner accordingly to account for the interruptions. // Generally the use of "passive voice" indicates poorly-taught grammar, however the use could also indicate desire to avoid responsibility.
- NWoodsman 4y ago
  The "about us" section of the webpage : "Who are we? NS is active in the public transportation sector. We encourage the use of public transportation and keep the Netherlands moving. Our travellers are our 1st, 2nd and 3rd priority in all of our activities, and we do our utmost to make their trip as pleasant and sustainable as possible from door to door." This is also fascinating. I would expect to find "NS (Netherlands Service) provides rail services for the entire country". Instead they choose vague grammar, more the idea of NS rather than what NS actually is. "I am a programmer" => "I encourage the use of computers to provide solutions for common business problems". Just a strange use of grammar.

651 points by ar0 4y ago

No - due to failure with management your IT failed.

I do hope they have a more detailed RCA at some point, right now this is the only helpful paragraph in the article about what happened: > The IT failure occurred at the end of the morning. It affected the system that generates up-to-date schedules for trains and staff. This system is important for safe and scheduled operations: if there is an incident somewhere, the system adjusts itself accordingly. This was not possible due to the failure.

qiskit 4y ago
I was expecting more as well. Not much information to glean from a barebones press release. If I had to guess, it probably was an issue with scheduling/timetable software rather than anything to do with the trains, rails, etc. Nothing exceedingly seriously or difficult to correct.
- XorNot 4y ago
  Keep in mind that train schedules control what trains are on what tracks when, so while their are literally fail-safes in the system, this is not exactly far removed from being "safety critical".
  - TylerE 4y ago
    That hasn’t been true in Europe for over a hundred years, and exceedingly rare in the US these days.
    - benjaminwootton 4y ago
      Isn’t this exactly what the press release states? “ if there is an incident somewhere, the system adjusts itself accordingly. This was not possible due to the failure.”
      - TylerE 4y ago
        No, it’s talking about crew scheduling/allocation , not signaling. The signaling system is totally fail safe. Even if that system totally went out, drivers are required to know where all the signals are, and to treat an unlit signal as a red.
        throwawayboise 4y ago
        What if the failure lit all the signals in green?
        TylerE 4y ago
        It can’t, they’re not wired like that. They’re engineered so any fault shows either red or nothing.
    - innocenat 4y ago
      The train schedule system data aren't fed into the Automatic Route Setting system in Europe and US?
      - TylerE 4y ago
        Sure, but failing that everything will just stop, not crash into other like GP implied

bmitc 4y ago

More and more we are seeing "the computer says no" in our lives. We've replaced human activity and decision making with computerized versions, which are nothing more than the automation of stored human knowledge and routines.

sofixa 4y ago
A much more efficient automation. For instance, running high speed rail without computer signalling would be a huge risk there's no sense in taking ( due to the speeds involved, by the time you see anything of issue like a train or damage in the tracks in front, it's too late to stop).
- bmitc 4y ago
  I think that's a bit of a leap from what I was saying. I didn't imply that all software and computation is bad. Obviously, a human cannot live handle any control system that needs even remotely fast operation. The point is that due to scaling issues, we're removing humans from systems where humans excel or could serve as backups in the event of failure (and actually vice versa). Scale creates problems that are partially solved via software systems, which only handle a subset of the existing variety and new variety introduced by scaling. This makes the systems leaky with regards to unhandled variety and incredibly rigid (i.e., not robust) against exceptions, exceptions that could often be readily be handled by humans. By software being a snapshot of some small group of humans' knowledge of a system, it is completely unable to respond to live operational exceptions. Stafford Beer's work is interesting background on this type of thing. > A much more efficient automation. There are many, many examples where that is both not true and true. For the not true cases, just see literally any customer service interaction with a large company in the past half decade. I suppose this could be arguable though if one asks the question "efficient for who?".
  - tsimionescu 4y ago
    > There are many, many examples where that is both not true and true. For the not true cases, just see literally any customer service interaction with a large company in the past half decade. I suppose this could be arguable though if one asks the question "efficient for who?". Another example I love, that is even less ambiguous, is restaurants (traditional, not fast food) that require waiters to take orders on touchscreen PoS systems, instead of taking the orders on a notepad and inputting them later. The touch screens are incredibly slower on a bigger party ordering appetizers, a main course, drinks and desert - it takes forever to navigate back and forth and find the right item, and usually has to be done again and again for each diner. By comparison, a notepad allows the waiter to write down exactly what they hear, often just making small extra notes to indicate duplicate items. Of course, this particular one could be improved by arranging the menus more naturally for someone taking orders, and several other conveniences (such as always keeping the current order on screen, to easily tap to add "oh, and another beer for me", "I'll have the same desert as her" etc).

dt3ft 4y ago

There is a swiss company based in Zürich that develops this kind of software. I believe it was a c#, wpf application, the last time I spoke with them. The schedules that their software produces is calculated and set for several years ahead. That baffled me! The tracks and infrastructure is booked up so far ahead! I think they mentioned that the german railway uses their software, nut only the sbb.

vanviegen 4y ago
From what I understand, the failing system is a bit different, in that its primary purpose is to deal with outages and other unexpected events.

gridspy 4y ago

tl;dr : Restarting train schedule mid-day is too hard and unsafe. So we're sending them back to depot instead.

amelius 4y ago
Why "unsafe"? Isn't safety supposed to be handled at a different abstraction level (in hardware)?
- coder543 4y ago
  Defense in depth? With this safety feature inoperable, the margin of safety is reduced. Other systems that might normally provide safety critical redundancy could be providing the sole measure of safety, with no other redundancy available in case one of those fails. “Unsafe” is always defined based on context.
  - throwaway0a5e 4y ago
    What's the point of having redundancy if you can't use it to avoid an impact to service? Edit: and to be clear, we're not talking about holding down a resetable circuit breaker to avoid being late, we're talking about the rail network of an entire nation being inoperable for a day.
    - coder543 4y ago
      The point is to still have it when you didn’t know you needed it. That’s the purpose of all redundant safety systems. The redundancy isn’t there to keep the service operating; it’s to keep people from dying. You can never know if the primary safety system is functioning perfectly, so you need other systems to be there to step in when the primary fails unexpectedly. If you detect the primary system has failed, isn’t it reasonable that you should stop operation as quickly and safely as possible, and be thankful nothing bad happened while you lacked redundancy? Any SPoF could be fatal for hundreds of people.
      - 4y ago

user_7832 4y ago

I was in the station for a different reason when I heard the announcement. The people entering and hearing the announcement were unsurprisingly very crestfallen, but it appears that twitter helped organize a lot of travel, with people posting where they were/going and picking up other travellers. Also probably worth noting is that trains here are used fairly frequently, however unfortunately this isn't the first time trains are stopping - for example snow is a common reason for delayed/reduced service. (Snow isn't very common here in the NLs)

mushyhammer 4y ago
> with people posting where they were/going and picking up other travellers. That’s ludicrous, I wish people helping other people this way was more common.
- wolverine876 4y ago
  Imagine if we did that all the time! An online ride board.
  - mark-r 4y ago
    ... which would stop working if the computer went down.
    - shapefrog 4y ago
      You could even install rails that go along regular routes, and put on scheduled carridges that people could all jump into at designated places along the route. The computer could generate a schedule and post it publicly so people knew when they could hop on.
      - wolverine876 4y ago
        You really don't see a difference between a public rideshare and public mass transit?

xiphias2 4y ago

That software bug made a lot of Hungarians miss elections today in Netherlands :(

csomar 4y ago
So you are saying it's a plot to change the election's outcome? edit: It was a joke. :/
- xiphias2 4y ago
  I don't think so. Anyways the same dictatorship (Viktor Orban) got full control again of the country.
  - mardifoufs 4y ago
    Were the elections illegitimate ? I'm a complete outsider to hungarian politics so I'm not sure if this is hyperbole... but talking about how a dictatorship has been reelected again doesn't make sense to me. Unless elections in Hungary are just theather, and results are just faked like in Syria for example.
    - Timwi 4y ago
      Outright dictatorships like North Korea are getting rarer. It's more common nowadays for de-facto dictatorships to maintain an appearance of democracy by actually holding elections but either having no viable candidates other than the incumbent (e.g. Russia), or by systematically banning or even murdering all opposition (e.g. Belarus). Hungary isn't quite that far gone yet. It still has elections that technically meet the requirements of EU membership. I don't know any more details though.
    - boomboomsubban 4y ago
      It's a relatively free election, there's a strong incumbent bias that undoubtedly helps him win but there's no evidence the entire thing is rigged. Dictators can be popular, the best example is Napoleon.

deleted 4y ago

[deleted]

miggol 4y ago

My curiosity is killing me as to what exactly went wrong. And as someone living in The Netherlands I'm also kind of mad at the apparent fragility of this huge chunk of our infrastructure. If this had happened on a work day, it would have been a real national emergency, rather than just a huge national inconvenience. I would appreciate one of those post-outage "what happened" reports like with the AWS and Facebook outages last year. But outside of IT I don't think anyone really expects those over here. And there might be national security considerations preventing such disclosure until any chance of a repeat has been engineered away, at which point everyone will have forgotten.

jokoon 4y ago
> And there might be national security considerations That especially. I wonder how expensive that thing is going to cost.
tjr225 4y ago
Physical infrastructure can be fragile/fail catastrophically, as well. I recall starting a job near Seattle, 40 miles from my home in Olympia, WA and a passenger train derailed onto the highway killing a few people and backing up traffic for days. That was not a pleasant first day! https://en.wikipedia.org/wiki/2017_Washington_train_derailment https://en.wikipedia.org/wiki/2017_Washington_train_derailme...
- Wowfunhappy 4y ago
  I assume, however, that this didn't ground every single train in the state!
  - noizejoy 4y ago
    I believe most trains in the world are grounded to this day. (Mixed metaphors always make me smile.)
    - axiolite 4y ago

throw7 4y ago

Whatever happened, I actually kinda like the wording in this notice in that it did not have the word "glitch".

qwertox 4y ago
The short sentences were what got me.

spac 4y ago

I wonder if this is connected to a similar incident in Italy. https://news.yahoo.com/italys-state-railway-may-target-163417138.html https://news.yahoo.com/italys-state-railway-may-target-16341...

mushyhammer 4y ago
Considering the state of Italian railways IT, I wonder if the attack was accomplished by entering admin/admin in some login box.
- denysvitali 4y ago
  I think it's more likely that they didn't have any password at all :)

JaimeThompson 4y ago

Maybe they outsourced running the rail system to the people that run the Texas power grid?

stuckindoors 4y ago

There was a massive southwest airlines IT failure compounded by weather issues over the weekend. This cancelled numerous flights but also snowballed into most of the southwest flights being delayed. We were lucky to return home. The news is only reporting weather delays yet for those at the airport this weekend...it was clearly larger. If it is true that Netherland's and Italy's transport infrastructure were also down I wonder if Russian supported hackers are to blame.

wsc981 4y ago
> If it is true that Netherland's and Italy's transport infrastructure were also down I wonder if Russian supported hackers are to blame. It's been Russian hackers so many times last few years (Hunter Biden's laptop, Trump elections, etc...), it's a sure bet this time it's the same. /s Nobody ever got fired for blaming Russians.
- BlueTemplar 4y ago
  Their bad reputation is their own damn fault. See also : assassinations - maybe the death of the top polish government & military and Total's CEO in plane crashes were just freak accidents, but a nagging suspicion that Russia was involved will always remain...
selimthegrim 4y ago
Southwest has put something out about it so it's in the press/Twitter for sure.

rossdavidh 4y ago

As a developer, any day in which you don't see a national headline about how your f*#(up has just made a lot of other people sad and/or angry, is a day that could have been worse.

323 4y ago

In 2019 there was a power blackout in London. When power was restored, a number of new trains refused to start without a technician physically coming over with a diagnostics laptop to them because power fluctuated outside allowed bounds or something before the blackout.

gridspy 4y ago
It's likely the same power fault could occur in a single train in other circumstances. When an electric train cannot access the grid reliably it would become far more dangerous - no more regenerating brakes, no reliable acceleration ... Better to force the operators to fix the boom arm / whatever other cause of power glitches with fault codes like this. Even if they false-trip when there are grid level outages.
- Symbiote 4y ago
  No, it was considered a design fault with these particular trains, and fixed soon afterwards. Being unable to accelerate or unable to use regenerative braking is not in any way a safety problem. Either situation can happen at any time. (Electric trains have banks of resistors used to dump the unwanted kinetic energy when it's not possible to return it to the power supply, and this was the normal method for decades before regenerative braking became common.)
  - iggldiggl 4y ago
    > Electric trains have banks of resistors used to dump the unwanted kinetic energy when it's not possible to return it to the power supply On AC-electrified railways (especially those with their own independent power supply network like Germany/Austria/Switzerland), resistors are actually rather uncommon on stock capable of regenerative braking, because the OHLE not being able to accept regenerated power is a relatively rare occurrence. Though of course in that case the conventional friction brakes are of course fully rated to bring the train to halt, only with a bit more wear and tear. DC-electrified railways are a different matter – transmission losses are higher, power supply sections are smaller and substations often not capable of returning any surplus power back into higher network layers of the power supply systems, so the OHLE/third rail not being able to accept the power generated by regenerative braking happens much more frequently and so trains continue carrying braking resistors as a fallback system.

mostertoaster 4y ago

Probably Putins fault. Conspiracy theories are so fun.

rdxm 4y ago

ncmncm 4y ago

It wasn't the IT failure that kept the trains from running. It was management failure, not ensuring they could run without the IT system.

lucasyvas 4y ago
Now we're getting somewhere!
Tepix 4y ago
That sounds unrealistic given the size of the problem.
- Aeolun 4y ago
  They did it before the whole thing was automated. Surely they can do it again.
  - CyanBird 4y ago
    Sure, *but not on a single day without any warning what's ever*
    - Aeolun 4y ago
      I’m sure nobody raised this issue before. They just didn’t care “Don’t worry, the system will always work”. But here we are.
  - innocenat 4y ago
    The automation allowed running more trains than ever before using less personnel than ever before. You want the train company to keep 2x-3x personnel on hand just for days like this? Not to mentioned transition time.
    - ncmncm 4y ago
      So, run fewer trains. Fewer is better than none.

chevman 4y ago

It’s always DNS!

comprev 4y ago
Does Not Schedule? :-)
- chevman 4y ago
  Where is the DNS test environment?????????? :)

deleted 4y ago

[deleted]

supernova87a 4y ago

While having algorithms and computers dictating every aspect of train movements, crew scheduling, resource allocation is efficient and in a sense optimal, I think there's something compelling to be said about having a system that can survive with human-rememberable patterns when such an optimizer fails. That it can fall back on (or have at the core) simple patterns that people can still operate without algorithmic intervention. I.e. the algorithmic tuning enhances at the edges, and isn't at its core some tangled non-interpretable system that someone can't operate without computer assistance. The idea of a protected set of core routes and staffing patterns that operates without having to be computer-determined? I used to be so impressed at European train stations that they had a single sheet that would give the entire day's arrivals and departures, down to which track. I assume that behind that were schedules that weren't so far from being hand-calculated with equivalent sheets that told people and equipment where to be at what times. I think we lose something when we jump so far forward that you cannot fall back gracefully without a system completely breaking down.

pjmlp 4y ago
There is an Isac Asimov book about this. A crew is stranded in space, because of a computer malfunction and no one on board remembers how they used to be programmed.
- at-fates-hands 4y ago
  Is that "The Last Question"? https://en.wikipedia.org/wiki/The_Last_Question https://en.wikipedia.org/wiki/The_Last_Question
  - Tenoke 4y ago
    Are you posting it reflexively because 9 out of 10 times Asimov is mentioned that's the story people talk about? Because that's definitely not the plot of it.
    - andybak 4y ago

Agentlien 4y ago

This reminds me of the story one of my university teachers used to introduce himself. He was the most brilliant educator I've had, really passionate about making sure people could actually follow and so humble. The course was called "Digital and Computer Technology" and it was a first year course teaching you to build a CPU starting from logic gates and then program it with assembly. He said that when he had recently graduated and just moved from Norway to Sweden his first job here was as a technician for the local railway. One of his first assignments was a call where the computer controlling all the track switches had stopped working. Luckily there was a backup, but they needed him to fix it immediately. He arrived, took a look at the computer and its backup running next to it. He started out by measuring voltages on both, comparing to see what the difference was. After a while some men in suits came in and asked how things were going. He said it was all good, they said it wasn't because now all trains had stopped. Apparently, he had short circuited the backup. "And that's when I decided to go for a theoretical career!" My teacher happily concluded. The classroom was left in a stunned silence.

Saser 4y ago
Rolf Snedsböl? :-) Agreed that he was one of the best educators (if not _the_ best) I had
- Agentlien 4y ago
  Yes, Rolf Snedsböl! I absolutely loved his classes. I bumped into him a few years ago when I was holding a recruitment thing for Ghost Games and it happened to be just after his lecture. I told him how much I enjoyed his courses.
  - RonaldOlzheim 4y ago
    Did he wrote a book? Do you have a link?
- valvar 4y ago
  An absolute legend! My favourite of his jokes: "when I moved to Sweden from Norway, the average IQ went up in both countries".

reilly3000 4y ago

I think it’s worth mentioning that while the IT system couldn’t coordinate the trains safely, it was a management decision to stop them from running. Coordination can happen myriad ways. For example, trains could operate according to their normal schedule, and if one needed to deviate they could radio in, and an operator could let downstream trains know to proceed accordingly. Maybe simple cell-phone GPS pings could suffice. Perhaps the state of the system was in such chaos that such orchestration wasn’t possible with the staff on hand. That said, a truly resilient system should have multiple coordination channels to fall back upon.

mnumber 4y ago
Computers are great at ensuring your workforce cannot function without them. This is a small price to pay.
throwaway48375 4y ago
The cost of a single mishap is too high. It's just not worth it.
gpvos 4y ago
The drivers and conductors would like to end up at home at the end of the day. Even if you just run the normal schedule, there will always be disruptions, and there needs to be a system to adjust the planning so people and things end up at the right place at the end of the day. That was the system that failed. Have you any idea how frequent the trains in the Netherlands are and how complex the schedule? "Simply" running the existing schedule is an absolute impossibility. About the only thing they could maybe have done was to fall back on back-and-forth shuttle trains between major node stations. But that would only provide a very basic service, with incorrect train materiel being used for the normally expected number of passengers. It would be just enough to get passengers and personnel home. I would like to see something like a pen-and-paper schedule that they can fall back on, but they don't have it. (They did something like that at the first corona lockdowns, although with more planning involved.)

CaptainZapp 4y ago

Reminds me of when the entire Swiss train network shut down in 2005[0] This was, however, not due to a computer glitch, but problems in SBB's electrical grid, which lead to overload of some of the lines and then to a complete shutdown of the entire grid. Warnings were displayed on control consoles, but were drowned in 1000s of other meaningless warnings (there's a lesson for an UI designer here). I was in the ICE from Basel to Zurich at the time and after fuming for a few minutes I figured: "What can you do?" and moved to the bar wagon. There we had something of an impromptu party until there was no more power for the cash register and they stopped sales. In the end it was a (relatively) funny adventure for most and a huge embarrassment for SBB. I don't think they like to be reminded of that faithful day. [0] https://www.swissinfo.ch/eng/swiss-train-network-shuts-down/4578052 https://www.swissinfo.ch/eng/swiss-train-network-shuts-down/...

Scoundreller 4y ago
> There we had something of an impromptu party until there was no more power for the cash register and they stopped sales. Funny how the entire system has ground to a halt and the bartender is still worried that their cash register receipts and inventory won’t reconcile and shuts that part down too.
- Tabular-Iceberg 4y ago
  I’m puzzled by the lack of resilience in retail sales. I thought “always be closing” was supposed to apply to all types of sales. A receipt pad and a credit card imprinter under the counter would be a very cheap backup. It would be slow, but at least they would be closing. It can even be considered a national security issue in the case of food, medicines and other vital services.
  - Freak_NL 4y ago
    This is Europe. Credit card imprinters are rare if they were ever used at all. I've never seen one. The most resilient back up (especially in the face of national crises) is cash of course, but a growing group of people have stopped carrying any cash at all, and a lot of people in power are doing their best to make sure a cashless future is nearer each day. I don't mind paying with a debit card; I just want to have a backup in place in an increasingly interconnected world where a glitch can take out any nationwide system.

yummybear 4y ago

Failed OTA update?

j0e1 4y ago

What jumps out at me after reading this notice is how it completely lacks any apology. I am not prescribing what it should say but merely observing that having lived in North America for sometime now, it seems hard for me to imagine for such a notice to not sound profusely apologetic here. Note: I do not consider words like 'unfortunately' and 'extremely unpleasant' apologetic. Also, I understand there are cultural differences between the Netherlands and the USA- no joke.

mnumber 4y ago
"Due to the enormous impact of the failure in the IT system, it is unfortunately not possible to run any trains today." I think the writer is more pointing the finger at IT than feeling apologetic. Imagine getting called an enormous failure by your adjacent department, that writer wants you to know who screwed up instead of apologize for their screw up. Seems more useful than an apology in my opinion.
withinboredom 4y ago
Why would they be sorry? They’re the ones who are missing millions? of Euros in revenue due to this issue while the passengers just had to take a bus or bike somewhere. They probably would tell you that if you had to bike, it’s healthier that way. A little bit of rain won’t make you melt. /s
fwr 4y ago
This is very consistent with customer service in The Netherlands - nothing is ever the fault of service providers, and you manage to convince them otherwise - you'll get nothing more than "unfortunately (...) not possible"
- MonaroVXR 4y ago
  Customer service is done by another comapny. Source: It's me this time, NS customers service errrr....We help, it's called Webhelp.
coolgoose 4y ago
I don't mind the tone, I got tired of US style politically correctness just for the sake of saying it. How would "We are trully sorry for this incident" make it any better ? I do mind the fact that they don't pay the people their money back if they try to find alternatives to get to their destination. That's 100% unacceptable.

CannisterFlux 4y ago

Sounds similar to a problem that happened a couple of weeks ago in Madrid, although that was not the whole network it pretty much shut down the trains for the day. That was an IT failure too. https://www.20minutos.es/noticia/4973632/0/graves-retrasos-en-cercanias-por-una-averia-informatica-en-chamartin/ https://www.20minutos.es/noticia/4973632/0/graves-retrasos-e... (in Spanish)

tuukkah 4y ago
Wow, talk about a single point of failure when an issue in the systems of one station can stop all but one train line. Interestingly, it seems to be a station where tracks from six directions meet so a backup mode of trains simply passing through might be impossible: https://en.wikipedia.org/wiki/Madrid_Chamart%C3%ADn_railway_station https://en.wikipedia.org/wiki/Madrid_Chamart%C3%ADn_railway_...

comprev 4y ago

I was walking up to Amsterdam Centraal when I noticed floods of people all exiting the building... within minutes an Uber was booked to Schiphol before prices rocketed due to demand.

AndyMcConachie 4y ago

The fact that it happened on a Sunday suggests to me that this is the result of a service outage gone awry. Realize that the trains failing on a Sunday is really not a big deal. However, if the trains don't run for Monday morning rush hour that is a big deal. I bet they made a change Saturday night and something broke Sunday morning. It's now Monday morning here and the trains are back to normal in time for rush hour. So it's not really a big deal.

regularfry 4y ago
I have to say that I do find this framing which says "well, it didn't affect business, so no big deal" to be somewhat distasteful. It's completely dehumanising. Why run the trains at all at the weekend, if a weekend outage is no big deal?
- tsimionescu 4y ago
  That's not the framing. For the majority of travelers, missing a weekend trip is an annoyance. In contrast, missing their work is a major headache, for them personally. It also cascades pretty badly if it happens on a lathe scale - meetings get pushed and overlap others, suplliers are not able to deliver, stalling production, financial operations get delayed, potentially affecting deadlines - it's a huge mess in everyone's life that won't just go away when trains come back. It's also as true for non-profits, state organizations and schools as it is for businesses.

lifty 4y ago

Most likely their Kubernetes cluster went down. (not a joke)

HNHatesUsers 4y ago

lqet 4y ago

Seems like trains are up and running again today as announced, with only little delays: https://travic.app/?z=9&x=628581.0&y=6814491.0 https://travic.app/?z=9&x=628581.0&y=6814491.0

woudsma 4y ago
I didn’t know about Travic, thanks!

qwezxcrty 4y ago

Based in my knowledge as a public transportation enthusiastist in China, I got really surprised to see a IT problem bring the entire system down. There should usually be some "more manual" backup mechanism that allows downgrade and operation of trains without computerized signaling system. By kinds of token signaling it will still be possible to run trains as long as the physical infrastructure is still there. The efficiency will be much lower and more prone to (potentialy deadly) errors, but no more trains today sounds unlikely.

apexalpha 4y ago
The Dutch rail network is top 3 busiest in the world. Many lines have 10+ trains in each direction, every hour. The usage of certain lines at a certain time is calculated in periods of 6 seconds! We're waaay beyond normal human coordination capacity.

NWoodsman 4y ago

This post is a fascinating use of grammatic passive voice. The hallmark is gratuitous use of "being" verbs, i.e. "is,was". Passive voice is a way to avoid using active voice, which is an implicit way to avoid responsibility for an error. "Due to the enormous impact of the failure in the IT system, it is unfortunately not possible to run any trains today." // => "Due to the enormous impact of the failure in the IT system, NS cannot run trains today." "Although the cause of the failure has now been resolved, the impact is considerable. To be able to start up reliably, systems must be updated and trains must be brought to the right place. That takes time. For our passengers, this is extremely unpleasant news." // => "Although we have fixed the cause of the failure, the failure impacted most of our customers. To fix this failure so that trains would start reliably, we needed to return all trains to a central depot, which took nearly 12 hours. We apologize for the extremely unpleasant interruption." "The expectation is that tomorrow morning the normal timetable can largely be resumed. The night trains can still run." // => We expect to resume minor operation with night trains first and then major operations tomorrow morning. "The IT failure occurred at the end of the morning. It affected the system that generates up-to-date schedules for trains and staff. This system is important for safe and scheduled operations: if there is an incident somewhere, the system adjusts itself accordingly. This was not possible due to the failure." // => The IT failure occured at 11am in the scheduling system, a major requirement for safe operation; if train operations are interrupted at one location, the system normally reroutes other trains to prevent collisions. This system failed. "The international trains are not affected by this failure. For information about the timetables of other transporters, passengers can consult the websites of these transporters." // => The failure did not affect international trains. If you have an inquiry about other transportation services, please consult those services. "The journey planner is updated." // => We updated the journey planner accordingly to account for the interruptions. // Generally the use of "passive voice" indicates poorly-taught grammar, however the use could also indicate desire to avoid responsibility.

NWoodsman 4y ago
The "about us" section of the webpage : "Who are we? NS is active in the public transportation sector. We encourage the use of public transportation and keep the Netherlands moving. Our travellers are our 1st, 2nd and 3rd priority in all of our activities, and we do our utmost to make their trip as pleasant and sustainable as possible from door to door." This is also fascinating. I would expect to find "NS (Netherlands Service) provides rail services for the entire country". Instead they choose vague grammar, more the idea of NS rather than what NS actually is. "I am a programmer" => "I encourage the use of computers to provide solutions for common business problems". Just a strange use of grammar.