7 ms·

Hardening the registers: A cascading failure of edge induced fault tolerance

• ncmncm 4y ago
  Wasn't Target where hackers were running loose in their POS ("point of sale", not the other meaning) system for months or years? Was that before or after this incident?
• PointyFluff 4y ago
• InCityDreams 4y ago
  I presume "guest" means "customer"?
  • bluedino 4y ago
    Some 90's thing that a couple retail stores started doing. Must have been popularized by whatever executives took advice from before Gary Vee and Seth Godin.
  • rjbwork 4y ago
    Yes. Same way they call their employees "associates". I don't quite understand the rationale, but if I had to guess, "customer" and "employee" are a bit too on the nose, and they wish to cultivate a more human-feeling relationship between the customers, employees, and corporation in the minds of the former two groups.
    • Avshalom 4y ago
      Walmart has associates, Target has team members.
      • Avshalom 4y ago
        Some real fun horseshit is that the stockers are "designated business owners" and cashier's are "guest advocates"
        • spelunker 4y ago
          I'll have you know, back at my high school job at Subway, employees were known as sandwich artists. So fancy!
    • numbsafari 4y ago
      Not just in the minds of the former two groups, but in the minds of their staff and leadership as well. If you refer to your team members or employees as "associates" you're much more likely to treat them as equals. Similarly, if you refer to your customers as "guests", you are much more likely to treat them as such rather than simply treating them as people in your store looking to spend money. It gets to the whole sense of trying to create an experience. As a store that sells a significant amount of home goods and goods for the home, referring to customers as guests instills the sense that employees are creating a home like experience for the customer. Neurolinguistic programming isn't just for hippies. It's a very popular pseudoscience in corporate America.
      • sokoloff 4y ago
        It is for this reason that I doggedly push back on the use of "resources" when talking specifically about people; I semi-frequently correct this mis-use (IMO) of language. If you ask "do we have enough resources to compete in segment X?" and you mean resources of all types [including people], that's fine. If you ask "could I have two additional resources on this project" and you mean exactly people, I'll speak up every time.
      • drjasonharrison 4y ago
        Not all associates are at the same level. Some people unfamiliar with this American Business Vocabulary might jump to conclusions. Some associates are the customers of the systems that you are responsible for and you are the customer for services other associates maintain. Unfortunately rather than talk about the importance of respect and what happens when respect between members of groups within the organization is violated, these sorts of neurolinguistic fashions are used.
    • formerkrogemp 4y ago
      Ugh, some retailers still call their employees "partners." Kroger would write that every check was 'brought to you by customers' on every paper and digital pay stub. The rosy language is always used to obfuscate the exploitation going on. It's fascinating to see Target slightly improve security over the years after multiple hacks and problems with register security.
    • SilasX 4y ago
      I'm not necessarily against that in general, but if it's a technical article for a technical audience, which this looks to be intended as, they really need to drop the marketing jargon.
      • jaywalk 4y ago
        Anything published by a corporation of Target's size is, at some level, marketing.
    • marcosdumay 4y ago
      I always assumed that "associates" was created to encode the idea that people's salary was mostly commission based. But with you talking about those giant corporations that call everybody by that name, this is either anachronistic or plain wrong.
  • deleted 4y ago
    [deleted]
  • EvanAnderson 4y ago
    My wife worked for Darden Restarurants for awhile and corporate training materials always referred to customers as "guests", too. On one level I suppose it's just silly terminology, but it grates with me. I guess it's supposed to imply some kind of familiar relationship, free of the gauche trappings of economics. To me a customer demands more attention than a "guest". It shocks me how many people don't recognize that their employer wouldn't exist if not for customers. That should be front-and-center in the minds of anyone working for a for-profit entity. I don't think there's anything gauche about economics.
    • barbecue_sauce 4y ago
      The history of food service goes hand-in-hand with the hospitality industry, so referring to a customer as a "guest" is very traditional and common amongst almost all restaurants.
• joedissmeyer 4y ago
  Glad to see that a big US retailer like Target is using the same types of "de-facto" observability tools that I've been using for a while at all of my various employers over the last 5+ years - which are Grafana, Prometheus and Elastic Stack (specifically the Kibana UI for the logging analysis screenshot).
  • ngc248 4y ago
    >>> Grafana, Prometheus and Elastic Stack Those 3 have almost become the industry standard for observability. Everywhere I have worked have used the same and it is almost a no-brainer.
• haroldl 4y ago
  This was really interesting both in exploring the architecture of a retail system and looking at how systems fail. Better to read about it and learn than to live it. I'd call it a 4 hour outage because the initial "recovery" was a result of cashiers manually typing in prices for items. Then when load decreased and they discovered that scanning items worked again the problem came right back. Maybe returning 404 for both a cache miss and a "there's no endpoint at this path" error is an issue too. For other status codes there's a distinction between temporary and permanent failure; e.g. 301 versus 302. It would've been good to use HTTP 400 Bad Request for the misconfigured URL and 404 for a cache miss. In the 10% of stores with the early roll out of the config change the cache hit rate went to 0 right away, and that started 12 days before the outage. Alerts on cache hit rates and per-store alerts would've caught that. Then there were 4 days where traffic to the main inventory micro-service in the data center jumped 3x which took it to what appears to be 80% of capacity. Load testing to know your capacity limits and alerts when you near that limit would've called out the danger. Then during the outage when services slowed down due to too many requests they were taken out of rotation for failing health checks. Applying back pressure/load shedding could have kept those servers in active use so that the system could keep up.
  • jabart 4y ago
    204 no content is an underused http status. 404 should be monitored as an error, 204 as, well no content available. If a status code has two responsibilities that will be a monitoring issue waiting to happen.
    • magicalhippo 4y ago
      So your suggestion is that if I have a /invoice endpoint, a "GET /invoice/abc123" should return 204 if it's an invalid/non-existing invoice number? Seems reasonable.
  • bombcar 4y ago
    It seems absolutely insane to me that a system was designed and developed that allows taking down all registers in the country at once. I would have thought it would be designed to be much more "batch" oriented and the worst that could happen is you lose price updates and sales info unto the batches can get through again.
• at_a_remove 4y ago
  I don't work at that level, or even want to, but I did detect a dark pattern that I often complain about, but have never managed to get people to pay attention to: do not collect data unless you have attached to it a decision with two or more distinct outcomes based on that data.
  • tylerrobinson 4y ago
    Can you give an example of what you mean?
    • at_a_remove 4y ago
      So when I ran the university website, the homepage naturally had links to other sites. One guy had this inflated sense of importance. If there weren't a lot of clicks over to his site, we should MAKE THE LINK BIGGER because people weren't seeing it. If clicks to his site went up, we should MAKE THE LINK BIGGER because it is that important. His flowchart had only one distinct outcome: MAKE THE LINK BIGGER. All of the effort that went into collecting the information was for nought, because the outcome was always the same. That was collection with a flowchart, but without two or more distinct outcomes. A second example would be search engine logs. Nobody wanted to make decisions on them, but "we could always trawl them for data later." A decade on, this had never occurred. That was collection with no flow chart. Offloading the logs, parsing them out, making the data available, week after week, month after month, year after year. Wasted effort. So part of it is "don't waste effort," but the other part is, if there is decent information to collect, you should be doing something with it.
      • drjasonharrison 4y ago
        Thank you.
  • drjasonharrison 4y ago
    Do you mean "don't collect metrics unless you monitor them and have alerts" or "don't collect data on products, customers, sales..."
• yardstick 4y ago
  > Although Autobahn contained all the item data, the 404 responses were interpreted by the SDM Proxy as an indicator that the item was missing in Autobahn and the SDM Proxy retried the request to the central ILS API in the data centers. This is why I never design web APIs to use the HTTP status code to indicate the application response. Always embed the application response within the HTTP payload. It should be independent of the transport mechanism. I’m ok with it not being a proper REST/RESTful service. { “status” : 1000, “message” : “Item not found” } And intentionally don’t use the same status numbers as http (Ie don’t use 404 as not found, because someone will mix them up!)
  • thedougd 4y ago
    You do have to consider middlebox and client caching when you do this. Returning a 200 with a 'not found' would be cached, and that may or may not be desired for the use case.
    • yardstick 4y ago
      That’s I guess an issue for cache header instructions to solve. With 200 codes being cached you still will have problems like stale data. Wouldn’t want the Target registers using yesterdays prices for today (especially if yesterday was a super sale day like Black Friday etc).
  • nerdponx 4y ago
    Sounds more like the server was misusing the 404 status and/or the clients were mishandling it. I am inclined to agree that for this particular usage, an "in-body" response makes sense. 404 should be reserved for when the actual HTTP endpoint is unavailable. But in REST semantics, you would only return 404 for an endpoint like /users/12345 when user 12345 doesn't exist. So the two usages line up. Returning 200 with a body that says "user 12345 does not exist" makes a lot less sense to me. A good example of overdoing it is when GraphQL servers return a 200 HTTP response that contains nothing but an error message, instead of returning a suitable HTTP status like 400.
    • 4y ago
• aftbit 4y ago
  Why don't the ILS services have their own cache in front of them? Supporting a per-store cache already requires good discipline on timeouts and invalidation, so adding an additional caching layer in the datacenter between the inbound requests and ILS itself seems like it would provide for a cheap extra layer of scalability in case the per-store caches become unavailable.
• jkaptur 4y ago
  It's very interesting that by building a system that's more resilient and reliable: > high profile processes (such as POS) implement their own fallback processes to handle the possibility of issues with the SDM system in store. In the case of item data, the POS software on each register is capable of bypassing the SDM Proxy and retrying its request directly to the ILS API in the data centers. ... the system as a whole became much more complex and difficult to observe. The system was running in a degraded, abnormal, less-tested, fallback mode for days without anyone caring. This is also a point about the normalization of deviance. When there is a background rate of the POS using the fallback path, who is to say how important an increase in that rate might be?
  • reaperducer 4y ago
    I find all of this fascinating. A few years ago, the guys who built Chick-fil-a's POS fog were on HN talking about their fault-tolerance and transaction queueing. It was quite interesting. There's a lot that you can learn from high-volume POS system design that applies to just bog-standard every day programming.
  • bombcar 4y ago
    Buried under another thread was this post: https://aws.amazon.com/builders-library/avoiding-fallback-in-distributed-systems/ https://aws.amazon.com/builders-library/avoiding-fallback-in... which is the exact same issue - a cache miss was backed up by a direct query and it took down all of Amazon trying to display shipping times. grok say complexity bad Fallback is not always necessary (sometimes it is, you can't just say "whelp the engines on this plane went out, time to die") but when you have a fallback system you should think about why you have it and how bad it is to fail, and if it could be worse to succeed.
• londons_explore 4y ago
  I'd like to see staff training for major outages in retail like this. For example, if the shop loses power, do they have the ability to sell goods still? One approach is to let staff members estimate the value of goods - for example at the register, the staff member looks at the cart contents, estimates that it's about $120 worth of goods, charges the customer $120, and hand writes a receipt saying "$120 of goods sold, Date, store name, signature". The staff member then uses a phone to photograph the cart and the receipt. At the end of the shift, the shaff member drops all the photos into a big store wide Dropbox account, that the accounts department can use to pay taxes. You'd probably want to practice this process ahead of time with every staff member. I imagine it might actually be a good process to use on very busy days too - it is probably quicker than scanning every item at the register.
  • Avshalom 4y ago
    >>For example, if the shop loses power, do they have the ability to sell goods still To some degree yes, we can check people out with a handheld (which has swappable batteries) and the self check registers are on the emergency power circuit. Couple years ago when the system went down nationwide we just told people to put their name on their cart and we gave them 10% off if the came back the next day.
• prithvi24 4y ago
  Target has 250k SKUs total - why is their inventory system so complicated? Why the hybrid on-prem store + data center cloud model - isn’t it easier if there is one source of truth? Seems like it would reduce the need for even dealing with all this eventually consistence cache sycning and whatnot I ofc don’t know what I dont know, but super curious if anyone has insight into why such a complex system is required Also, if this microservice is used for brick and mortgage mortar, can’t imagine more than a couple hundred per second? ( 2000 stores, 5 registers a store - and humans manually scanning items ) - why did that overload the micro service (guessing it wasn’t an endless exponential backoff)
  • lalaland1125 4y ago
    > I ofc don’t know what I dont know, but super curious if anyone has insight into why such a complex system is required Because it's much more efficient, which allows them to use simpler tech that doesn't need to scale as well. You are also underestimated the throughput the system needs to handle. 2000 stores * 10 registers per store * 1000 scans per register per hour = 5000 scans per second.
    • prithvi24 4y ago
      I’m not sure the throughput is that high - scans take quiet a bit of time, I would doubt that a register scans an item every 3.6 seconds - don’t have data on this but would easily triple that estimate as an average (so in the hundreds) Also , I get the simpler tech, but complexity breeds failure - if you have a hybrid on prem / cloud model, especially with only 250k skus, at that point doesn’t it make sense to keep that exclusively in the cloud. It’s a system that scans a barcode and returns an item at its core - this is still well under the limits of using an off the shelf system like Redis behind an endpoint
      • EricE 4y ago
        "I would doubt that a register scans an item every 3.6 seconds" Indeed, that sounds WAY too slow for me - traffic like this is bursty. Ever try to scan five of the same thing at some self check out registers? On some it's instantaneous (an awesome customer experience) on others there are one second or more delays (horrible customer experience). Latency = friction and friction is the ultimate deal killer.
• sydthrowaway 4y ago
  Surprised they’re still not on IBM Mainframes
• EricE 4y ago
  "It’s not enough to implement redundant systems and failovers, we must monitor and alert when those systems are being exercised." My air conditioner in my house has a secondary drain pan under it. The outlet for that drain pan is right above a main window outside. If the primary condensate drain gets plugged/fails and the water overflows into the backup pan there would be a stream of water in front of a window that shouldn't otherwise be there. They want you to be able to readily notice it as you are now at risk for significant water damage if that secondary drain manages to plug up too. Always something worth considering when designing any system - how to make it fail in a way that is noticeable!