8 ms·
Building and operating a pretty big storage system called S3
- Twirrim 3y ago> That’s a bit error rate of 1 in 10^15 requests. In the real world, we see that blade of grass get missed pretty frequently – and it’s actually something we need to account for in S3. One of the things I remember from my time at AWS was conversations about how 1 in a billion events end up being a daily occurrence when you're operating at S3 scale. Things that you'd normally mark off as so wildly improbable it's not worth worrying about, have to be considered, and handled. Glad to read about ShardStore, and especially the formal verification, property based testing etc. The previous generation of services were notoriously buggy, a very good example of the usual perils of organic growth (but at least really well designed such that they'd fail "safe", ensuring no data loss, something S3 engineers obsessed about).
- ignoramous 3y agoJames Hamilton, AWS' chief architect, wrote about this phenomena in 2017: At scale, rare events aren't rare; https://news.ycombinator.com/item?id=14038044 https://news.ycombinator.com/item?id=14038044
- jdwithit 3y agoJames' posts are always a treat. It's so rare to encounter such plain, straightforward content from someone with a title and responsibilities like his. Without layers of marketing sugar over everything. Dude just wants to post about the cool shit he did on his GeoCities-tier website and I love it.
- aborsy 3y agoThis phenomenon is just multiplication of the sample size (scale) times a probability (rare).
- maweki 3y agoIt shows that, however improbable, people do win the lottery. It's good to be reminded of that, if you've been trained for years, not to play the lottery because you personally won't ever win. In this case, the Cloud vendor is the lottery organizer and they indeed need to plan for people winning.
- da39a3ee 3y agoI agree with what I think is your sentiment -- that people seem to be treating this as if it's some sort of profound insight that you only get if you work at a very senior level in engineering for major US cloud providers, when it's in fact blindingly obvious!
- Spooky23 3y agoWell it is - nobody maintains the level of detail required to actually know about these sorts of events. I worked on a safety critical system where we’d find all sorts of unusual bugs… because we were looking for them. It really narrowed the scope for product selection, many vendors were just disqualified.
- Twirrim 3y ago> that people seem to be treating this as if it's some sort of profound insight that you only get if you work at a very senior level in engineering for major US cloud providers, when it's in fact blindingly obvious! I don't mean to imply it's a profound insight, and the discussions I had in AWS were never in those terms. It's just that when you're designing and building things that are going to operate at that scale, you have to very seriously consider the improbable. What's more difficult is actually knowing what needs to be considered. e.g. prior to working at AWS, I don't think I'd have even considered "NIC corrupts packet, in such a way it gets to the OS mangled" as something that would be worth handling. Yet S3 and similar scale services see that and other improbable events so regularly that they actually have to consciously design for it, everywhere. It's also one reason why larger services end up being incredibly conservative about the use of technology. You know what the failure modes are, however improbable, and can account for them. New technology tends to be kept on the fringes, and only adopted in more significant places once proven and improbable failures become understood.
- mjb 3y ago> daily occurrence when you're operating at S3 scale Yeah! With S3 averaging over 100M requests per second, 1 in a billion happens every ten seconds. And it's not just S3. For example, for Prime Day 2022, DynamoDB peaked at over 105M requests per second (just for the Amazon workload): https://aws.amazon.com/blogs/aws/amazon-prime-day-2022-aws-for-the-win/ https://aws.amazon.com/blogs/aws/amazon-prime-day-2022-aws-f... In the post, Andy also talks about Lightweight Formal Methods and the team's adoption of Rust. When even extremely low probability events are common, we need to invest in multiple layers of tooling and process around correctness.
- rubiquity 3y agoDaily? A component I worked on that supported S3’s Index could hit a 1 in a billion issue multiple times a minute. Thankfully we had good algorithms and hardware that is a lot more reliable these days!
- Waterluvian 3y agoEver see a UUID collision?
- mabbo 3y agoEven at a billion requests per second, 128 bit UUIDs shouldn't collide for something like a billion years. And that's if you're going completely random and not taking care to try to reduce collisions.
- Dylan16807 3y agoAre you sure about that math? A billion seconds at a billion requests per second is already 2^60 items. You'd only need a few billion seconds to have a 50:50 collision chance with 128 random bits, and even less with a real UUID that only has 122 random bits. You'd hit 1% odds of collision after less than a decade. If you actually want to go for a billion years, you need to expand that UUID by 50%.
- danielmarkbruce 3y agoThis seems off. A few billion seconds to have a 50:50 chance? Why wouldn't it be a billion seconds at a billion per second (2^60 total requests) would give a 1 in 2^68 chance (or 1 in 2^62 if its really only 122 bits)?
- Dylan16807 3y agoBirthday paradox. The number of opportunities to collide is the number of items squared. (Divided by two and a smidge)
- danielmarkbruce 3y agoLol. I must be brain dead. Yes.
- penteract 3y ago
- ilyt 3y agoI think Ceph hit similar problems and they had to add more robust checksumming to the system, as relying on just tcp checksums for integrity for example was no longer enough
- benou 3y agoNot that surprising, given this was already extensively documented in the 2000's (so already widely known by then) with iSCSI and such, see https://www.rfc-editor.org/rfc/rfc3385 https://www.rfc-editor.org/rfc/rfc3385 for example.
- Twirrim 3y agoYes, I remember tcp checksumming coming up as not sufficient at one stage. Even saw S3 deal with a real head-scratcher of a non-impacting event that came down to a single NIC in a single machine corrupting the tcp checksum under very specific circumstances.
- jamesblonde 3y agoHDFS never relied on only network checksums. Blocks should be checksummed and validated at clients - a reliable end-to-end guarantee.
- Twirrim 3y agoWell... yeah. S3 has checksums and all sorts of fixity checks right throughout. At no stage do they ever rely on a single mechanism. If there's one thing they're insanely paranoid about, it's data correctness and durability. It has been several years, so I really don't remember much about the tcp checksum / corrupting NIC thing. Typically tcp checksum failures are handled entirely by the NIC, you wouldn't even notice it. My vague recollection was it coming up between two services not in the customer synchronous path (so e.g. not involved in getting data to or from the customer), and it caused something on the OS side. I do remember that there was a contingent of engineers that were convinced it was a cosmic ray bit flip, which seems this whole thing certain types of engineers end up doing when presented with improbable seeming circumstances. It wasn't until it had happened a second or third time (weeks later) that they realised the origin machine was the same each time, and were able to dig in deeper to the point of reproduction.
- ldjkfkdsjnv 3y agoAlso worked at Amazon, saw some issues with major well known open source libraries that broke in places nobody would ever expect.
- wrboyce 3y agoAny examples you can share?
- deleted 3y ago[deleted]
- ldjkfkdsjnv 3y agoApache tomcat starts to break down
- thewakalix 3y agoCould you elaborate?
- ruckfool 3y agoRedis Node failover
- baz00 3y agoWe get this on a much lower scale. We have to maintain many forks because no one is responsive on taking patches.
- rkagerer 3y agoPersonally I'd love working in that kind of environment. That one in a billion hole still itches at me. There's also a slightly-perverse little voice in my head ready with popcorn in case I'm lucky enough to watch the ensuing fallout from the first major crypto hash collision :-).
- fooker 3y agoThat probability is significantly lower than one in a billion. One in a billion would be if keys were ~30 bits. Luckily it isn't.
- PaulRobinson 3y agoWas an SDM of a team of brand new SDEs standing up a new service. In a code review, pointed to an issue that could cause a Sev2, and the SDE pushed back "that's like one in a million chance, at most". Pointed out once we were dialled up to 500k TPS (which is where we needed to be at), that was 30 times a minute... "You want to be on call that week?". Insist on Highest Standards takes on a different meaning in that stack compared to most orgs.
- jacobgorm 3y agoTo think that when Andy’s Coho Data built their first prototype on top of my abandoned Lithium [1] code base from VMware, the first thing they did was remove “all the crazy checksumming code” to not slow things down… [1] https://dl.acm.org/doi/10.1145/1807128.1807134 https://dl.acm.org/doi/10.1145/1807128.1807134
- jakupovic 3y agoThe part about distributing loads takes me back to S3 KeyMap days and me trying to migrate to it, from initial implementation. What I learned is that even after you identify the hottest objects/partitions/buckets you cannot simply move them and be done. Everything had to be sorted. The actual solution was to sort and then divide the host's partition load into quartiles and move the second quartile partitions onto the least loaded hosts. If one tried to move the hottest buckets, 1st quartile, it would put even more load on the remaining members which would fail, over and over again. Another side effect was that the error rate went from steady ~1% to days without any errors. Consequently we updated the alerts to be much stricter. This was around 2009 or so. Also came from academic background, UM, but instead of getting my PhD I joined S3. It even rhymes :).
- deleted 3y ago[deleted]
- epistasis 3y agoWorking in genomics, I've dealt with lots of petabyte data stores over the past decade. Having used AWS S3, GCP GCS, and a raft of storage systems for collocated hardware (Ceph, Gluster, and an HP system whose name I have blocked from my memory), I have no small amount of appreciation for the effort that goes into operating these sorts of systems. And the benefits of sharing disk IOPs with untold numbers of other customers is hard to understate. I hadn't heard the term "heat" as it's used in the article but it's incredibly hard to mitigate on single system. For our co-located hardware clusters, we would have to customize the batch systems to treat IO as an allocatable resource the same as RAM or CPU in order to manage it correctly across large jobs. S3 and GCP are super expensive, but the performance can be worth it. This sort of article is some of the best of HN, IMHO.
- kuchenbecker 3y agoAs someone in this area: we very much want to make your EiB of data to feel local. It's hard and I'm sorry we only have 3.5 9's of read availability.
- epistasis 3y agoPeople working on storage systems are doing amazing things. When I first heard about Ceph more than a decade ago, I immediately emailed one of the founders asking for an exabyte data store, because I knew just how amazingly difficult it would be and that it was very much needed. 3.5 9s is incredible on large stores. S3 and GCS are just amazing machines. I have nothing but admiration for the people that make this happen.
- dekhn 3y agoUnfortunately many tools in genomics (and biotech in general) still depend on local filesystems- and even if they do support S3, performance is far slower than it could be.
- seized 3y agoThe latency is higher so the key is parallelism... Which means you need more cores/hardware/VMs/pick your poison. New but same problem...
- deathanatos 3y ago> Now, let’s go back to that first hard drive, the IBM RAMAC from 1956. Here are some specs on that thing: > Storage Capacity: 3.75 MB > Cost: ~$9,200/terabyte Those specs can't possibly be correct. If you multiply the cost by the storage, the cost of the drive works out to 3¢. This site[1] states, > It stored about 2,000 bits of data per square inch and had a purchase price of about $10,000 per megabyte So perhaps the specs should read $9,200 / megabyte? (Which would put the drive's cost at $34,500, which seems more plausible.) [1]: https://www.historyofinformation.com/detail.php?entryid=952 https://www.historyofinformation.com/detail.php?entryid=952
- acdha 3y agohttps://en.m.wikipedia.org/wiki/IBM_305_RAMAC https://en.m.wikipedia.org/wiki/IBM_305_RAMAC has the likely source of the error: 30M bits (using the 6 data bits but not parity), but it rented for $3k per month so you didn’t have a set cost the same as buying a physical drive outright - very close to S3’s model, though.
- nijave 3y agoI think this is still IBMs license model (at least a few years ago). It was explained to me you basically license a certain amount of compute even though the hardware is in your data center and you pay overages if you exceed your licensed throughput. Since you license a fixed amount, there were projects at the company looking at running batch/non time sensitive jobs on the mainframe since it was effectively free off peak (I guess power cost was trivially compared to licensing).
- mkesper 3y agoYou had online jobs during the day and batch at night then. That's why you always had to have one night between. Obviously doesn't work when load is 24/7.
- andywarfield 3y agooh shoot. good catch, thanks!
- apitman 3y agoThe things we could build if S3 specified a simple OAuth2-based protocol for delegating read/write access. The world needs an HTTP-based protocol for apps to access data on the user's behalf. Google Drive is the closest to this but it only has a single provider and other issues[0]. I'm sad remoteStorage never caught on. I really hope Solid does well but it feels too complex to me. My own take on the problem is https://gemdrive.io/ https://gemdrive.io/, but it's mostly on hold while I'm focused on other parts of the self-hosting stack. [0]: https://gdrivemusic.com/help https://gdrivemusic.com/help
- Spivak 3y agoSuch a system would be amazing. It would really force companies whose products are UIs on top of S3 to compete hard because adversarial interoperability would be an ever present threat from your competitors. It really is such a shame that all the projects that tried/are trying to create data sovereignty for users became weird crypto.
- ttymck 3y agoI agree with both halves of your comment, but I realized I can't identify the connection between S3 oauth and data sovereignty. Could you elaborate?
- Spivak 3y agoSo the idea would be that you have an account with AWS (or realistically a more consumer friendly service that's Amazon branded) where all your data lives. Then when you use say Dropbox you can pick "Use my own storage" and grant Dropbox via OAuth the ability to write to /dropbox in your bucket and all your files would live there instead of Dropbox's servers. Lots of the data sovereignty solutions also include a database like interface you can grant apps the ability to use but I can't imagine that catching on initially. Apple actually already does this with iCloud storage but hides it really well so it feels seamless.
- BHSPitMonkey 3y ago
- baq 3y ago> What’s interesting here, when you look at the highest-level block diagram of S3’s technical design, is the fact that AWS tends to ship its org chart. This is a phrase that’s often used in a pretty disparaging way, but in this case it’s absolutely fascinating. I’d go even further: at this scale, it is essential and required to develop these kind of projects with any sort of velocity. Large organizations ship their communication structure by design. The alternative is engineering anarchy.
- hobo_in_library 3y agoThis is also why reorgs tend to be pretty common at large tech orgs. They know they'll almost inevitably ship their org chart. And they'll encounter tons of process-based friction if they don't. The solution: Change your org chart to match what you want to ship
- mr_toad 3y agoA more cynical take is that it makes it look like the new management is doing something. An even more cynical take is that it makes it difficult to compare performance with past performance.
- Severian 3y agoStraight from The Mythical Man Month: Organizations which design systems are constrained to produce systems which are copies of the communication structures of these organizations.
- deleted 3y ago[deleted]
- _aaed 3y agosomething something Conway's law
- CobrastanJorji 3y agoI'll take the metaphor one step further. The architecture will, over time, inevitably change to resemble its org chart, therefore it is the job of a sufficiently senior technical lead to organize the teams in such a way that the correct architecture emerges.
- jl6 3y agoGreat to see Amazon employees being allowed to talk openly about how S3 works behind the scenes. I would love to hear more about how Glacier works. As far as I know, they have never revealed what the underlying storage medium is, leading to a lot of wild speculation (tape? offline HDDs? custom HDDs?).
- deleted 3y ago[deleted]
- inopinatus 3y agoNever officially stated, but frequent leaks from insiders confirm that Glacier is based on Very Large Arrays of Wax Phonograph Records (VLAWPR) technology.
- Twirrim 3y agoWe came up with that idea in Glacier during the run up to April one year (2014, I think?), half jokingly suggested it as an April Fool's Day Joke, but Amazon quite reasonably decided against doing such jokes. One of the tag line ideas we had was "8 out of 10 customers say they prefer the feel of their data after it is restored"
- inopinatus 3y agoThe real problem is the lack of Star Wars references.
- jdwithit 3y agoThis would have been incredible. But I guess I get the angle of not wanting to risk pissing off the audiophile CTO paying you 10 figures per month. Cause he can TOTALLY hear the difference listening to Dark Side of the Moon on vinyl via Monster Cables.
- Twirrim 3y agoGlacier is a big "keep your lips sealed" one. I'd love AWS to talk about everything there, and the entire journey it was on because it is truly fascinating.
- dsalzman 3y ago> Imagine a hard drive head as a 747 flying over a grassy field at 75 miles per hour. The air gap between the bottom of the plane and the top of the grass is two sheets of paper. Now, if we measure bits on the disk as blades of grass, the track width would be 4.6 blades of grass wide and the bit length would be one blade of grass. As the plane flew over the grass it would count blades of grass and only miss one blade for every 25 thousand times the plane circled the Earth.
- Sai_ 3y agoThe standing joke is that Americans love strange units of measure but this is one is so outre that it deserves an award.
- mcapodici 3y agoS3 is more than storage. It is a standard. I like how you can get S3 compatible (usually with some small caveats) storage from a few places. I am not sure how open the standards is, and if you have to pay Amazon to say you are "S3 compatible" but it is pretty cool. Examples: iDrive has E2, Digital Ocean has Object Storage, Cloudflare has R2, Vultr has Object Storage, Backblaze has B2
- CobrastanJorji 3y agoGoogle's GCS as well, and I haven't used Microsoft, but it'd be weird if they didn't also have an "S3 compatible" option. Edit: I looked it up and apparently no, Azure does not have one :-/
- nijave 3y agoI think there's a good call-out about ownership here. Ownership and autonomy go hand in hand (you can't force someone to own something)
- Narciss 3y ago"As a really senior engineer in the company, of course I have strong opinions and I absolutely have a technical agenda. But If I interact with engineers by just trying to dispense ideas, it’s really hard for any of us to be successful. It’s a lot harder to get invested in an idea that you don’t own. So, when I work with teams, I’ve kind of taken the strategy that my best ideas are the ones that other people have instead of me. I consciously spend a lot more time trying to develop problems, and to do a really good job of articulating them, rather than trying to pitch solutions. There are often multiple ways to solve a problem, and picking the right one is letting someone own the solution." "I learned that to really be successful in my own role, I needed to focus on articulating the problems and not the solutions, and to find ways to support strong engineering teams in really owning those solutions." I love this. Reminds me of the Ikea effect to an extent. Based on this, to get someone to be enthusiastic about what they do, you have to encourage ownership. And a great way is to have it be 'their idea'.
- forrestthewoods 3y agoThat section really stood out to be as well. If Andy Warfield is reading, and I bet he is, I have a question. When developing a problem how valuable is it to sketch possible solutions? If you articulate the problem that probably springs to mind a few possible solutions. Is it worth sharing those possible solutions to help kickstart the gears for potential owners? Or is it better to focus only on the problem and let the solution space be fully green? Additionally, anyone have further reading for this type of “very senior IC” operation?
- tsxxst 3y agoFor the "very senior IC", I'd recommend https://staffeng.com https://staffeng.com
- andywarfield 3y agoHere's a really quick story on how i accidentally worked out this strategy by getting it wrong first. When I started at Amazon and was trying to convince the team that we should do certain things, I did what I'd always been trained to do: I wrote down the problem and then sketched a solution to it. Then I'd start floating the doc around to try to get folks excited about it. And invariably, they'd do what they were trained to do, which was to have a critical response to the proposed solution. They'd argue that I was solving it the wrong way, and I'd be in a spot where we'd have a conversation where I was defending a position. But this was the last thing I wanted — I was trying to get everyone excited about fixing a problem, but I slowly realized that when I approached it this way, I was just getting feedback on my proposed solution. So I started doing an experiment where I'd write that same doc, including the ideas i had on the shape of the work we should do, but then I'd delete my solution before sharing it. To your question: I'd still totally write my solution ideas down. Partially because I can't help myself and honestly it was a helpful way to think things through. But when I deleted it and shared a doc with just a problem statement, I'd get feedback on the problem statement. It's pretty obvious, but it was also a pretty surprising result: all of a sudden i was in conversations where we were all on the same side of the table. Feedback was either refining the problem (which was awesome) or proposing solutions. And when the person reading your problem statement starts trying to solve it, it's really cool... because they totally start getting invested and the conversations are great. Like everything, none of this is actually either/or. There are points in between, like including a sketch of the shape of a solution, or properties that a solution would have to have. But the overall thing of separating the problem and the end state of where you want to get to, from the solution and the plan on how to get there is a pretty effective tool from a sharing ownership perspective.
- paulddraper 3y ago> All in, S3 today is composed of hundreds of microservices wow
- mannyv 3y agoWhat most people don't realize is that the magic isn't in handling the system itself; the magic is making authorization appear to be zero-cost. In distributed systems authorization is incredibly difficult. At the scale of AWS it might as well be magic. AWS has a rich permissions model with changes to authorization bubbling through the infrastructure at sub-millisecond speed - while handling probably trillions of requests. This and logging/accounting for billing are the two magic pieces of AWS that I'd love to see an article about. Note that S3 does AA differently than other services, because the permissions are on the resource. I suspect that's for speed?
- awithrow 3y agoKeep in mind that S3 predates IAM by several years. So part of the reason that access to buckets/keys is special is because it was already in place by the time IAM came around. Its likely persisted since than largely since removing the old model would be a difficult taks without potentially breaking a lot of customer's setup
- mannyv 3y agoExactly. This difference makes it easier to (1)understand how IAM works, and (2) how the s3 works...because IAM and S3 work together, but in a different way than the other services. I heard that AA is done via asics, but resource-level permissions implies that authorization is done at the local level for s3. To me that implies that the system extracts S3 permissions from IAM and sends them downstream s3, which get merged with stuff that s3 manages. I guess that occurs when permissions are saved up in IAM world. At some point those need to be joined against a principal somewhere, as roles can exist without assignment. Again, it's be so interesting to see how this is done IRL.
- vdm 3y agoAWS re:Invent 2022 - A day in the life of a billion requests (SEC404) https://www.youtube.com/watch?v=tPr1AgGkvc4 https://www.youtube.com/watch?v=tPr1AgGkvc4
- gooseyman 3y agoThis is a fantastic point on ownership that those “placing” it on others can often miss. “Ownership carries a lot of responsibility, but it also carries a lot of trust – because to let an individual or a team own a service, you have to give them the leeway to make their own decisions about how they are going to deliver it.”
- whoknowswhat11 3y agoOver 100 million requests per second authenticated, billed, versioned, logged, checksummed, encrypted against 200+ trillion objects.
- g9yuayon 3y agoS3 is a truly amazing piece of technology. It offers peace of mind (well, almost), zero operations, and practically unlimited bandwidth for at least analytics workload. Indeed, it's so good that there has not been much progress in building an open-source alternative to S3. There seems not much activity in the Hadoop community. I have yet heard any company who uses RADOS on Ceph to handle PBs of data for analytics workload. MinIO made its name recently, but its license is restrictive and its community is quite small compared to that of Hadoop of its hay days.
- Sparkyte 3y agoThere was a time when S3 was getting resilient. Today it is excellent. Pepridge Farms remembers.
- ddorian43 3y ago> There seems not much activity in the Hadoop community There is apache ozone https://ozone.apache.org/ https://ozone.apache.org/
- g9yuayon 3y agoYeah, Ozone looks interesting. I was just not sure who used it at scale other than a Japanese startup. The community engagement seems much lower than other communities, though.
- _han 3y agoThe talk that this article is based on is available on YouTube: https://www.youtube.com/watch?v=sc3J4McebHE https://www.youtube.com/watch?v=sc3J4McebHE
- supermatt 3y agoHow does S3 handle particularly hot objects? Is there some form of rebalancing to account for access rates?
- dosman33 3y agoI was disappointed too, this article was very light on details about the subject matter. I wasn't expecting a blue-print, but what was presented was all very hand-wavy. In large systems (albeit smaller than S3) the way this works is that you slurp out some performance metrics from storage system to identify your hot spots and then feed that into a service that actively moves stuff around (below the namespace of the filesystem though, will be fs-dependant). You have some higher-performance disk pools at your disposal, and obviously that would be nvme storage today. So in practice, it's likely proprietary vendor code chewing through performance data out of a proprietary storage controller and telling a worker job on a mounted filesystem client to move the hot data to the high performance disk pool. Always constantly rebalancing and moving data back out of the fast pool once it cools off. Obviously for S3 this is happening at an object level though using their own in-house code.
- romantomjak 3y agoApologies if this comes off as blunt, but this is the type of content I come to read at hacker news rather than it being just a series of obituaries. The author has made a lot of great points, but one that stuck with me was: > I consciously spend a lot more time trying to develop problems, and to do a really good job of articulating them, rather than trying to pitch solutions. I haven’t thought of it in this way, but this is an excellent way of motivating someone to “own” a problem.
- dosman33 3y agoNot trying to be an arse, but the guy spent a lot more time talking about himself and other unrelated stuff than about how S3 works. And I don't mind a good article on RAMAC, but that seems... out of place in a discussion about peta-scale storage. I got the strong impression he doesn't really know the finer details of how S3 really works. And that's probably fine for what he's doing, there is plenty of room for application coding, firefighting, and problem management without having to get into the finer details of how it all works.
- kaycebasques 3y ago> we’d read and generally have pretty lively discussions about a collection of “classic” systems research papers Does anyone have the list of papers? > we managed to kind of “industrialize” verification, taking really cool, but kind of research-y techniques for program correctness, and get them into code where normal engineers who don’t have PhDs in formal verification can contribute to maintaining the specification, and that we could continue to apply our tools with every single commit to the software Is any of this open source?
- j_not_j 3y ago> It’s all one thing, and you can’t really think about it just as software. It’s > software, hardware, and people, and it’s always growing and constantly evolving. This is a lesson a lot of software people haven't yet learned. Bad UI, bad operational experiences, insufficient logging to resolve issues, un-fixable code because it's too complicated, and so on. But they use git. The other term of art for this concept is "system engineering", in the aerospace sense. There are a lot of good texts and courses. One example: Wesson: System Analysis Design and Development, Wiley, 2005. ISBN-10 0-471-39333-9
- devilsAdv0cate 3y ago[dead]
- simonebrunozzi 3y agoFrom 2009, a talk I gave about S3 internals [0], when I was Technology Evangelist for AWS. Still relevant today, I believe. [0]: https://vimeo.com/7330740 https://vimeo.com/7330740