12 ms·

Amazon’s distributed computing manifesto (1998)

247 points by werner 4y ago

PaulDavisThe1st 4y ago
As one of the main designers of the original system (but who had left by the time this architectural change was done), that is an interesting read. Always good to see the things that we missed in 1994/1995, even though we believed we were thinking far, far ahead.
- asim 4y ago
  I'm sure it would have been nice to have that tech in 94 and yet at the same time I get the feeling it had to play out the way it did for Amazon to succeed. Without the first part of the journey Amazon would not have gone on to build AWS.
  - pjmlp 4y ago
    The rest of us in 1994 were doing Sun RPC calls, while getting started with DCOM and CORBA, actually quite interesting Amazon's bet on distributed computing given the landscape back then.
    - asim 4y ago
      So interestingly, they made that bet internally, standardised their own platform and then released some sort of abstracted on demand cloud compute services. However the tools they developed for themselves and the SOA style of development would have been valuable to others too. Google did the same. Netflix did the same. None of this stuff really emerged as a product. I'd argue it still hasn't. IF it had, maybe we'd be doing things a bit differently now. But then I guess proprietary RPC based architectures sort of failed along the way when we look at the list you mentioned.
      - pjmlp 4y ago
        The didn't fail per se, hence why we now have gRPC, after WebServices, Jini, RMI, .NET Remoting, XML-RPC, JSON-RPC,... Every generation keeps re-inventing them.
    - PaulDavisThe1st 4y ago
      Shel and I would not have touched DCOM nor CORBA with a 22m fishing rod.
      - pjmlp 4y ago
        Sure, but plenty of us did, Nokia Networks infrastructure had plenty of CORBA for several years, and so did many CERN research projects processing HLT data.
      - manv1 4y ago
        CORBA was interesting, but the authorization side really strangled it. I worked with Tivoli back in the day, which was pretty much the largest production CORBA application in existence. CORBA allowed them to be super flexible when implementing methods, but the auth was brutal. In the end they had to cache all the auth information everywhere just to get decent performance out of it.
      - asim 4y ago
        I've read the stories about what you guys built. It's pretty epic. And not like you were trying to do cool stuff, it was literally based on a need. That's what's amazing. Just manipulating software and infrastructure to do something it wasn't particularly made for just yet.
deleted 4y ago
[deleted]
kerblang 4y ago
> All of this was being done before terms like service-oriented architecture existed. I feel like the first time I heard the term was early 2000's, and wasn't it a mainframe thing first? Dunno, just wondering. Anyhow, it's nicely written, very concise, and worth noting how the original author focuses more on "What kind of realistic options do we have?" than winning the A vs. B vs. C argument in one fell swoop.
- fiddlerwoaroof 4y ago
  Yeah, the buzzwords have changed, but some version of the concept has been in the air at least since I was learning Delphi in the 90s
- awithrow 4y ago
  I remember seeing the term around the mid to late 2000s. But it was also used primarily in the context of enterprisy J2EE, weblogic servers and various IBM hardware that made the everything way more complicated than it needed to be.
- qqtt 4y ago
  Things like Java RMI existed beforehand and there was the elements of industry moving towards server-based partitioning of services - the big difference is none of it was formalized and there was little consistent language of which to speak about these paradigms. At the beginning yes, people would discuss having one mainframe call another mainframe but today that would be SoA.
- numbsafari 4y ago
  It was definitely pre-2000. First “SoA” firm I worked for, I started at in 99, and they had been doing it for 2 years already and most of the crew brought if from a prior gig.
  - dboreham 4y ago
    Although not using the same buzzwords, we had the same architecture deep into the 1980s in Inmos/Transputer/Occam-land.
oscholz 4y ago
- EGreg 4y ago
  What makes it so cool?
oscholz 4y ago
faangst 4y ago
tillmannhgl 4y ago
0xbadcafebee 4y ago
"And with every few orders of magnitude of growth the current architecture would start to show cracks in reliability and performance, and engineers would start to spend more time with virtual duct tape and WD40 than building new innovative products. At each of these inflection points, engineers would invent their way into a new architectural structure to be ready for the next orders of magnitude growth." That last part, to me, is the key to success: getting the whole business to do things in a new way. That is fucking hard. If you can get your business to do it, you have an invaluable superpower. The more things that you can reinvent, faster, gives you more and more superpowers. It's one thing to change your architecture. But also imagine getting every employee to change how they deal with vacations, suppliers, customers, finance, or involving entirely new industries. The easier it is to adapt and change, the longer you survive and the more you thrive. Evolution, baby.
- bwestergard 4y ago
  "But also imagine getting every employee to change how they deal with vacations" Interesting example. Why would changing distributed computing architecture have an impact on vacation policy?
  - 0xbadcafebee 4y ago
    I'm saying architecture is just one way of changing an organization. Other ways of changing an organization, separate from anything technical, might include changing people's schedules or vacation policy, or who you hire, or where, or how. Another would be how you store parts, make orders, assemble products. Or starting work in an entirely new industry. Maybe you work at a company that sometimes works with the government. As a result, the whole company might develop a hiring process which is very slow, very detailed, and excludes certain people from being hired. But probably only a very small number of employees actually have to conform to those government requirements. You can apply them to all new hires "for simplicity", but it makes it harder to hire for non-government positions. So changing how you hire, to make it easier and faster to hire people of a wider background, benefits your organization. If your org can't easily make those changes, it will be disadvantaged.
viburnum 4y ago
I don't think this is really from 1998.
- wging 4y ago
  The blog post is recent, but it describes much older work, so I think the “(1998)” tag is right. “Distributed Computing Manifesto Created: May 24, 1998”
  - thomastjeffery 4y ago
    The blog post (what the OP link takes you to) is from 2022, but the manifesto itself (the substance of the post) is from 1998; so both dates should be used: > Amazon's distributed computing manifesto (1998) (2022)
    - OJFord 4y ago
      Or: > Amazon's 1998 distributed computing manifesto Title has already been changed from TFA to include 'Amazon', and it's still 2022 so no need for that.
epberry 4y ago
> Currently much of our database access is ad hoc with a proliferation of Perl scripts that to a very real extent run our business. There are companies started later than 2010 where this was still the case. Interesting to think about how shipping things quickly is so different than scaling them up.
nevir 4y ago
> We propose moving towards a three-tier architecture where presentation (client), business logic and data are separated. This has also been called a service-based architecture. The applications (clients) would no longer be able to access the database directly, but only through a well-defined interface that encapsulates the business logic required to perform the function. It is really interesting to see a recent(ish) trend away from this three tier design and back towards tighter coupling between application layers. Usually due to increased convenience & developer ergonomics. We've got tools that 'generate' business layers from/for the data layer (Prisma, etc). We've got tools that push business logic to the client (Meteor, Firebase, etc)
- amcvitty 4y ago
  A big part of the difference has to be that if you have a small number of developers (esp. n=1) and you can deploy everything at once, then those layers just get in the way of fast change. It seems Amazon were optimising for the ability to distribute data because they had big volume, and hide its form so they could change it without having to change lots of applications. Of course, there’s some cargo culting around services where people jump to that architecture before they need it, but for most apps YAGNI. it’s cool that their architecture was driven by clear needs “just in time” to allow them to continue to scale
- tsss 4y ago
  Nowadays you separate service by business capability and not by "layer". Layers just lead to a dependencies and dependencies lead to bad reliability and terrible development speed.
  - derefr 4y ago
    What Amazon were describing here is simply the division between a frontend web gateway service (or, in modernity, client-delivered SPAs); an API backend service to serve the XHRs of the web-gateway / SPA; and some kind of DBMS where user-visible query schema is separable from storage architecture via e.g. views. I don't think there's any modern system that doesn't have those things, no?
manv1 4y ago
This is one example of the CEO making something happen that essentially birthed AWS. Bezos, of all people, was like "make it happen." And it did. It was basically work for no reason except future proofing. Having someone up the food chain OK this much work for the future (and no hard dollar benefit) is highly unusual. And besides that they've done some incredible things with their infrastructure, like authorization. Distributed authorization is really hard, but at AWS it's completely invisible. Remove a permission from an IAM role and it moves through AWS really, really fast. It's totally magic. Anyone who was abused by CORBA knows how hard that is to do well. Their newer stuff (like Cognito) is sort of weird, but other things are surprisingly solid given how big AWS is. Small shops have trouble shipping feature complete software, and BigCorps can be even worse. AWS has gotten really good at it.
- p_l 4y ago
  It's easy till you break IAM itself with your policy complexity and random services start dying because other AWS components few layers deep cnst get IAM to parse
  - another_devy 4y ago
    Intriguing, can you share details or overview why it failed for you. Will be kind of gotchas for me
    - p_l 4y ago
      Essentially there's a maximum size of IAM policy, which AFAIK is not documented properly anywhere - get close to it or exceed it and you start getting random failures everywhere.
      - donavanm 4y ago
        Character limits & the number of applied policies are all publicly documented https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_iam-quotas.html#reference_iam-quotas-entity-length https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_i.... Im not aware of any evaluation complexity limits and have never run in to that sort of problem in my ~10 years of dealing with IAM. I expect you ran in to this sharp bit "You can add as many inline policies as you want to an IAM user, role, or group. But the total aggregate policy size (the sum size of all inline policies) per entity cannot exceed the following limits." Calculating the sum would be a pain as a user.
wging 4y ago
It’s interesting to think about how much of a perspective shift this must have been, especially the service oriented bits. Interestingly, I think it might not have even been made completely in the authors’ minds at the time of this proposal. (Which is understandable, of course. It’s a proposal, not a retrospective on already accepted ideas.) For example, > In the case of DC processing, customer service and other functions need to be able to determine where a customer order or shipment is in the pipeline. The mechanism that we propose using is one where certain nodes along the workflow insert a row into some centralized database instance to indicate the current state of the workflow element being processed. definitely doesn’t seem to reflect the hiding of a database behind an interface. (From a workflow node’s perspective, rows in that centralized database should be an implementation detail it has no knowledge of.) Then again, this is part of their pitch for workflow processing, not service-oriented architecture.
- ajkjk 4y ago
  Anecdotally, it was at least 2015 before the DC processing system was actually mostly operating against service-oriented interfaces (when I left in 2016 we had a few old tools left that still talked to the databases directly :/ ).
benhoyt 4y ago
I think this is a very interesting and clear manifesto, and almost certainly the right thing for Amazon at the right time (and presumably was part of what led to AWS). However, at one of the previous companies I worked at we got an ex-Amazon person as CTO and he grew the company from 50 engineers to 500 in 2 years, and pushed microservices everywhere. Very impressive, and I think all-microservices made sense at one level to handle that sort of growth, but it doesn't make sense technically in a lot of cases. Essentially I think we've gone too far: service-oriented architectures turned into "micro" services, which come with a lot of complexity and distributed systems issues. I think for most small companies monoliths are right, for medium-sized companies (say 50+) it makes sense to carefully introduce a few separate services, and only for large companies (say 300+) does many services (which may or may not be "micro") start to pay off. I've heard it said that "microservices solve a people problem, not a technical one", and I think that's true.
- jerglingu 4y ago
  This sounds similar to Flexport's CTO situation (he came from Amazon) and attempted microservice-ification of everything. Except it sounds like they weren't even able to get wheels up and are still floundering after years of planning and attempted execution.
- mongol 4y ago
  It is most likely true. If you take away all constraints related to planning, communication, prioritization, collaboration, development efficiency etc, most of the arguments against monoliths goes away. What remains are considerations for memory, bandwidth and similar that could motivate a breakdown.
npalli 4y ago
Seeing that Werner Vogels has submitted this entry, I wonder if he can comment how long it took to actually build this out. When did a form of this service oriented architecture work in production at Amazon?

247 points by werner 4y ago

As one of the main designers of the original system (but who had left by the time this architectural change was done), that is an interesting read. Always good to see the things that we missed in 1994/1995, even though we believed we were thinking far, far ahead.

asim 4y ago
I'm sure it would have been nice to have that tech in 94 and yet at the same time I get the feeling it had to play out the way it did for Amazon to succeed. Without the first part of the journey Amazon would not have gone on to build AWS.
- pjmlp 4y ago
  The rest of us in 1994 were doing Sun RPC calls, while getting started with DCOM and CORBA, actually quite interesting Amazon's bet on distributed computing given the landscape back then.
  - asim 4y ago
    So interestingly, they made that bet internally, standardised their own platform and then released some sort of abstracted on demand cloud compute services. However the tools they developed for themselves and the SOA style of development would have been valuable to others too. Google did the same. Netflix did the same. None of this stuff really emerged as a product. I'd argue it still hasn't. IF it had, maybe we'd be doing things a bit differently now. But then I guess proprietary RPC based architectures sort of failed along the way when we look at the list you mentioned.
    - pjmlp 4y ago
      The didn't fail per se, hence why we now have gRPC, after WebServices, Jini, RMI, .NET Remoting, XML-RPC, JSON-RPC,... Every generation keeps re-inventing them.
  - PaulDavisThe1st 4y ago
    Shel and I would not have touched DCOM nor CORBA with a 22m fishing rod.
    - pjmlp 4y ago
      Sure, but plenty of us did, Nokia Networks infrastructure had plenty of CORBA for several years, and so did many CERN research projects processing HLT data.
    - manv1 4y ago
      CORBA was interesting, but the authorization side really strangled it. I worked with Tivoli back in the day, which was pretty much the largest production CORBA application in existence. CORBA allowed them to be super flexible when implementing methods, but the auth was brutal. In the end they had to cache all the auth information everywhere just to get decent performance out of it.
    - asim 4y ago
      I've read the stories about what you guys built. It's pretty epic. And not like you were trying to do cool stuff, it was literally based on a need. That's what's amazing. Just manipulating software and infrastructure to do something it wasn't particularly made for just yet.

deleted 4y ago

[deleted]

kerblang 4y ago

> All of this was being done before terms like service-oriented architecture existed. I feel like the first time I heard the term was early 2000's, and wasn't it a mainframe thing first? Dunno, just wondering. Anyhow, it's nicely written, very concise, and worth noting how the original author focuses more on "What kind of realistic options do we have?" than winning the A vs. B vs. C argument in one fell swoop.

fiddlerwoaroof 4y ago
Yeah, the buzzwords have changed, but some version of the concept has been in the air at least since I was learning Delphi in the 90s
awithrow 4y ago
I remember seeing the term around the mid to late 2000s. But it was also used primarily in the context of enterprisy J2EE, weblogic servers and various IBM hardware that made the everything way more complicated than it needed to be.
qqtt 4y ago
Things like Java RMI existed beforehand and there was the elements of industry moving towards server-based partitioning of services - the big difference is none of it was formalized and there was little consistent language of which to speak about these paradigms. At the beginning yes, people would discuss having one mainframe call another mainframe but today that would be SoA.
numbsafari 4y ago
It was definitely pre-2000. First “SoA” firm I worked for, I started at in 99, and they had been doing it for 2 years already and most of the crew brought if from a prior gig.
- dboreham 4y ago
  Although not using the same buzzwords, we had the same architecture deep into the 1980s in Inmos/Transputer/Occam-land.

oscholz 4y ago

EGreg 4y ago
What makes it so cool?

oscholz 4y ago

faangst 4y ago

tillmannhgl 4y ago

0xbadcafebee 4y ago

"And with every few orders of magnitude of growth the current architecture would start to show cracks in reliability and performance, and engineers would start to spend more time with virtual duct tape and WD40 than building new innovative products. At each of these inflection points, engineers would invent their way into a new architectural structure to be ready for the next orders of magnitude growth." That last part, to me, is the key to success: getting the whole business to do things in a new way. That is fucking hard. If you can get your business to do it, you have an invaluable superpower. The more things that you can reinvent, faster, gives you more and more superpowers. It's one thing to change your architecture. But also imagine getting every employee to change how they deal with vacations, suppliers, customers, finance, or involving entirely new industries. The easier it is to adapt and change, the longer you survive and the more you thrive. Evolution, baby.

bwestergard 4y ago
"But also imagine getting every employee to change how they deal with vacations" Interesting example. Why would changing distributed computing architecture have an impact on vacation policy?
- 0xbadcafebee 4y ago
  I'm saying architecture is just one way of changing an organization. Other ways of changing an organization, separate from anything technical, might include changing people's schedules or vacation policy, or who you hire, or where, or how. Another would be how you store parts, make orders, assemble products. Or starting work in an entirely new industry. Maybe you work at a company that sometimes works with the government. As a result, the whole company might develop a hiring process which is very slow, very detailed, and excludes certain people from being hired. But probably only a very small number of employees actually have to conform to those government requirements. You can apply them to all new hires "for simplicity", but it makes it harder to hire for non-government positions. So changing how you hire, to make it easier and faster to hire people of a wider background, benefits your organization. If your org can't easily make those changes, it will be disadvantaged.

viburnum 4y ago

I don't think this is really from 1998.

wging 4y ago
The blog post is recent, but it describes much older work, so I think the “(1998)” tag is right. “Distributed Computing Manifesto Created: May 24, 1998”
- thomastjeffery 4y ago
  The blog post (what the OP link takes you to) is from 2022, but the manifesto itself (the substance of the post) is from 1998; so both dates should be used: > Amazon's distributed computing manifesto (1998) (2022)
  - OJFord 4y ago
    Or: > Amazon's 1998 distributed computing manifesto Title has already been changed from TFA to include 'Amazon', and it's still 2022 so no need for that.

epberry 4y ago

> Currently much of our database access is ad hoc with a proliferation of Perl scripts that to a very real extent run our business. There are companies started later than 2010 where this was still the case. Interesting to think about how shipping things quickly is so different than scaling them up.

nevir 4y ago

> We propose moving towards a three-tier architecture where presentation (client), business logic and data are separated. This has also been called a service-based architecture. The applications (clients) would no longer be able to access the database directly, but only through a well-defined interface that encapsulates the business logic required to perform the function. It is really interesting to see a recent(ish) trend away from this three tier design and back towards tighter coupling between application layers. Usually due to increased convenience & developer ergonomics. We've got tools that 'generate' business layers from/for the data layer (Prisma, etc). We've got tools that push business logic to the client (Meteor, Firebase, etc)

amcvitty 4y ago
A big part of the difference has to be that if you have a small number of developers (esp. n=1) and you can deploy everything at once, then those layers just get in the way of fast change. It seems Amazon were optimising for the ability to distribute data because they had big volume, and hide its form so they could change it without having to change lots of applications. Of course, there’s some cargo culting around services where people jump to that architecture before they need it, but for most apps YAGNI. it’s cool that their architecture was driven by clear needs “just in time” to allow them to continue to scale
tsss 4y ago
Nowadays you separate service by business capability and not by "layer". Layers just lead to a dependencies and dependencies lead to bad reliability and terrible development speed.
- derefr 4y ago
  What Amazon were describing here is simply the division between a frontend web gateway service (or, in modernity, client-delivered SPAs); an API backend service to serve the XHRs of the web-gateway / SPA; and some kind of DBMS where user-visible query schema is separable from storage architecture via e.g. views. I don't think there's any modern system that doesn't have those things, no?

manv1 4y ago

This is one example of the CEO making something happen that essentially birthed AWS. Bezos, of all people, was like "make it happen." And it did. It was basically work for no reason except future proofing. Having someone up the food chain OK this much work for the future (and no hard dollar benefit) is highly unusual. And besides that they've done some incredible things with their infrastructure, like authorization. Distributed authorization is really hard, but at AWS it's completely invisible. Remove a permission from an IAM role and it moves through AWS really, really fast. It's totally magic. Anyone who was abused by CORBA knows how hard that is to do well. Their newer stuff (like Cognito) is sort of weird, but other things are surprisingly solid given how big AWS is. Small shops have trouble shipping feature complete software, and BigCorps can be even worse. AWS has gotten really good at it.

p_l 4y ago
It's easy till you break IAM itself with your policy complexity and random services start dying because other AWS components few layers deep cnst get IAM to parse
- another_devy 4y ago
  Intriguing, can you share details or overview why it failed for you. Will be kind of gotchas for me
  - p_l 4y ago
    Essentially there's a maximum size of IAM policy, which AFAIK is not documented properly anywhere - get close to it or exceed it and you start getting random failures everywhere.
    - donavanm 4y ago
      Character limits & the number of applied policies are all publicly documented https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_iam-quotas.html#reference_iam-quotas-entity-length https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_i.... Im not aware of any evaluation complexity limits and have never run in to that sort of problem in my ~10 years of dealing with IAM. I expect you ran in to this sharp bit "You can add as many inline policies as you want to an IAM user, role, or group. But the total aggregate policy size (the sum size of all inline policies) per entity cannot exceed the following limits." Calculating the sum would be a pain as a user.

wging 4y ago

It’s interesting to think about how much of a perspective shift this must have been, especially the service oriented bits. Interestingly, I think it might not have even been made completely in the authors’ minds at the time of this proposal. (Which is understandable, of course. It’s a proposal, not a retrospective on already accepted ideas.) For example, > In the case of DC processing, customer service and other functions need to be able to determine where a customer order or shipment is in the pipeline. The mechanism that we propose using is one where certain nodes along the workflow insert a row into some centralized database instance to indicate the current state of the workflow element being processed. definitely doesn’t seem to reflect the hiding of a database behind an interface. (From a workflow node’s perspective, rows in that centralized database should be an implementation detail it has no knowledge of.) Then again, this is part of their pitch for workflow processing, not service-oriented architecture.

ajkjk 4y ago
Anecdotally, it was at least 2015 before the DC processing system was actually mostly operating against service-oriented interfaces (when I left in 2016 we had a few old tools left that still talked to the databases directly :/ ).

benhoyt 4y ago

I think this is a very interesting and clear manifesto, and almost certainly the right thing for Amazon at the right time (and presumably was part of what led to AWS). However, at one of the previous companies I worked at we got an ex-Amazon person as CTO and he grew the company from 50 engineers to 500 in 2 years, and pushed microservices everywhere. Very impressive, and I think all-microservices made sense at one level to handle that sort of growth, but it doesn't make sense technically in a lot of cases. Essentially I think we've gone too far: service-oriented architectures turned into "micro" services, which come with a lot of complexity and distributed systems issues. I think for most small companies monoliths are right, for medium-sized companies (say 50+) it makes sense to carefully introduce a few separate services, and only for large companies (say 300+) does many services (which may or may not be "micro") start to pay off. I've heard it said that "microservices solve a people problem, not a technical one", and I think that's true.

jerglingu 4y ago
This sounds similar to Flexport's CTO situation (he came from Amazon) and attempted microservice-ification of everything. Except it sounds like they weren't even able to get wheels up and are still floundering after years of planning and attempted execution.
mongol 4y ago
It is most likely true. If you take away all constraints related to planning, communication, prioritization, collaboration, development efficiency etc, most of the arguments against monoliths goes away. What remains are considerations for memory, bandwidth and similar that could motivate a breakdown.

npalli 4y ago

Seeing that Werner Vogels has submitted this entry, I wonder if he can comment how long it took to actually build this out. When did a form of this service oriented architecture work in production at Amazon?