8 ms·

Predicting the Future of Distributed Systems

173 points by borisjabes 2y ago

ent101 2y ago
[flagged]
- remram 2y ago
  What does "distributed" mean? Looking through your link, what I see is a remote desktop app (e.g. client/server).
  - smokeydoe 2y ago
    It's not really related to the article. It's just a reason to plug his project again. Like they do on reddit.
  - LoganDark 2y ago
    Puter hasn't anything to do with remote desktop. It's a desktop environment-like thing that runs in the browser, as a webpage. All its apps are HTML5 apps.
    - deleted 2y ago
      [deleted]
    - remram 2y ago
      And none of it distributed. Ok.
      - LoganDark 2y ago
        I guess by distributed they mean everything you do is stored in the cloud.
awkii 2y ago
I think the author has a point with one-way doors slowing down the adoption of distributed systems. The best way to build two way doors is to push for industry adoption of a particular API. In theory the backend of these APIs matter little to me, the developer, so long as they are fast and consistent. Some examples that come to mind is that Apache Beam is a "programming model" for Data pipelines, Akka is a "programming model" for stateful distributed systems, OpenTelemetry for logging/telemetry, and Kubernetes for orchestration. Oh, and local development is a strong preference.
- jauntywundrkind 2y ago
  OTel being a capture & ingest only specification is kind of messed up. There's no attempt from what I can tell for how to query or present stored data; it's just an over-the-wire specification, & that drastically limits usable scope. It means vendors each get to make their own services & backends & tools, but it's greviously limiting the effort as a whole, makes even an open spec like OTel a one-way door. Ideally OTel would be more than observability, imo. Traces would be event-sources, would be a thing that begets more computing. The system to observe computing should in turn also be the system to react & respond to computing, should be begetting more computing elsewhere. That's the distributed system I want to see; local agents reporting their actions, other agents seeing that & responding. OTel would fit the need well, if only we could expand ourselves beyond thinking of observability as an operator affordance & start thinking of it as the system to track computation.
  - singron 2y ago
    Otel works as a standard since there isn't any need to innovate at that level. Despite the over complications it has, all the implementations largely have the same requirements, and it's useful to instrument everything the same way. Querying unfortunately has lots of room for innovation, and it's really hard to nail down in a spec especially when the vendors all want to compete.
    - __turbobrew__ 2y ago
      Otel is nice and all but I still think you are best off going 100% all in prometheus. Prometheus is so common that it has become a de-facto standard in metrics. At BigCo we have migrated a number of internal things to Otel but I don’t think it has been worth the effort. So many projects come with prometheus metrics, dashboards, and alerts out of the box that it becomes hard to use anything else. When I pick some random helm chart to install you can almost guarantee that is comes with prometheus integrations. With grafana mimir you can now scale easily to a few billion metrics streams so a lot of the issues with the old model of prometheus have been fixed. Like you said I don’t think there is much to innovate on in this area, which is a good thing.
notverysubtle 2y ago
The right time to mention Designing Data-Intensive Applications by Martin Kleppmann. Amazing book explaining distributed systems concepts in a digestible language.
samstave 2y ago
>>the biggest opportunity for a new programming model is extracting the majority of the code from an application and moving it into the infrastructure instead. The second biggest opportunity is for the remaining code—what people refer to as the business logic, the essence of the program—to be portable and secure. This was such a well put comment, that truly made me grok the entire article in just this one statement. --- Infrastructure needs to be invisible, and that is where the future of AI-enabled orchestration/abstraction will allow development to be more poetry than code - whereby we can describe complex logic paths/workflows in a language of intent - and all the components required to accomplish that desired outcome will be much more quickly, elegantly be a reality. THe real challenge ahead is the divide between those who have the capability and power of all the AI tools available to them, and those who are subjugated by those who do. For example, an individual can build a lot with the current state of the available tool universe... but a more sophisticated and well funded organization will have a lot more potential capability.' What I am really interested to know, is if there is a dark Dystopian Cyberpunk AI under-world happening yet? Whats the state of BadActor/BigCorpo/BigSpy's capability and covert actions currently? While we are distracted by AI_ClipArt and celebrity voice squabbles, and seemingly Top AI Voices are being ignored after founding organizations for Alignment/Governance/Humane/etc and warning of catastrophe - define The State of Things? But yeah - extracting the code and letting logic just be handled yet portable, clonable, refactorable easily is where we are already headed. Its amazing and terrifying at the same time. I'm thankful that all my Cyberpunk Fantasy reading, thinking, imagining and then my tiny part in the overall evolution of the world of tech today, having the opportunity to be here, worked with and build to, in with -- and now seeing the birth of AI and using it daily in my actual interactions with my IRL. Such an amazing moment in Human History to be here through this.
BraveNewCurency 2y ago
> One-Way-Door and Two-Way-Door Decisions See also the "Linux kernel management style" document that's been in the kernel since forever: https://docs.kernel.org/6.1/process/management-style.html https://docs.kernel.org/6.1/process/management-style.html
- ent101 2y ago
  > Most people are idiots, and being a manager means you’ll have to deal with it, and perhaps more importantly, that they have to deal with you. > It turns out that while it’s easy to undo technical mistakes, it’s not as easy to undo personality disorders. You just have to live with theirs - and yours. this was definitely written by Linus XD
- ZaoLahma 2y ago
  I really like the avoidance (elimination) of one-way-door decisions by turning them into several small(er) two-way-door decisions. I guess the software development interpretation of it is clearly defined boundaries of responsibility, and avoiding to leak implementation details beyond those?
deleted 2y ago
[deleted]
jamesblonde 2y ago
I really enjoyed this article. The one point I have issue with is that the dominance of object storage in today's distributed systems is very much due to economics, not technology. There's basically cheering every little step S3 takes towards a POSIX-like distributed file system like HDFS - "consistent listing of files, yeah!". Last week it was preconditions for writing files. There's still huge gymnastics needed in Iceberg/Delta to work with S3 given the lack of atomic rename.
jensneuse 2y ago
I'd like to add that I'm seeing more and more companies unifying synchronous and asynchronous APIs. With the concept of GraphQL Federation, it's possible to "extend" Entities by defining their (primary) keys in a GraphQL Schema. If we're combining this with Async APIs, e.g. NATS or Kafka, we can enable teams to build APIs around events, while still being able to distribute the implementation of how certain fields can be resolved. The Federation Router then joins the Stream with additional data from synchronous services, a very powerful pattern I believe. I wrote a bit more on the topic here: https://wundergraph.com/blog/distributed_graphql_subscriptions_with_nats_and_event_driven_architecture https://wundergraph.com/blog/distributed_graphql_subscriptio...
buro9 2y ago
Things I have come to know about distributed systems: The S3 API (object storage) is the accepted storage API, but you do not need AWS (but they are very good at this). The Kafka API is the accepted stream/ buffer/ queue API, but you do not need Confluent. SQL is the query language, but you do not need a relational database.
- jensneuse 2y ago
  I'd argue that a lot of people are moving from Kafka to NATS. NATS and Kafka serve different purposes and for many use cases related to APIs, NATS has a lot more to offer, like like wildcard topic topologies.
  - buro9 2y ago
    it's the Kafka API, not Kafka itself, that I see as having become the standard.
    - xyzzy_plugh 2y ago
      Which is honestly a shame. It's an awful API.
      - ako 2y ago
        Why?
    - 62951413 2y ago
      Ce n'est pas un Kafka: Kafka is a Protocol Apache Kafka is an aging open source project. It's time to accept that Kafka's protocol is what matters. (https://materializedview.io/p/ce-nest-pas-un-kafka https://materializedview.io/p/ce-nest-pas-un-kafka)
willvarfar 2y ago
Something I anticipate is smarter storage that can do some filtering on push down predicates. There's compute on the storage nodes that is being wasted today. I was kinda expecting BigQuery to do this under the hood, but it seems like they don't, which is a shame. BigQuery isn't faster than, say, trino on gcs, even though Google could do some major optimisations here.
- okr 2y ago
  I also wonder if Athena does this with AWS. Parquet supports pushdown. But i would suspect, pushdown predicates would mean that the file storage unit has to have some logic to execute custom code, bringing back the code to the data. The promise of spark, once. It would be a huge win, definitly. Hmmm. But opens up also a threat vector. And you have competing users running their predicates. So one has to think also about queues and pipelining and so on. But probably also solvable, just like on any multiuser system. Interesting.
- leventov 2y ago
  BigQuery Storage Read API claims to support filters and simple projections pushed down to the storage: https://cloud.google.com/bigquery/docs/reference/storage https://cloud.google.com/bigquery/docs/reference/storage. See also this recent paper: https://research.google/pubs/biglake-bigquerys-evolution-toward-a-multi-cloud-lakehouse/ https://research.google/pubs/biglake-bigquerys-evolution-tow... I've also recently proposed a Table Read protocol that should be a "non-vendor-controlled" equivalent of BigQuery Storage APIs: https://engineeringideas.substack.com/p/table-transfer-protocols-for-universal https://engineeringideas.substack.com/p/table-transfer-proto...
purpleidea 2y ago
> Programming Models If you read this section, the author gets a lot of things right, but clearly doesn't know the space that well since there have been people building things along these lines for years. And making vague commentary instead of describing the nitty-gritty doesn't evoke much confidence. I work on one such language/tool called mgmt config, but I have had virtually no interest and/or skill in marketing it. TBQH, I'm disenchanted by the fact that it seems to get any recognition you need to have VC's and a three-year timeline, short-term goals, and a plan to be done by then or move on. If you're serious about future infra, then it's all here: https://github.com/purpleidea/mgmt/ https://github.com/purpleidea/mgmt/ Looking for coding help for some of the harder bits that people might wish to add, and for people to take it into production and find issues that we've missed.
- lifty 2y ago
  I remember seeing your presentation many years ago, at Fosdem. Very cool project and if I would have to manage classic OS deployments I would definitely give mgmt a try. That being said, I think the world is moving to more immutable systems similar to how Talos works (https://talos.dev https://talos.dev).
  - karmarepellent 2y ago
    I would be hesitant to claim "the world is moving to" anything, really. Deployments that would now be called "traditional", so anything that does not run in a container but in a VM, will continue to exist for quite some time. And not only because of legacy systems that are hard to migrate to a modern platform. At my place of work there are workloads that can easily run on Kubernetes and it would be wise to do so. On the other hand there are systems that are not designed to run in a container and there is frankly no need to, because not everything needs to scale up and down or be available 100% of the time at all costs. I think configuration management systems like mgmt (or Ansible and Puppet) are here to stay.
    - lifty 2y ago
      Not disagreeing with you there; technology lingers for many years. But in terms of market share and mind share, configuration management has shrank in dominance and I suspect it will continue to do so.
nyrikki 2y ago
On a unrelated note, does anyone know the origins of the one way vs two way door analogy? In this post it is attributed to Jeff Bezos quotes, but it was popular in the Pacific North West before his rise.
pjdesno 2y ago
Something missing here in the discussion of object storage and databases is any mention of the declining importance of the file system. From the 70s through the 90s or 00s everything was file system-based, and it was just assumed that the best way to store data in a distributed system - even a globally-distributed one - was some sort of distributed file system. (e.g. Andrew File System, or research projects like OceanStore. Nowadays the file system holds applications and configuration, but applications mostly store data in databases and object stores. In distributed systems this is done almost exclusively through system-specific network connections (e.g. port 3306 to MySQL, or HTTP for S3) rather than OS-level mounting of a file system. (not counting HPC, where distributed file systems are used to preserve the developer look and feel of early non-distributed HPC systems)
__turbobrew__ 2y ago
Pushing as much down to the infra sounds like aws lambda and friends. You basically upload a zip or container and say, “just run this business code somewhere, I don’t care”. OCI bundles are basically a two day door at this point, you can build them with many tools, and run them with many other tools. It works great for stateless things, but not so great for stateful things. I guess this plays into state being persisted in object storage or DBs, this allows the application to be stateless.

173 points by borisjabes 2y ago

ent101 2y ago

[flagged]

remram 2y ago
What does "distributed" mean? Looking through your link, what I see is a remote desktop app (e.g. client/server).
- smokeydoe 2y ago
  It's not really related to the article. It's just a reason to plug his project again. Like they do on reddit.
- LoganDark 2y ago
  Puter hasn't anything to do with remote desktop. It's a desktop environment-like thing that runs in the browser, as a webpage. All its apps are HTML5 apps.
  - deleted 2y ago
    [deleted]
  - remram 2y ago
    And none of it distributed. Ok.
    - LoganDark 2y ago
      I guess by distributed they mean everything you do is stored in the cloud.

awkii 2y ago

I think the author has a point with one-way doors slowing down the adoption of distributed systems. The best way to build two way doors is to push for industry adoption of a particular API. In theory the backend of these APIs matter little to me, the developer, so long as they are fast and consistent. Some examples that come to mind is that Apache Beam is a "programming model" for Data pipelines, Akka is a "programming model" for stateful distributed systems, OpenTelemetry for logging/telemetry, and Kubernetes for orchestration. Oh, and local development is a strong preference.

jauntywundrkind 2y ago
OTel being a capture & ingest only specification is kind of messed up. There's no attempt from what I can tell for how to query or present stored data; it's just an over-the-wire specification, & that drastically limits usable scope. It means vendors each get to make their own services & backends & tools, but it's greviously limiting the effort as a whole, makes even an open spec like OTel a one-way door. Ideally OTel would be more than observability, imo. Traces would be event-sources, would be a thing that begets more computing. The system to observe computing should in turn also be the system to react & respond to computing, should be begetting more computing elsewhere. That's the distributed system I want to see; local agents reporting their actions, other agents seeing that & responding. OTel would fit the need well, if only we could expand ourselves beyond thinking of observability as an operator affordance & start thinking of it as the system to track computation.
- singron 2y ago
  Otel works as a standard since there isn't any need to innovate at that level. Despite the over complications it has, all the implementations largely have the same requirements, and it's useful to instrument everything the same way. Querying unfortunately has lots of room for innovation, and it's really hard to nail down in a spec especially when the vendors all want to compete.
  - __turbobrew__ 2y ago
    Otel is nice and all but I still think you are best off going 100% all in prometheus. Prometheus is so common that it has become a de-facto standard in metrics. At BigCo we have migrated a number of internal things to Otel but I don’t think it has been worth the effort. So many projects come with prometheus metrics, dashboards, and alerts out of the box that it becomes hard to use anything else. When I pick some random helm chart to install you can almost guarantee that is comes with prometheus integrations. With grafana mimir you can now scale easily to a few billion metrics streams so a lot of the issues with the old model of prometheus have been fixed. Like you said I don’t think there is much to innovate on in this area, which is a good thing.

notverysubtle 2y ago

The right time to mention Designing Data-Intensive Applications by Martin Kleppmann. Amazing book explaining distributed systems concepts in a digestible language.

samstave 2y ago

>>the biggest opportunity for a new programming model is extracting the majority of the code from an application and moving it into the infrastructure instead. The second biggest opportunity is for the remaining code—what people refer to as the business logic, the essence of the program—to be portable and secure. This was such a well put comment, that truly made me grok the entire article in just this one statement. --- Infrastructure needs to be invisible, and that is where the future of AI-enabled orchestration/abstraction will allow development to be more poetry than code - whereby we can describe complex logic paths/workflows in a language of intent - and all the components required to accomplish that desired outcome will be much more quickly, elegantly be a reality. THe real challenge ahead is the divide between those who have the capability and power of all the AI tools available to them, and those who are subjugated by those who do. For example, an individual can build a lot with the current state of the available tool universe... but a more sophisticated and well funded organization will have a lot more potential capability.' What I am really interested to know, is if there is a dark Dystopian Cyberpunk AI under-world happening yet? Whats the state of BadActor/BigCorpo/BigSpy's capability and covert actions currently? While we are distracted by AI_ClipArt and celebrity voice squabbles, and seemingly Top AI Voices are being ignored after founding organizations for Alignment/Governance/Humane/etc and warning of catastrophe - define The State of Things? But yeah - extracting the code and letting logic just be handled yet portable, clonable, refactorable easily is where we are already headed. Its amazing and terrifying at the same time. I'm thankful that all my Cyberpunk Fantasy reading, thinking, imagining and then my tiny part in the overall evolution of the world of tech today, having the opportunity to be here, worked with and build to, in with -- and now seeing the birth of AI and using it daily in my actual interactions with my IRL. Such an amazing moment in Human History to be here through this.

BraveNewCurency 2y ago

> One-Way-Door and Two-Way-Door Decisions See also the "Linux kernel management style" document that's been in the kernel since forever: https://docs.kernel.org/6.1/process/management-style.html https://docs.kernel.org/6.1/process/management-style.html

ent101 2y ago
> Most people are idiots, and being a manager means you’ll have to deal with it, and perhaps more importantly, that they have to deal with you. > It turns out that while it’s easy to undo technical mistakes, it’s not as easy to undo personality disorders. You just have to live with theirs - and yours. this was definitely written by Linus XD
ZaoLahma 2y ago
I really like the avoidance (elimination) of one-way-door decisions by turning them into several small(er) two-way-door decisions. I guess the software development interpretation of it is clearly defined boundaries of responsibility, and avoiding to leak implementation details beyond those?

deleted 2y ago

[deleted]

jamesblonde 2y ago

I really enjoyed this article. The one point I have issue with is that the dominance of object storage in today's distributed systems is very much due to economics, not technology. There's basically cheering every little step S3 takes towards a POSIX-like distributed file system like HDFS - "consistent listing of files, yeah!". Last week it was preconditions for writing files. There's still huge gymnastics needed in Iceberg/Delta to work with S3 given the lack of atomic rename.

jensneuse 2y ago

I'd like to add that I'm seeing more and more companies unifying synchronous and asynchronous APIs. With the concept of GraphQL Federation, it's possible to "extend" Entities by defining their (primary) keys in a GraphQL Schema. If we're combining this with Async APIs, e.g. NATS or Kafka, we can enable teams to build APIs around events, while still being able to distribute the implementation of how certain fields can be resolved. The Federation Router then joins the Stream with additional data from synchronous services, a very powerful pattern I believe. I wrote a bit more on the topic here: https://wundergraph.com/blog/distributed_graphql_subscriptions_with_nats_and_event_driven_architecture https://wundergraph.com/blog/distributed_graphql_subscriptio...

buro9 2y ago

Things I have come to know about distributed systems: The S3 API (object storage) is the accepted storage API, but you do not need AWS (but they are very good at this). The Kafka API is the accepted stream/ buffer/ queue API, but you do not need Confluent. SQL is the query language, but you do not need a relational database.

jensneuse 2y ago
I'd argue that a lot of people are moving from Kafka to NATS. NATS and Kafka serve different purposes and for many use cases related to APIs, NATS has a lot more to offer, like like wildcard topic topologies.
- buro9 2y ago
  it's the Kafka API, not Kafka itself, that I see as having become the standard.
  - xyzzy_plugh 2y ago
    Which is honestly a shame. It's an awful API.
    - ako 2y ago
      Why?
  - 62951413 2y ago
    Ce n'est pas un Kafka: Kafka is a Protocol Apache Kafka is an aging open source project. It's time to accept that Kafka's protocol is what matters. (https://materializedview.io/p/ce-nest-pas-un-kafka https://materializedview.io/p/ce-nest-pas-un-kafka)

willvarfar 2y ago

Something I anticipate is smarter storage that can do some filtering on push down predicates. There's compute on the storage nodes that is being wasted today. I was kinda expecting BigQuery to do this under the hood, but it seems like they don't, which is a shame. BigQuery isn't faster than, say, trino on gcs, even though Google could do some major optimisations here.

okr 2y ago
I also wonder if Athena does this with AWS. Parquet supports pushdown. But i would suspect, pushdown predicates would mean that the file storage unit has to have some logic to execute custom code, bringing back the code to the data. The promise of spark, once. It would be a huge win, definitly. Hmmm. But opens up also a threat vector. And you have competing users running their predicates. So one has to think also about queues and pipelining and so on. But probably also solvable, just like on any multiuser system. Interesting.
leventov 2y ago
BigQuery Storage Read API claims to support filters and simple projections pushed down to the storage: https://cloud.google.com/bigquery/docs/reference/storage https://cloud.google.com/bigquery/docs/reference/storage. See also this recent paper: https://research.google/pubs/biglake-bigquerys-evolution-toward-a-multi-cloud-lakehouse/ https://research.google/pubs/biglake-bigquerys-evolution-tow... I've also recently proposed a Table Read protocol that should be a "non-vendor-controlled" equivalent of BigQuery Storage APIs: https://engineeringideas.substack.com/p/table-transfer-protocols-for-universal https://engineeringideas.substack.com/p/table-transfer-proto...

purpleidea 2y ago

> Programming Models If you read this section, the author gets a lot of things right, but clearly doesn't know the space that well since there have been people building things along these lines for years. And making vague commentary instead of describing the nitty-gritty doesn't evoke much confidence. I work on one such language/tool called mgmt config, but I have had virtually no interest and/or skill in marketing it. TBQH, I'm disenchanted by the fact that it seems to get any recognition you need to have VC's and a three-year timeline, short-term goals, and a plan to be done by then or move on. If you're serious about future infra, then it's all here: https://github.com/purpleidea/mgmt/ https://github.com/purpleidea/mgmt/ Looking for coding help for some of the harder bits that people might wish to add, and for people to take it into production and find issues that we've missed.

lifty 2y ago
I remember seeing your presentation many years ago, at Fosdem. Very cool project and if I would have to manage classic OS deployments I would definitely give mgmt a try. That being said, I think the world is moving to more immutable systems similar to how Talos works (https://talos.dev https://talos.dev).
- karmarepellent 2y ago
  I would be hesitant to claim "the world is moving to" anything, really. Deployments that would now be called "traditional", so anything that does not run in a container but in a VM, will continue to exist for quite some time. And not only because of legacy systems that are hard to migrate to a modern platform. At my place of work there are workloads that can easily run on Kubernetes and it would be wise to do so. On the other hand there are systems that are not designed to run in a container and there is frankly no need to, because not everything needs to scale up and down or be available 100% of the time at all costs. I think configuration management systems like mgmt (or Ansible and Puppet) are here to stay.
  - lifty 2y ago
    Not disagreeing with you there; technology lingers for many years. But in terms of market share and mind share, configuration management has shrank in dominance and I suspect it will continue to do so.

nyrikki 2y ago

On a unrelated note, does anyone know the origins of the one way vs two way door analogy? In this post it is attributed to Jeff Bezos quotes, but it was popular in the Pacific North West before his rise.

pjdesno 2y ago

Something missing here in the discussion of object storage and databases is any mention of the declining importance of the file system. From the 70s through the 90s or 00s everything was file system-based, and it was just assumed that the best way to store data in a distributed system - even a globally-distributed one - was some sort of distributed file system. (e.g. Andrew File System, or research projects like OceanStore. Nowadays the file system holds applications and configuration, but applications mostly store data in databases and object stores. In distributed systems this is done almost exclusively through system-specific network connections (e.g. port 3306 to MySQL, or HTTP for S3) rather than OS-level mounting of a file system. (not counting HPC, where distributed file systems are used to preserve the developer look and feel of early non-distributed HPC systems)

__turbobrew__ 2y ago

Pushing as much down to the infra sounds like aws lambda and friends. You basically upload a zip or container and say, “just run this business code somewhere, I don’t care”. OCI bundles are basically a two day door at this point, you can build them with many tools, and run them with many other tools. It works great for stateless things, but not so great for stateful things. I guess this plays into state being persisted in object storage or DBs, this allows the application to be stateless.