8 ms·

That work is still relevant! It's just that end-users don't need to be aware of it. The position of the paper I submitted, which I basically agree with, is that

by scott_s 10d ago

That work is still relevant! It's just that end-users don't need to be aware of it. The position of the paper I submitted, which I basically agree with, is that "streaming" shouldn't need to be something end-users care about. It's something the system does based on needs. Databases already have a dataflow style architecture: that's how they implement queries. Because SQL is relational, SQL queries become dataflow execution plans. One way to think about the programming model I worked on is that it was like exposing a query plan API directly to users, instead of giving them SQL.

fmjrey 8d ago
A project to watch in that space is Rama from Red Planet Labs [0]. It pushes further the trend of destructuring databases into a very flexible streaming engine and database. It has no query engine, instead the platform offers powerful primitives to partition and distribute both data and compute, effectively destructuring transactions and query engine into microbatch and stream topologies. I think the real value of such platform is in how it can replace tons of microservices and nosql databases into a single platform with a coherent model. Replacing 1M LOC across 100 services and databases by one cluster with 10000 LOC is an operational dream come true. Also I expect LLMs limitations to force us to resolve ourselves the operational complexity of IT, as I can't imagine letting LLMs multiply the number of services and databases like PRs on github. Platforms like Rama provide the necessary constraints to channel LLM output into a coherent whole easy to operate and observe. [0] https://redplanetlabs.com/ https://redplanetlabs.com/
- scott_s 8d ago
  It is indeed interesting, and it's from Nathan Marz, who was the creator of Storm. That was the first major open source streaming platform. It predated what I consider the default open source streaming platform, Flink. Glancing through the docs, my main three reactions are: 1) It's sophisticated system which, as you say, effectively opens up the components of a database engine to be used as needed. 2) Folks will still want to eventually land their data in a "normal" data-at-rest storage format like Hive tables or Parquet. 3) Folks will still want SQL.

by scott_s 10d ago

fmjrey 8d ago

A project to watch in that space is Rama from Red Planet Labs [0]. It pushes further the trend of destructuring databases into a very flexible streaming engine and database. It has no query engine, instead the platform offers powerful primitives to partition and distribute both data and compute, effectively destructuring transactions and query engine into microbatch and stream topologies. I think the real value of such platform is in how it can replace tons of microservices and nosql databases into a single platform with a coherent model. Replacing 1M LOC across 100 services and databases by one cluster with 10000 LOC is an operational dream come true. Also I expect LLMs limitations to force us to resolve ourselves the operational complexity of IT, as I can't imagine letting LLMs multiply the number of services and databases like PRs on github. Platforms like Rama provide the necessary constraints to channel LLM output into a coherent whole easy to operate and observe. [0] https://redplanetlabs.com/ https://redplanetlabs.com/

scott_s 8d ago
It is indeed interesting, and it's from Nathan Marz, who was the creator of Storm. That was the first major open source streaming platform. It predated what I consider the default open source streaming platform, Flink. Glancing through the docs, my main three reactions are: 1) It's sophisticated system which, as you say, effectively opens up the components of a database engine to be used as needed. 2) Folks will still want to eventually land their data in a "normal" data-at-rest storage format like Hive tables or Parquet. 3) Folks will still want SQL.