5 ms·

Neki – Sharded Postgres by the team behind Vitess

251 points by thdxr 1y ago

purpleidea 1y ago
Maybe make the announcement after you've actually released code? Today I'm announcing I've cured cancer. Well not yet, but coming soon hopefully!
- tpetry 1y ago
  More like: We‘re announcing to work on a cure in cancer! Isn‘t that aweslme? No, we dont have anything yet. But we started working on it. ETA? When its done. We dont share roadmaps with outsiders.
- crowcroft 1y ago
  Probably want to try front run comms with Supabase getting into this as well.
shoeb00m 1y ago
Looks like there is two ongoing vitess for postgres projects. Hopefully this competition leads to a better postgres ecosystem. https://supabase.com/blog/multigres-vitess-for-postgres https://supabase.com/blog/multigres-vitess-for-postgres
- qaq 1y ago
  Supabase also working on OrioleDB
  - eatonphil 1y ago
    OrioleDB is not about sharding, it's about the storage layer.
    - qaq 1y ago
      I did not claim OrioleDB is about sharding. It was just an observation that Supabase is contributing to Postgres ecosystem through multiple projects.
      - selfhosttoday 1y ago
        they likely said that because the context is "vitess for postgres projects" and OrioleDB is not "vitess for postgres"
- n2d4 1y ago
  There is also pgdog by the author of pgcat: https://pgdog.dev https://pgdog.dev
- dangoodmanUT 1y ago
  It gets more spicy when you realize the founder of vitess, also the founder of planet scale, left planet scale to build this at supabase
  - samlambert 1y ago
    he left PlanetScale 4 years ago.
hubertzhang 1y ago
Does Neki still need sharding key in query, just like Citus?
- rockwotj 1y ago
  If it’s like vitess then no, but IIRC you get relaxed consistency across shards
  - n_u 1y ago
    Regarding cross-shard consistency, there's a doc here[1] on the options. In short you can just send the writes to all replicas, you can disallow cross-replica transactions (read and write), use two-phase commit to perform a distributed transaction or use their own hybrid approach that they set as the default. [1] https://vitess.io/docs/20.0/user-guides/configuration-advanced/shard-isolation-atomicity/#method-1--the-naive-way https://vitess.io/docs/20.0/user-guides/configuration-advanc...
ozgrakkurt 1y ago
Is anyone working on replacing postgres? Feels like it might be very useful since a lot of new technologies came out since spinning disks.
- erpellan 1y ago
  If you look at the changes that have been made to Postgres, and continue to be made, the answer is yes. The Postgres team is working on replacing Postgres. With even better Postgres.
  - bddicken 1y ago
    The Postgres team incorporating io_uring into PG 18 is a good example of this: https://pganalyze.com/blog/postgres-18-async-io https://pganalyze.com/blog/postgres-18-async-io.
  - zozbot234 1y ago
    inb4 "It's the best Postgres we've ever made!"
- eatonphil 1y ago
  Who isn't? Cockroach rewrote Postgres in Go. CedarDB rewrote Postgres in C++. And then to lesser degrees you've got Yugabyte, AlloyDB, and Aurora DSQL (and certainly more I'm forgetting) that only replace parts of Postgres.
  - vladich 1y ago
    Both Cockroach and CedarDB didn't rewrite anything, they built stuff from scratch. Just used the same client protocol. There are a bunch of other unrelated databases using Postgres protocol btw.
    - eatonphil 1y ago
      I'm not talking about speaking the protocol. I'm talking about trying as hard as they can to be as indistinguishable from Postgres (to a non-operations user) as they can. And that list is very small.
atombender 1y ago
This is exciting. The announcement says it will be open source. I really hope that this includes a functionally complete control plane so you realistically self-host. I looked Neon recently, and it appears that it's designed as a SaaS product from the outset; while it is technically possible to self-host the individual components of the architecture, it does not look trivial, in large part because the control plane is closed source (and probably extremely specific to Neon's SaaS operations).
- benjiro 1y ago
  Your probably better off with the original flavor (the guy that made Vitesse) https://multigres.com/ https://multigres.com/ He is making a open source version of porting Vitesse to Postgres.
  - bddicken 1y ago
    Why? Neki is built by the the engineers who have built, maintain, and operate massive-scale Vitess databases.
    - kelp 1y ago
      To add to this, if you look at the top 10 committers to Vitess over the last 12 months, 8 of them are helping with Neki in one way or another: https://github.com/vitessio/vitess/graphs/contributors?from=8%2F10%2F2024 https://github.com/vitessio/vitess/graphs/contributors?from=...
    - pjjpo 1y ago
      Multigres is made by the guy that made Vitess, Sugu, before it became a startup. Doesn't mean it will be better, but I think it's why people have high hopes for both products.
      - andrew_mason1 1y ago
        there are also a significant number of previous vitess maintainers that have gone to supabase to work on it with him, so there's that. it is not at all accurate to say that neki is made by "the team that brought you vitess" and that multigres is somehow not
kyrra 1y ago
They did a interview 2 weeks ago about this on the changelog podcast with Sugu Sougoumarane, the co-creator of Vitess, who is at Supabase now. Watching the competition year will be interesting https://youtu.be/y1aq8RsnJeI https://youtu.be/y1aq8RsnJeI
- josevalerio 1y ago
  Just FYI that interview is for Multigres, a sharded Postgres in development by Supabase. Sugu hasn’t been at Planetscale for a few years https://github.com/multigres/multigres https://github.com/multigres/multigres
Reubend 1y ago
Very cool, and I'm looking forward to using this. But unfortunately it seems like it's not actually released yet? I'm a little surprised to hear that PlanetScale is doing the work to make this considering I thought their entire system was based on Vitess. Maybe the demand for Postgres compatible DBs is so high nowadays that they need to offer compatibility for customers that don't want to port their apps to MySQL's syntax?
abrkn 1y ago
I love this website!
saurik 1y ago
I wonder how this compares to YugabyteDB?
- v5v3 1y ago
  Yugabyte is a 'postgresql compatible' database. These are forks or extensions of Postgresql.
  - saurik 1y ago
    YugabyteDB is also a fork of PostgreSQL. Last year, they got around to rebasing from 11 to 15, and spent some time working on better isolating their changes to make that easier going forward. https://www.yugabyte.com/blog/yugabytedb-moves-beyond-postgresql-11/ https://www.yugabyte.com/blog/yugabytedb-moves-beyond-postgr... > When we set out to make YugabyteDB Postgres-compatible, we took a fork of Postgres, and modified all of the operations that use shared memory or storage to instead talk to our LSM- and Raft-based distributed storage and transaction layer.

251 points by thdxr 1y ago

Maybe make the announcement after you've actually released code? Today I'm announcing I've cured cancer. Well not yet, but coming soon hopefully!

tpetry 1y ago
More like: We‘re announcing to work on a cure in cancer! Isn‘t that aweslme? No, we dont have anything yet. But we started working on it. ETA? When its done. We dont share roadmaps with outsiders.
crowcroft 1y ago
Probably want to try front run comms with Supabase getting into this as well.

shoeb00m 1y ago

Looks like there is two ongoing vitess for postgres projects. Hopefully this competition leads to a better postgres ecosystem. https://supabase.com/blog/multigres-vitess-for-postgres https://supabase.com/blog/multigres-vitess-for-postgres

qaq 1y ago
Supabase also working on OrioleDB
- eatonphil 1y ago
  OrioleDB is not about sharding, it's about the storage layer.
  - qaq 1y ago
    I did not claim OrioleDB is about sharding. It was just an observation that Supabase is contributing to Postgres ecosystem through multiple projects.
    - selfhosttoday 1y ago
      they likely said that because the context is "vitess for postgres projects" and OrioleDB is not "vitess for postgres"
n2d4 1y ago
There is also pgdog by the author of pgcat: https://pgdog.dev https://pgdog.dev
dangoodmanUT 1y ago
It gets more spicy when you realize the founder of vitess, also the founder of planet scale, left planet scale to build this at supabase
- samlambert 1y ago
  he left PlanetScale 4 years ago.

hubertzhang 1y ago

Does Neki still need sharding key in query, just like Citus?

rockwotj 1y ago
If it’s like vitess then no, but IIRC you get relaxed consistency across shards
- n_u 1y ago
  Regarding cross-shard consistency, there's a doc here[1] on the options. In short you can just send the writes to all replicas, you can disallow cross-replica transactions (read and write), use two-phase commit to perform a distributed transaction or use their own hybrid approach that they set as the default. [1] https://vitess.io/docs/20.0/user-guides/configuration-advanced/shard-isolation-atomicity/#method-1--the-naive-way https://vitess.io/docs/20.0/user-guides/configuration-advanc...

ozgrakkurt 1y ago

Is anyone working on replacing postgres? Feels like it might be very useful since a lot of new technologies came out since spinning disks.

erpellan 1y ago
If you look at the changes that have been made to Postgres, and continue to be made, the answer is yes. The Postgres team is working on replacing Postgres. With even better Postgres.
- bddicken 1y ago
  The Postgres team incorporating io_uring into PG 18 is a good example of this: https://pganalyze.com/blog/postgres-18-async-io https://pganalyze.com/blog/postgres-18-async-io.
- zozbot234 1y ago
  inb4 "It's the best Postgres we've ever made!"
eatonphil 1y ago
Who isn't? Cockroach rewrote Postgres in Go. CedarDB rewrote Postgres in C++. And then to lesser degrees you've got Yugabyte, AlloyDB, and Aurora DSQL (and certainly more I'm forgetting) that only replace parts of Postgres.
- vladich 1y ago
  Both Cockroach and CedarDB didn't rewrite anything, they built stuff from scratch. Just used the same client protocol. There are a bunch of other unrelated databases using Postgres protocol btw.
  - eatonphil 1y ago
    I'm not talking about speaking the protocol. I'm talking about trying as hard as they can to be as indistinguishable from Postgres (to a non-operations user) as they can. And that list is very small.

atombender 1y ago

This is exciting. The announcement says it will be open source. I really hope that this includes a functionally complete control plane so you realistically self-host. I looked Neon recently, and it appears that it's designed as a SaaS product from the outset; while it is technically possible to self-host the individual components of the architecture, it does not look trivial, in large part because the control plane is closed source (and probably extremely specific to Neon's SaaS operations).

benjiro 1y ago
Your probably better off with the original flavor (the guy that made Vitesse) https://multigres.com/ https://multigres.com/ He is making a open source version of porting Vitesse to Postgres.
- bddicken 1y ago
  Why? Neki is built by the the engineers who have built, maintain, and operate massive-scale Vitess databases.
  - kelp 1y ago
    To add to this, if you look at the top 10 committers to Vitess over the last 12 months, 8 of them are helping with Neki in one way or another: https://github.com/vitessio/vitess/graphs/contributors?from=8%2F10%2F2024 https://github.com/vitessio/vitess/graphs/contributors?from=...
  - pjjpo 1y ago
    Multigres is made by the guy that made Vitess, Sugu, before it became a startup. Doesn't mean it will be better, but I think it's why people have high hopes for both products.
    - andrew_mason1 1y ago
      there are also a significant number of previous vitess maintainers that have gone to supabase to work on it with him, so there's that. it is not at all accurate to say that neki is made by "the team that brought you vitess" and that multigres is somehow not

kyrra 1y ago

They did a interview 2 weeks ago about this on the changelog podcast with Sugu Sougoumarane, the co-creator of Vitess, who is at Supabase now. Watching the competition year will be interesting https://youtu.be/y1aq8RsnJeI https://youtu.be/y1aq8RsnJeI

josevalerio 1y ago
Just FYI that interview is for Multigres, a sharded Postgres in development by Supabase. Sugu hasn’t been at Planetscale for a few years https://github.com/multigres/multigres https://github.com/multigres/multigres

Reubend 1y ago

Very cool, and I'm looking forward to using this. But unfortunately it seems like it's not actually released yet? I'm a little surprised to hear that PlanetScale is doing the work to make this considering I thought their entire system was based on Vitess. Maybe the demand for Postgres compatible DBs is so high nowadays that they need to offer compatibility for customers that don't want to port their apps to MySQL's syntax?

abrkn 1y ago

I love this website!

saurik 1y ago

I wonder how this compares to YugabyteDB?

v5v3 1y ago
Yugabyte is a 'postgresql compatible' database. These are forks or extensions of Postgresql.
- saurik 1y ago
  YugabyteDB is also a fork of PostgreSQL. Last year, they got around to rebasing from 11 to 15, and spent some time working on better isolating their changes to make that easier going forward. https://www.yugabyte.com/blog/yugabytedb-moves-beyond-postgresql-11/ https://www.yugabyte.com/blog/yugabytedb-moves-beyond-postgr... > When we set out to make YugabyteDB Postgres-compatible, we took a fork of Postgres, and modified all of the operations that use shared memory or storage to instead talk to our LSM- and Raft-based distributed storage and transaction layer.