13 ms·
Neki – Sharded Postgres
- wewewedxfgdf 6d agoNice work by the marketing department getting two posts on HN front page at same time.
- gk1 6d agoIt doesn’t take a marketing person to submit a link.
- vaibhav2614 6d agoPretty exciting to see a sharded Postgres product in the market. The pitch is compelling. I wonder how many teams will be able to operate sharded database setups in production as a result of this.
- gk1 6d agoFolks, in your launch posts remember to describe what you're launching and what it's for. Ideally in the opening paragraph. - Intro: Explains problem but not what is Neki or what it's for. - Why Neki: Explains alternatives and why they suck but not what is Neki or what it's for. - How does Neki work: Describes the technical components but not what is Neki or what it's for. - What you get beyond sharding: Explains how to run/deploy but not what is Neki or what it's for. - What is a platform preview + Try Neki today: 140 words on the definition of "preview" and links to get started... but not what is Neki or what it's for. Edit: The landing page has 100x more useful information up front -> https://neki.dev/ https://neki.dev/ Edit #2: They've since added a "What is Neki" section. Good reaction time.
- deleted 6d ago[deleted]
- dwb 6d agoThe landing page which is linked at the start of the second sentence of this page…? I don’t completely disagree with you, but this is mostly hypertext working as intended, is it not?
- fastball 6d agoYeah this meta commentary on HN is very confusing for me. The structure of the article is literally - short paragraph saying "we launched neki" - short paragraph saying why neki was needed - paragraphs about what neki is (with a header) That seems like a very reasonable structure for a blog post like this one. Not being able to get to the third paragraph of a blog post seems like a "you" problem, not a blog problem.
- crooked-v 6d agoThe "what is" section didn't exist when they commented.
- simonw 6d agoSadly, in 2026, you should never assume anyone will click a link.
- entrope 6d agoWhy "sadly"? The person writing a link should provide enough detail for readers to figure out whether they should follow the link. I rolled my eyes the first time I saw this entry on HN because it just said "Neki" and the (generic) target domain name. A two-word description got added, and that makes it interesting instead.
- simonw 6d agoIt's sad because I think links are the best thing about the web, and it's increasingly clear to me that many internet users either misunderstand or actively avoid them. Why do people post screenshots of an article on social media instead of linking to it? Sometimes ignorance, but often because the web is such a hostile experience these days - paywalls and subscribe banners and cookie warnings and slow pages. That's sad.
- owebmaster 6d agoIf that's what you think, why you keep posting comments linking to your blog? If you know that to be true and still add your links everywhere ("helping"), isn't this a spammer's confession?
- DrammBA 6d agoFolks, your blog post sucks at marketing. Edit: your marketing site actually has excellent marketing information. Edit #2: They've since replaced the blog post with a link to the marketing page. Good reaction time.
- ahoka 6d agoI scrolled through the landing page and even the "Get started" page and I have still no idea what is this. Is this a managed db? Something I can run myself? No idea.
- tjohnell 6d agoA C2 level diagram included in the blog post would be helpful. What are the drawbacks to using Neki aside from the obvious (new software has bugs)?
- arrowleaf 6d agoThe first thing I want to know when looking at a distributed system is how consistency is handled, ctrl+f for "consis" on both the blog post and neki.dev has zero hits. So I can only assume it will have eventual consistency with a considerable replica lag for real-time systems.
- khy 6d agoMan, I was deep into this blog post before I was able to answer the question "what is Neki?".
- imjosh-dev 6d agohttps://neki.dev/ https://neki.dev/ is a much more terse overview of neki
- 1121redblackgo 6d agoNow that's a nice release page.
- ahknight 6d agoIf I don't know what I'm reading about by the third or fourth phrase, I skip it. Empty language comes from empty minds and I don't have time for that.
- nicechianti 6d ago[dead]
- rs_rs_rs_rs_rs 6d agoWill this be open source at some point?
- samlambert 6d agoundecided right now
- ezekg 6d agoI assume it's already on your radar, but look into fair source: https://fair.io https://fair.io.
- ragall 6d agoI see no mention of foreign keys, or any other constraints, across shards. If, as I suspect, they're not implemented, it would still be useful but at the level of Spanner 10 years ago.
- aabhay 6d agoCross shard transactions are “coming soon”. So, this is just like having multiple postgres clusters.
- spongeboi 6d agoCross shard transactions are supported, but we don't have support for atomic 2pc cross shard transactions, its WIP
- ragall 6d agoAre you one of the developers ? If so, you need to add a specific page explaining in detail the guarantees that this gives. For example: point 08 says "Assign different tables or workloads to different shard groups" and point 02 says "Split hot shards as workloads grow". How do those interact ? Can a single table be split across multiple shards ? If so don't you need 2pc to enforce primary key constraints ? The intro pages just read as AI slop.
- levkk 6d agoCurious to compare your implementation to ours [1] some day (if you guys open source it). [1]: https://docs.pgdog.dev/features/sharding/2pc/ https://docs.pgdog.dev/features/sharding/2pc/
- behole 6d agoSite design is tight. I like it.
- samlambert 6d agothank you!
- arpinum 6d ago[dead]
- xnx 6d agoWhat does this compete with? Google Cloud SQL?
- jbonatakis 6d agoMultigres is the one that comes to mind
- kelp 6d agoOnce they implement sharding...
- jbonatakis 6d agoYet. But it’s built by the guy who created Vitess and cofounded PlanetScale, so while it doesn’t have sharding implemented at the moment I think it would be foolish to discount it. Plus it’s open source.
- samlambert 6d agoWe literally have the counterfactual of the old product and company before he left. We can fully discount Multigres doing anything.
- uberman123 6d agoSugu? The guy that "barely worked" and "shipped nothing at PlanetScale" according to Planetscale's CEO?
- karlmush 6d agoAnyone who has worked with Sugu knows this simply isn’t true. Sam Lambert is lying.
- samlambert 6d agoevery metric we have proves otherwise
- erulabs 6d agoI love to see this but as a heavy Vitess/MySQL user I do fear the split focus from PlanetScale. Hoping to see continued improvements on the Vitess side as well. Selfish doubts aside, congrats to Planetscale on the launch!
- imjosh-dev 6d agodon't worry, we still give a lot of love to vitess. at the end of the day they are both databases. the beauty of doing both is we can take learnings from each product and apply it to the other.
- vt_throwaway 6d ago[dead]
- kaycebasques 6d agoI am on a 12 Muni right now that is plastered with a Neki ad, and I looked up neki.dev literally 10 mins ago to see what this product is about. Did they time the release of this bus ad with the launch date?
- kelp 6d agoBus ads have been running for a little while now. A few billboards around too.
- vira28 6d agoObviously welcome more tools for Postgres and sharding but it’s a tough hill to climb against the hyper clouds. Best wishes to the team.
- samlambert 6d agoThe largest database to ever run on Aurora MySQL runs on PlanetScale, and the largest database to ever run on Aurora limitless also runs on PlanetScale.
- OrangeDelonge 6d agoWhat do you mean? Your comment reads like PlanetScale uses Aurora under the hood, you mean PlanetScale can handle bigger DB’s than Aurora?
- theanirudh 6d agoHe means that they moved from AWS to Planetscale and yes it handles much larger DBs.
- tedd4u 6d agoWe went as far as we could on GCP before PlanetScale. Spanner is amazing but also amazingly expensive. And CloudSQL, also amazing as long as you don’t need write scaling. PlanetScale (Vitess+MySQL plus their branching / deployment and monitoring tools) is just not a combination offered on GCP. And while I didn’t use Aurora, from talking to and exploring AWS it doesn’t really have this combination either.
- samlambert 6d agoIt makes me so happy to hear you had a good experience. Thank you for being a customer.
- Atreiden 6d agoThe top concern I've gotten from dev teams when proposing HA distributed postgres (e.g through RDS Aurora global) is that eventual consistency is not suitable for many workloads. Does Neki solve for this, and if so how? My understanding of CAP theorem is that this basically requires some compromises around availability, but I'm curious as to what that looks like in practice here.
- mcpherrinm 6d agoAssuming it works the same way as their MySQL product, Vitess: The usual way to run it is that you partition your db based on something like a user, so that single user gets a consistent DB, but anything cross-shard may not be. I know when I worked at Block, Cashapp was using Vitess and getting cross-shard DB writes down and functioning correctly was one of the major blockers to adoption. (though I just did tls management for vitess and didn't write any workloads on top of it, so my impression might be a bit off)
- ronfriedhaber 6d agoTherefore, Consistency guarantees are based on implementing application-specific partitioning (correctly)?
- mcpherrinm 6d agoYeah. Vitess does have some options for atomic cross-shard commits using an additional database to track two-phase commits, but that's expensive so you probably want to avoid that. https://vitess.io/docs/25.0/reference/features/distributed-transaction/ https://vitess.io/docs/25.0/reference/features/distributed-t...
- osalberger 5d agoThe consistency guarentee assuming repeatable read or higher on each shard is parallel snapshot isolation, which is technically still higher than postgres read committed. If you have serializable on each shard and you chop transactions so you have no cross shard transactions, the application is technically still serializable, it just behaves like two independent deployments of the application for different shards
- Hasz 6d agoIndependent of the product, love the branding design for this page.
- aamederen 6d agoThe first question came to my mind was, how would this compare against Citus.
- imjosh-dev 6d agoI actually wrote a bit about this on our blog! https://planetscale.com/blog/the-history-of-postgres-sharding#then-someone-put-it-in-postgres https://planetscale.com/blog/the-history-of-postgres-shardin...
- ahachete 5d agoBut it's not really accurate: > This adds some undesirable side effects, however. First, the coordinator becomes the bottleneck. Apart from being able to use any node as a coordinator (and you can load balance them to avoid having "multiple connection strings), there's a new pattern which effectively allows you to have as many coordinators as you want. They are effectively "data-less" nodes. We have devised and implemented this pattern in StackGres [1]. > Adding a shard with more resources for a noisy tenant, or many small shards for a wide shard space requires substantial manual configuration of not only the database servers themselves, but wiring them up together with Citus. Adding nodes (infrastructure) is what operators solve. In StackGres, adding new nodes means editing one/two characters from your YAML file: the integer number that represents the number of workers that you have. Shard rebalancing is fully built-into Citus as a UDF, which you can call (manually or in an automated manner) over Postgres protocol. > Managing backups is also external to Citus, so operators still need to build the proper infrastructure Agreed, but it's also solved (see distributed backups in StackGres [2]). > PgDog is a spiritual successor to PgCat, both of which improve on Citus's architecture substantially. Unsubstantiated why. I assume it's because of the assumption that a proxy model is superior than Citus. To which I have to say that Citus model is also a proxy model, where the proxy just happens to be Postgres, which unsurprisingly, speaks Postgres protocol. Sure, there are nuances that we could debate in this area and we can say that Citus is not a "pure proxy", but that doesn't lead to concluding that a proxy model is better --it's arguably not. [1]: https://stackgres.io/doc/latest/administration/sharded-cluster/citus/#query-routers https://stackgres.io/doc/latest/administration/sharded-clust... [2]: https://stackgres.io/doc/latest/administration/sharded-cluster/backup-restore/ https://stackgres.io/doc/latest/administration/sharded-clust...
- ripe 6d agoNeki is a sharded Postgres implementation. It introduces one level of indirection between the application and the Postgres shards. Found this from neki.dev.
- fastball 6d agoYou can also find that in first section of the linked post (after the intro)...
- phoenix24 6d agois this open-sourced?
- dvtkrlbs 6d agoIs this going to be open sourced? I thought this was the plan at the start. But ow I am confused.
- xyst 6d agoonly planet scale? Rookie numbers. I am looking for interdimensional and interuniversal scale. Which of you trust fund babies has a startup which is working on this problem?
- samlambert 6d agoi have been struggling to hire enough physicists
- parthdesai 6d agoThe tech might be amazing, but the CEO comes across as an asshole
- tombert 6d agoGenuine question, since I'm a little out of the loop; why do you say that?
- parthdesai 6d agohttps://x.com/samlambert/status/2098069474205470970?s=20 https://x.com/samlambert/status/2098069474205470970?s=20 https://x.com/samlambert/status/2098075451373834681?s=20 https://x.com/samlambert/status/2098075451373834681?s=20
- samlambert 6d agonothing i have said is untrue. a founder of a company leaving to compete out of greed is a bad thing and i am glad to call it out.
- margalabargala 6d agoIn case you were unaware, if someone's defense against being called an asshole is to say "well nothing I said was false" they're almost certainly an asshole. No one accused you of dishonesty.
- mbesto 6d agoDid anyone say you said anything untruthful? No, they just called you an asshole. You can both be an asshole and be right. PS - Just some advice that I learned in sales 101 early in my career...focusing on shit talking the competition generally isn't a strategy that works well.
- omnicognate 6d ago> You can both be an asshole and be right. "You're not wrong Walter, you're just an asshole." - The Dude
- xyzzy_plugh 6d agoI love how the Planetscale CEO has been shitting on multigres[0], Supabase's equivalent to Neki, and gloating about how superior Neki is, and yet Neki is not even open source! Sam, it feels like you're just playing games. It's hard to take anything you say or do seriously. Are you going to OSS this thing? Or have you decided to backpedal? Congrats on the checks notes closed launch, I guess? 0: https://github.com/multigres/multigres https://github.com/multigres/multigres
- samlambert 6d agoHow does open source have anything to do with Neki's ability to scale against a service that doesn't even do sharding.
- margalabargala 6d agoThey both are characteristics of software that people consider when deciding whether to use it. Open source being by far the more important of the two, since it means the user is more insulated by terrible corporate decisions thus making it more suitable for long term planning.
- samlambert 6d agohave you ever worked for a customer of AWS? If so you have contributed heavily to the destruction of commercial open source.
- xyzzy_plugh 6d agoBut your customers are, of course, somehow different? What point are you even trying to make? Commerical open source is fine. It's nobody else's fault if you raised too much to make it work sustainably.
- samlambert 6d agoMy customers aren't pretending it's a bad thing that Neki isn't open source.
- karuppiah7890 6d agoIt's not Open Source Software. So, I'm not even interested. Lol
- samlambert 6d agolol are you an amazon customer?
- overfeed 6d agolol are you an amazon wannabe?
- samlambert 6d agoyes
- deleted 6d ago[deleted]
- devy 6d agoHow does Neki compare to https://pgdog.dev/ https://pgdog.dev/?
- samlambert 6d agoit works and does sharding
- levkk 6d agoSame here! Except: - PgDog is open source - It runs anywhere, not just your cloud - It's built for Postgres, not a MySQL pivot
- samlambert 6d agowhat does a mysql pivot mean? like literally what do you think we have done here technically that could be considered a mysql pivot?
- hanspagel 6d agoI need this for my side project that has zero users, but I wonder if Neki will scale well enough.
- _kidlike 6d agocan we choose read-after-write consistency at query time, if needed?
- _1tan 6d agoCool, but we're on MySQL - anything similar available? Percona XtraDB?
- IceWreck 6d agoVitess/Planetscale? Neki is essentially a version of Vitess for Postgres fom the same guys.
- kelp 6d agoMany of the engineers who built Neki also worked on Vitess for years. But it's a ground up new implementation designed to specifically target Postgres.
- vt_throwaway 6d ago[dead]
- tedd4u 6d agoThat is what the “regular” PlanetScale is all about. It uses the Vitess open source project to do write scaling through shading. It is very powerful and customizable and stable. It was built to run YouTube. Former happy user here.
- MrDrMcCoy 6d agoMariaDB Galera or Vitess.
- dveeden2 6d agoTiDB: Distributed and MySQL compatible
- fsuts 6d agoKinda ironic that Planetscale built their company on open source Vitess that google built And now planetscale have made a Vitess like equivalent for Postgres which they have made proprietary
- hiyer 6d agoThey have made it proprietary so that providers like AWS can't build their own service with it, thus eating into PS's income. Google is big enough to survive open-sourcing Vitess, k8s, angular, etc. Planetscale isn't.
- samlambert 6d agocorrect. AWS do anti-competitive stuff to us all the time we can't make it any easier for them.
- Onavo 5d agoWhat a convenient excuse
- figmert 6d agoGoogle, Azure, and AWS already have their highly available PostgreSQL alternatives.
- ksec 5d ago>Kinda ironic that Planetscale built their company on open source Vitess that google built That Youtube built. I know at the time Youtube was already part of Google but I don't think of Vitess as a Google Stack.
- manlymuppet 6d agoI think this will become the new standard.
- eile23 6d agoHow are cross-shard joins and transactions handled? Are there query patterns that would require application changes despite using the same drivers and ORM?
- imjosh-dev 5d agocross shard joins break down into single shard selects, and are joined in the routers. There are some query patterns that are currently unsupported, but we are working on adding support for them. The biggest thing to keep in mind when running on a sharded database of any kind is decreasing the number of cross shard queries.
- Tepix 6d agoSo apparently the neki router uses many parts of Postgres. But is it open source itself? Apparently not.
- kelp 5d agoThe neki router shares no code with Postgres. However we have a whole team working to improve Postgres and we always prefer to contribute those efforts to the Postgres community.
- kamikazechaser 6d agoNot even freeware/shareware?
- Yannik_Sc 6d agoI love the offer which is running on the "PlanetScale Metal". They claim to provide unlimited IOPS (I/O operations Per Second). I would like to get some of these drives myself. Sounds like a technical miracle. On the other hand claiming such miracles makes me wonder which other aspects in this are are actually a working miracle or rather a mythos/false marketing claim.
- halilBB 6d ago[flagged]
- fatata123 6d ago[dead]
- klaussilveira 6d agoHow does it deal with big tables (8B+ rows)? Does it split the table onto multiple shards or each shard only gets full tables?
- imjosh-dev 5d agoCompletely up to the user. Neki doesnt automatically shard, its all controlled by the data topology (https://planetscale.com/blog/what-is-a-data-topology https://planetscale.com/blog/what-is-a-data-topology).
- SkiFire13 5d agoThat page doesn't mention how joins between tables using different split keys work, or how indexes work.
- haul_up 5d agoA launch post should answer 'what is this' in the first sentence. Digging to paragraph three is a failure, not a 'you' problem.