7 ms·

It's Time to Stop Building KV Databases

65 points by foldU 1y ago

tyzoid 1y ago
Sometimes a KV datastore is the right abstraction, though. Caching is an excellent example, but also for distributed session storage, configuration management, nonce enforcement, etc.
- esafak 1y ago
  KV stores can be used to build distributed relational databases.
  - eru 1y ago
    KV stores can be used to build relational databases in general. Whether it's a good idea depends on your goal, and what alternative buildings blocks you have available. (Eg if you are building your distributed relational database to run on top of lots of computers with spinning hard disks, you might want to expose some more characteristics of the hard disk directly to your database, so you can manage them; instead of trying to hide them behind an abstraction.)
fud101 1y ago
Is this a good take? I'm just a below average dev and trying to figure it out.
- treyd 1y ago
  Like most articles that make strong assertive statements like this, it's an oversimplification. Every tool has its place. The author clearly wants to use SQL, and seems to have a problem that would benefit from it, so they should use a SQL DB and not try to use a KV DB.
  - setr 1y ago
    He’s basically just asking for SQL without the query planner because it’s obnoxious and how often do you have unconstrained arbitrary queries hitting the database that you don’t have the chance to vet anyways? The database is hidden behind applications 100% of the time; why is this a predominant design use case KVs give you this behavior, they just drop everything else with it
    - nine_k 1y ago
      You usually want a query planner, else you'll end up writing a query planner yourself to produce efficient queries. What is sometimes needed is query plan stability, lack of surprises, and influencing the planner. This is very attainable in the existing SQL databases and is a core feature of the older ones, like Oracle.
      - setr 1y ago
        I’m not really talking about eliminating the query compilation step — that’s always useful. The dynamic query compilation however — the planner executing on every submitted query — is generally less so but fundamental to RDBMS goals. Query plan stability is the point — if you’re stabilizing the query planner, then you’re explicitly opting out of its dynamic capabilities. You don’t need a planner, you need a query compiler (in that it’s one and done, and you can even allow the thing the luxury of having actual time to optimize) For proper data warehouses with multiple applications talking to it, planning is probably more useful than not. For the common modern situation with small/medium sized applications with a 1:1 relationship to their database, I’ve found it fairly rare that my data distribution changes significantly over time — and where it does, a lot more work needs to happen around it anyways. Hints are the effective solution but they’re opt-in, weird non-standard extensions to the language and intrinsically tied to the planning engine; the big issue though is that they mainly exist to coerce/override the heuristics, which mainly fall over because planners are built to re-execute on every query with no real time to optimize/explore properly (you’ll give a C++ codebase hours to explore, but the RDBMS is granted mere milliseconds — a great injustice)
- foldU 1y ago
  Yes, it’s good
- technick 1y ago
  Very opinionated clickbait if you ask me.
- lmm 1y ago
  Is he right that this imaginary future product might be a better solution than current KV datastores? Maybe - he makes a good case, it certainly sounds worth pursuing, and he can point to a case where it sort of works well already (FoundationDB). Doen that imply that you should give up on KV datastores today, when this product category he's asking for barely exists? No, obviously not.
- MrLeap 1y ago
  It's an opinion. IMO key value stores tend to live in the space between a third normal form ultra relational UML diagram database like the college textbooks assure you exist and a high chaos cowboy document storage system like mongodb. They enable you to make a lot of things up as you go and iterate on your design. I like them because they remove a lot of ceremony around letting me get on with persisting things without having to ALTER TABLE or CREATE TABLE and all that entails. At the same time, they're constrained and often organized in a way that storing big ol' json blobs aren't. I like them for doing multiplayer gamedev things.
  - jiggawatts 1y ago
    > ALTER TABLE or CREATE TABLE Fundamentally this isn't a theoretical limitation of the relational model, instead it is a historical artefact that "doesn't have to be that way". Systems like Kubernetes or Azure Resource Manager show how this ought to have been implemented: Declarative resource definitions via an API with idempotency as a core paradigm. I.e.: Instead of the web developer having to figure out the delta change between two schema versions, they should just "declare" that they want some specific schema. This could happen with every code deployment, with most such changes doing nothing. If the schema does change, then it's up to the database engine to figure out how to make it happen, much like how a Kubernetes cluster reconciles the cluster state with a deployment definition. KV stores become popular because they do this implicitly, with no explicit schema at all. Relational databases (or even KV stores!) could instead use an explicit schema with automatic reconciliation instead of manual delta changes implemented by developers. TL;DR: The tooling is bad. KV is an over-reaction to this, but instead we just needed better tools.
    - Izkata 1y ago
      Django (python web framework) migrations have worked similar to this for well over a decade: You modify the models, then generate the migrations from the models and commit those. The changes aren't generated on the fly at runtime because it can prompt you for things it suspects or can't figure out, for example if you rename a column the naive way would be an add+delete, which erases data. If the types are the same, it checks whether you wanted that or a rename, and generates the appropriate migration.
- deleted 1y ago
  [deleted]
- notfed 1y ago
  Never believe that any tool is good or bad. That's always going to be a generalization, and therefore wrong. Learn as many tools as possible, and know which use cases they're good at and which use cases they're bad at. If someone implies the tool is bad for all use cases, know that we all live in our own bubbles and are ignorant about the plethora of other use cases that exist in the world.
  - eru 1y ago
    > Never believe that any tool is good or bad. That's always going to be a generalization, and therefore wrong. Some generalisations are close enough to true to be worth it. Eg I'm fairly confident to generalise and say that for most beginners in 2025 picking Python is a better choice than PHP or Cobol. Of course, you can come up with some contrived scenarios where the beginner would be better served with Cobol.
    - gregjor 1y ago
      > Eg I'm fairly confident to generalise and say that for most beginners in 2025 picking Python is a better choice than PHP or Cobol. Limiting choices to “pick one” seems contrived. Beginners should learn to program and think like programmers, which means learning multiple languages and tools. Programming languages have far more in common with each other than not and the sooner a beginner thinks of themselves as a problem solver rather than a Python programmer the better. > Of course, you can come up with some contrived scenarios where the beginner would be better served with Cobol. I don’t think looking at job opportunities and pay qualify as “contrived scenarios.” PHP has a huge footprint in web applications and every beginner steered away from PHP to the saturated Python world forgoes a lot of opportunity. And again nothing prevents learning both. COBOL as usual gets trotted out as the dinosaur, but right now in the current tech job apocalypse knowing COBOL would get a lot more job offers than knowing Python.
      - eru 1y ago
        Thanks for illustrating my point.
Spivak 1y ago
Finally! Someone else reaching the conclusion that the query planner is really annoying and for most queries I would just like to skip it. I don't want the dynamic nature of the planner. I don't want to send SQL over the wire, I want to send the already completed plan that I either generated or wrote by hand. So many annoying performance bugs are because the planner did the slow thing. Just let me write/adjust it.
- theamk 1y ago
  All I want is K-V store with indexes. Let database enforce serialization format (JSON, BSON, MessagePack, protobuf.. anything really) + create and maintain indices, using this fancy crash-proof logic it has. That'll cover 95% of all my database needs. (OP also asks for row-based layout, types, and non-trivial language. I think those parts are entirely optional)
  - porridgeraisin 1y ago
    +1. Indexes, triggers (very good abstraction covering everything from computed fields to dependent fields), transactions.
  - rapsey 1y ago
    Use foundationdb then
  - nine_k 1y ago
    A K-V store by construction has an index (over K). But you seem to also want referential integrity and transactions, not small features, limiting the implementation quite a lot. You can attain what you desire by using an RDBMS, and having all tables with one key column, and a TEXT column with your serialized non-key fields; it's going to be a fun approximation of 6NF. Realistically, you can have all joinable columns as normal columns, indexed as you desire, and the rest of the columns as a serialized blob. When you want high parallelism for guaranteed independent segments of data, use sharding.
    - mr_toad 1y ago
mjevans 1y ago
K:V - maps / dictionaries can be the correct tool for some jobs. I think I'd prefer to stop calling _large_ resources that are only K:V a 'database' though. A 'database' shouldn't require SQL, but a distributed filesystem, however similar, isn't quite a database.
- tptacek 1y ago
  Rebuttal: a filesystem is a database.
  - OccamsMirror 1y ago
    I was immediately offended by your rebuttal and gave it some thought. It is an interesting definitional boundary. Perhaps the distinction is more pragmatic than fundamentally technical. We typically use the term "database" to describe systems designed primarily for structured data management with query capabilities, while filesystems optimize for hierarchical storage of opaque binary objects. Therefore a KV is not a database either.
    - Kinrany 1y ago
      Does that make Postgres not a database too? It can store binary blobs
      - necovek 1y ago
        There is more to the above comment than just "opaque binary objects".
    - eru 1y ago
      Filesystems are more like a graph database than a pure key value store. I think 'database' is a term with multiple (related) meanings depending on context. Another example is the term 'colour'. Depending on context, it sometimes makes sense to call black and white and grey 'colours', and sometimes it's better to treat them as something else.
    - 1y ago
stevefan1999 1y ago
I sort of disagree. KV databases are so fundamental, that they are considered one of the most foundational tech to any advanced database management system. Think of KV databases as a persistent associative mapping/hash map that needs to store data in a safe and secure way, then we can build advanced stuff on top of it. Take TiDB for example, it is a distributed database based on MySQL (its own query language can be considered as a subset of MySQL), but actually most of the heavylifting is handled by TiKV, which is a distributed KV datastore with Raft distributed consensus. And then SurrealDB also leveraged TiKV to build their own graph-document hybrid database product...as one of the data transport. P.S.: used to be a contributor for SurrealDB.
- mrlongroots 1y ago
  Deepseek just used FoundationDB to build a parallel filesystem. Parallel filesystem are a big deal -- their number, including proprietary ones, is probably in single digits.
  - metadat 1y ago
    Parallel Filesystems aren't a new or novel concept, and there have been lots of implementations. The first one I encountered was DrFTPD circa 2004. But these days, any object storage system qualifies because they all support varying replication schemes and reading from any valid in-sync replica.
    - funcDropShadow 1y ago
      Object stores are not filesystems. They have a paths, but they are not hierarchical.
      - em-bee 1y ago
        unless it's a hierarchical object store, like the one i am using.
        1y ago
exabrial 1y ago
Honestly though, why not just take the existing SQL standard and trim it down then?
rockwotj 1y ago
I do like this approach in theory, but agreed I don't think that the devex has been solved and I don't know how many people would value a different approach as SQL feels like king in many places... FWIW it's kind of the approach that IndexedDB in the web takes, however that the API is quite bad IMO. Or maybe there is a higher level DSL that you could apply to create query plans (something like MongoDB aggregation pipelines maybe?), but it quickly becomes basically the same as SQL.
- osigurdson 1y ago
  >> SQL feels like king in many places I think it is because most people can make something work with SQL.
- mike_hearn 1y ago
  The author doesn't know it, but he's asking for Permazen - exactly what you're asking for: https://github.com/permazen/permazen/blob/master/README.md https://github.com/permazen/permazen/blob/master/README.md It's a bit like the record layer in FoundationDB but more advanced. You specify query plans manually, so you can't accidentally forget an index for example.
DidYaWipe 1y ago
Yeah! Then I can stop grousing about yet another grammatical peeve: It's "key/value" pairs, not "key-value" pairs (which would mean pairs of key values).
sroussey 1y ago
All you need is MemcacheD

65 points by foldU 1y ago

Sometimes a KV datastore is the right abstraction, though. Caching is an excellent example, but also for distributed session storage, configuration management, nonce enforcement, etc.

esafak 1y ago
KV stores can be used to build distributed relational databases.
- eru 1y ago
  KV stores can be used to build relational databases in general. Whether it's a good idea depends on your goal, and what alternative buildings blocks you have available. (Eg if you are building your distributed relational database to run on top of lots of computers with spinning hard disks, you might want to expose some more characteristics of the hard disk directly to your database, so you can manage them; instead of trying to hide them behind an abstraction.)

fud101 1y ago

Is this a good take? I'm just a below average dev and trying to figure it out.

treyd 1y ago
Like most articles that make strong assertive statements like this, it's an oversimplification. Every tool has its place. The author clearly wants to use SQL, and seems to have a problem that would benefit from it, so they should use a SQL DB and not try to use a KV DB.
- setr 1y ago
  He’s basically just asking for SQL without the query planner because it’s obnoxious and how often do you have unconstrained arbitrary queries hitting the database that you don’t have the chance to vet anyways? The database is hidden behind applications 100% of the time; why is this a predominant design use case KVs give you this behavior, they just drop everything else with it
  - nine_k 1y ago
    You usually want a query planner, else you'll end up writing a query planner yourself to produce efficient queries. What is sometimes needed is query plan stability, lack of surprises, and influencing the planner. This is very attainable in the existing SQL databases and is a core feature of the older ones, like Oracle.
    - setr 1y ago
      I’m not really talking about eliminating the query compilation step — that’s always useful. The dynamic query compilation however — the planner executing on every submitted query — is generally less so but fundamental to RDBMS goals. Query plan stability is the point — if you’re stabilizing the query planner, then you’re explicitly opting out of its dynamic capabilities. You don’t need a planner, you need a query compiler (in that it’s one and done, and you can even allow the thing the luxury of having actual time to optimize) For proper data warehouses with multiple applications talking to it, planning is probably more useful than not. For the common modern situation with small/medium sized applications with a 1:1 relationship to their database, I’ve found it fairly rare that my data distribution changes significantly over time — and where it does, a lot more work needs to happen around it anyways. Hints are the effective solution but they’re opt-in, weird non-standard extensions to the language and intrinsically tied to the planning engine; the big issue though is that they mainly exist to coerce/override the heuristics, which mainly fall over because planners are built to re-execute on every query with no real time to optimize/explore properly (you’ll give a C++ codebase hours to explore, but the RDBMS is granted mere milliseconds — a great injustice)
foldU 1y ago
Yes, it’s good
technick 1y ago
Very opinionated clickbait if you ask me.
lmm 1y ago
Is he right that this imaginary future product might be a better solution than current KV datastores? Maybe - he makes a good case, it certainly sounds worth pursuing, and he can point to a case where it sort of works well already (FoundationDB). Doen that imply that you should give up on KV datastores today, when this product category he's asking for barely exists? No, obviously not.
MrLeap 1y ago
It's an opinion. IMO key value stores tend to live in the space between a third normal form ultra relational UML diagram database like the college textbooks assure you exist and a high chaos cowboy document storage system like mongodb. They enable you to make a lot of things up as you go and iterate on your design. I like them because they remove a lot of ceremony around letting me get on with persisting things without having to ALTER TABLE or CREATE TABLE and all that entails. At the same time, they're constrained and often organized in a way that storing big ol' json blobs aren't. I like them for doing multiplayer gamedev things.
- jiggawatts 1y ago
  > ALTER TABLE or CREATE TABLE Fundamentally this isn't a theoretical limitation of the relational model, instead it is a historical artefact that "doesn't have to be that way". Systems like Kubernetes or Azure Resource Manager show how this ought to have been implemented: Declarative resource definitions via an API with idempotency as a core paradigm. I.e.: Instead of the web developer having to figure out the delta change between two schema versions, they should just "declare" that they want some specific schema. This could happen with every code deployment, with most such changes doing nothing. If the schema does change, then it's up to the database engine to figure out how to make it happen, much like how a Kubernetes cluster reconciles the cluster state with a deployment definition. KV stores become popular because they do this implicitly, with no explicit schema at all. Relational databases (or even KV stores!) could instead use an explicit schema with automatic reconciliation instead of manual delta changes implemented by developers. TL;DR: The tooling is bad. KV is an over-reaction to this, but instead we just needed better tools.
  - Izkata 1y ago
    Django (python web framework) migrations have worked similar to this for well over a decade: You modify the models, then generate the migrations from the models and commit those. The changes aren't generated on the fly at runtime because it can prompt you for things it suspects or can't figure out, for example if you rename a column the naive way would be an add+delete, which erases data. If the types are the same, it checks whether you wanted that or a rename, and generates the appropriate migration.
deleted 1y ago
[deleted]
notfed 1y ago
Never believe that any tool is good or bad. That's always going to be a generalization, and therefore wrong. Learn as many tools as possible, and know which use cases they're good at and which use cases they're bad at. If someone implies the tool is bad for all use cases, know that we all live in our own bubbles and are ignorant about the plethora of other use cases that exist in the world.
- eru 1y ago
  > Never believe that any tool is good or bad. That's always going to be a generalization, and therefore wrong. Some generalisations are close enough to true to be worth it. Eg I'm fairly confident to generalise and say that for most beginners in 2025 picking Python is a better choice than PHP or Cobol. Of course, you can come up with some contrived scenarios where the beginner would be better served with Cobol.
  - gregjor 1y ago
    > Eg I'm fairly confident to generalise and say that for most beginners in 2025 picking Python is a better choice than PHP or Cobol. Limiting choices to “pick one” seems contrived. Beginners should learn to program and think like programmers, which means learning multiple languages and tools. Programming languages have far more in common with each other than not and the sooner a beginner thinks of themselves as a problem solver rather than a Python programmer the better. > Of course, you can come up with some contrived scenarios where the beginner would be better served with Cobol. I don’t think looking at job opportunities and pay qualify as “contrived scenarios.” PHP has a huge footprint in web applications and every beginner steered away from PHP to the saturated Python world forgoes a lot of opportunity. And again nothing prevents learning both. COBOL as usual gets trotted out as the dinosaur, but right now in the current tech job apocalypse knowing COBOL would get a lot more job offers than knowing Python.
    - eru 1y ago
      Thanks for illustrating my point.

Spivak 1y ago

Finally! Someone else reaching the conclusion that the query planner is really annoying and for most queries I would just like to skip it. I don't want the dynamic nature of the planner. I don't want to send SQL over the wire, I want to send the already completed plan that I either generated or wrote by hand. So many annoying performance bugs are because the planner did the slow thing. Just let me write/adjust it.

theamk 1y ago
All I want is K-V store with indexes. Let database enforce serialization format (JSON, BSON, MessagePack, protobuf.. anything really) + create and maintain indices, using this fancy crash-proof logic it has. That'll cover 95% of all my database needs. (OP also asks for row-based layout, types, and non-trivial language. I think those parts are entirely optional)
- porridgeraisin 1y ago
  +1. Indexes, triggers (very good abstraction covering everything from computed fields to dependent fields), transactions.
- rapsey 1y ago
  Use foundationdb then
- nine_k 1y ago
  A K-V store by construction has an index (over K). But you seem to also want referential integrity and transactions, not small features, limiting the implementation quite a lot. You can attain what you desire by using an RDBMS, and having all tables with one key column, and a TEXT column with your serialized non-key fields; it's going to be a fun approximation of 6NF. Realistically, you can have all joinable columns as normal columns, indexed as you desire, and the rest of the columns as a serialized blob. When you want high parallelism for guaranteed independent segments of data, use sharding.
  - mr_toad 1y ago

mjevans 1y ago

K:V - maps / dictionaries can be the correct tool for some jobs. I think I'd prefer to stop calling _large_ resources that are only K:V a 'database' though. A 'database' shouldn't require SQL, but a distributed filesystem, however similar, isn't quite a database.

tptacek 1y ago
Rebuttal: a filesystem is a database.
- OccamsMirror 1y ago
  I was immediately offended by your rebuttal and gave it some thought. It is an interesting definitional boundary. Perhaps the distinction is more pragmatic than fundamentally technical. We typically use the term "database" to describe systems designed primarily for structured data management with query capabilities, while filesystems optimize for hierarchical storage of opaque binary objects. Therefore a KV is not a database either.
  - Kinrany 1y ago
    Does that make Postgres not a database too? It can store binary blobs
    - necovek 1y ago
      There is more to the above comment than just "opaque binary objects".
  - eru 1y ago
    Filesystems are more like a graph database than a pure key value store. I think 'database' is a term with multiple (related) meanings depending on context. Another example is the term 'colour'. Depending on context, it sometimes makes sense to call black and white and grey 'colours', and sometimes it's better to treat them as something else.
  - 1y ago

stevefan1999 1y ago

I sort of disagree. KV databases are so fundamental, that they are considered one of the most foundational tech to any advanced database management system. Think of KV databases as a persistent associative mapping/hash map that needs to store data in a safe and secure way, then we can build advanced stuff on top of it. Take TiDB for example, it is a distributed database based on MySQL (its own query language can be considered as a subset of MySQL), but actually most of the heavylifting is handled by TiKV, which is a distributed KV datastore with Raft distributed consensus. And then SurrealDB also leveraged TiKV to build their own graph-document hybrid database product...as one of the data transport. P.S.: used to be a contributor for SurrealDB.

mrlongroots 1y ago
Deepseek just used FoundationDB to build a parallel filesystem. Parallel filesystem are a big deal -- their number, including proprietary ones, is probably in single digits.
- metadat 1y ago
  Parallel Filesystems aren't a new or novel concept, and there have been lots of implementations. The first one I encountered was DrFTPD circa 2004. But these days, any object storage system qualifies because they all support varying replication schemes and reading from any valid in-sync replica.
  - funcDropShadow 1y ago
    Object stores are not filesystems. They have a paths, but they are not hierarchical.
    - em-bee 1y ago
      unless it's a hierarchical object store, like the one i am using.
      - 1y ago

exabrial 1y ago

Honestly though, why not just take the existing SQL standard and trim it down then?

rockwotj 1y ago

I do like this approach in theory, but agreed I don't think that the devex has been solved and I don't know how many people would value a different approach as SQL feels like king in many places... FWIW it's kind of the approach that IndexedDB in the web takes, however that the API is quite bad IMO. Or maybe there is a higher level DSL that you could apply to create query plans (something like MongoDB aggregation pipelines maybe?), but it quickly becomes basically the same as SQL.

osigurdson 1y ago
>> SQL feels like king in many places I think it is because most people can make something work with SQL.
mike_hearn 1y ago
The author doesn't know it, but he's asking for Permazen - exactly what you're asking for: https://github.com/permazen/permazen/blob/master/README.md https://github.com/permazen/permazen/blob/master/README.md It's a bit like the record layer in FoundationDB but more advanced. You specify query plans manually, so you can't accidentally forget an index for example.

DidYaWipe 1y ago

Yeah! Then I can stop grousing about yet another grammatical peeve: It's "key/value" pairs, not "key-value" pairs (which would mean pairs of key values).

sroussey 1y ago

All you need is MemcacheD