6 ms·

Curious if you prototyped DuckDB before deciding to invest further into SQLite. DuckDB works great as an in-memory database (it's also the default mode).

by aranke 3y ago

rektide 3y ago
I expect most workloads are more OLTP/transactional than OLAP/analytical.
- elithrar 3y ago
  Correct. DuckDB is really interesting technology, but it's not a direct successor to SQLite for transactional workloads. It's also very new: there's a LOT of new code in DuckDB on top of the (heavily fuzzed) SQLite parts. (I use it personally, but it's not the same thing as what we're building with D1)
- piperswe 3y ago
  Also, for OLAP workloads there's Workers Analytics Engine. Analytics Engine is arbitrarily scalable as opposed to a D1-like solution - I can almost guarantee I've inserted more data into its internal variant (which is built on extremely similar underlying infrastructure) than just about any customer would think of doing, and it handles it like a champ. Basically, we already have a product for doing DuckDB-like things, and the D1 architecture isn't great for high volume OLAP workloads. Disclaimer: I'm an engineer at Cloudflare, but not on a developer platform team. I'm not speaking for our developer platform strategy or anything, I'm just commenting on how it looks from where I sit.
kentonv 3y ago
We didn't, no. I'm sure there's a lot of really cool local-first databases out there, but SQLite has the benefit of being incredibly widely battle-tested, with literally billions of installations worldwide. It has received thorough security research and fuzzing (it's part of Chrome's attack surface after all). And there's tons of resources online to help people understand how to use it. Although I'm sure there are alternatives that serve certain use cases better it's hard to imagine anything coming close for ours. That said, the storage engine we've built is not that heavily dependent on SQLite specifically. Any database that uses a write-ahead log like SQLite does should be possible to adapt to it in the future. So maybe we'll eventually open it up to a variety of choices, or even let you bring your own as a Wasm module.
- kentonv 3y ago
  Oh, I've been informed that DuckDB uses SQLite under the hood, so maybe compiling DuckDB to Wasm and running it on top of this will be possible, we'll see.
  - aranke 3y ago
    From https://news.ycombinator.com/item?id=23290512 https://news.ycombinator.com/item?id=23290512: > DuckDB is indeed a free columnar database system, but it is not entirely built on top of SQLite. It exposes the same front-end and uses components of SQLite (the shell and testing infrastructure), but the execution engine/storage code is new.

Curious if you prototyped DuckDB before deciding to invest further into SQLite. DuckDB works great as an in-memory database (it's also the default mode).

by aranke 3y ago

Curious if you prototyped DuckDB before deciding to invest further into SQLite. DuckDB works great as an in-memory database (it's also the default mode).

rektide 3y ago

I expect most workloads are more OLTP/transactional than OLAP/analytical.

elithrar 3y ago
Correct. DuckDB is really interesting technology, but it's not a direct successor to SQLite for transactional workloads. It's also very new: there's a LOT of new code in DuckDB on top of the (heavily fuzzed) SQLite parts. (I use it personally, but it's not the same thing as what we're building with D1)
piperswe 3y ago
Also, for OLAP workloads there's Workers Analytics Engine. Analytics Engine is arbitrarily scalable as opposed to a D1-like solution - I can almost guarantee I've inserted more data into its internal variant (which is built on extremely similar underlying infrastructure) than just about any customer would think of doing, and it handles it like a champ. Basically, we already have a product for doing DuckDB-like things, and the D1 architecture isn't great for high volume OLAP workloads. Disclaimer: I'm an engineer at Cloudflare, but not on a developer platform team. I'm not speaking for our developer platform strategy or anything, I'm just commenting on how it looks from where I sit.

kentonv 3y ago

We didn't, no. I'm sure there's a lot of really cool local-first databases out there, but SQLite has the benefit of being incredibly widely battle-tested, with literally billions of installations worldwide. It has received thorough security research and fuzzing (it's part of Chrome's attack surface after all). And there's tons of resources online to help people understand how to use it. Although I'm sure there are alternatives that serve certain use cases better it's hard to imagine anything coming close for ours. That said, the storage engine we've built is not that heavily dependent on SQLite specifically. Any database that uses a write-ahead log like SQLite does should be possible to adapt to it in the future. So maybe we'll eventually open it up to a variety of choices, or even let you bring your own as a Wasm module.

kentonv 3y ago
Oh, I've been informed that DuckDB uses SQLite under the hood, so maybe compiling DuckDB to Wasm and running it on top of this will be possible, we'll see.
- aranke 3y ago
  From https://news.ycombinator.com/item?id=23290512 https://news.ycombinator.com/item?id=23290512: > DuckDB is indeed a free columnar database system, but it is not entirely built on top of SQLite. It exposes the same front-end and uses components of SQLite (the shell and testing infrastructure), but the execution engine/storage code is new.