7 ms·
Choose DuckDB rather than SQLite
- deleted 2mo ago[deleted]
- pixelesque 2mo agoEven for row-based data?
- cronping 2mo ago[flagged]
- brightball 2mo ago> DuckDB's columnar engine That is workload specific. Title should be "Choose DuckDB rather than SQLite for Analytics" IMHO
- ray_v 2mo agoYes - it's a specific workload for sure. SQLite is still the GOAT when it comes to OLTP, but DuckDB is really becoming the GOAT in the OLAP world - I think DuckDB is simply amazing and truly an amazing piece of technology for anyone working with large amounts of data.
- dataviz1000 2mo ago> SQLite is still the GOAT when it comes to OLTP, but DuckDB is really becoming the GOAT in the OLAP world Can you explain this more, especially why SQLite is best at OLTP and what happens at scale?
- storywatch 2mo agoMost people forget that clickhouse embedded exists
- iw2rmb 2mo agoLet's keep discussion bar high. I've just checked their website, and they state "relying on ClickHouse to power these analytics use cases". That's not OLTP. https://clickhouse.com/comparison/postgresql https://clickhouse.com/comparison/postgresql Fair to say, seeing 1000x w/o any trace of proof won't help me to choose.
- misiek08 2mo agoYou new to the world, right? ClickHouse proven being a beast for analytics, tracing and even some basic metrics storage (VM won of course, but still CH is probably 2nd or 3rd)
- storywatch 2mo ago* for other readers, VM here most likely refer to Victoria Metrics
- iw2rmb 2mo agoEvery day someone is new to the world. Analytics is not OLTP. Would appreciate for anything to read except “you stupid” and “believe me, bro”.
- dangoodmanUT 2mo agoThis. The title might as well be “Choose a hammer rather than a wrench (when driving nails)”
- cognitiveinline 2mo agoYeah, someone who has more tokens than simple sense.
- adsharma 2mo agoDuckDB also implements MVCC. In DuckLake, the DB is used for absorbing small writes and compacting them before sending to object storage. It has some characteristics typical of OLTP engines. But they are targeted and limited to areas DuckDB feels are important.
- sharpvik 2mo agogreat research there
- egeozcan 2mo agoTL;DR: SQLite was doing OLAP work it was never built for (and it was okay at it, I must add), and the perf. ceiling moves two orders of magnitude when you use something that's more fit for the purpose (DuckDB in this case). TLDRTL;DR: If everything you do is column-store territory, use a column-store.
- knuckleheads 2mo agoI ran this through an AI checker and it flagged half of it immediately. @dang, I know Substack just enabled Pangram integration, is there anyway you could get Y Combinator to spring for a Pangram subscription for the front page or something ?
- nickpeterson 2mo agoDo you think venture capital firms are made of money?
- ethin 2mo agoYes? They certainly act like they are...
- spooneybarger 2mo agowell played sir. well played.
- tptacek 2mo agoAI comments aren't allowed on HN. AI submissions are.
- MrBuddyCasino 2mo agoIt would still be nice to mark them.
- knuckleheads 2mo agoYes, I am asking if that particular policy could be changed. A little AI here and there is fine, to each their own, but I would prefer not to read posts that are overwhelmingly so.
- tomhow 2mo agoIt's one thing for us to detect and autokill generated comments on our own site; it's our site and we can set the rules and run software on our own servers to process the comments and handle things in the way we and the community are happy with. It's a big additional leap to for our software to try to reach into others' sites, get through any anti-bot defenses they may be running, try to scrape their content and evaluate on whether it's sufficiently human-authored to be on HN. There's generally a wider range of LLM involvement with a long-form post than the typical, relatively brief HN comment, which then opens the way for more debate on HN about "how much" LLM influence the post has and how much should be allowed on HN. Part of what we're trying to optimize for on HN is minimizing offtopic/meta discussion, so we don't want to encourage this kind of debate. Our heuristic about article quality is largely unchanged from before LLMs were an issue: if an article is badly written, it shouldn't be on HN, and should be flagged.
- clumsysmurf 2mo agoToo bad Android doesn't have JDBC APIs. Getting the native binaries compiled on Android is the easy part, but there is no straightforward way to access it from Java. The Room APIs are tied to SQLite as well.
- tosti 2mo agoYou could use the framework built-in sqlite, or you could drop in h2 and use that. It's much more advanced than sqlite.
- bbqbbqbbq 2mo ago[dead]
- cynicalsecurity 2mo agoGreat job on comparing apples to oranges. DuckDB is a columnar OLAP engine, SQLite is row-oriented OLTP. DuckDB should stomp SQLite in that particular use case.
- tptacek 2mo agoTitle, which is already synthetic, should be "Choose DuckDB rather than SQLite for Analytics Workloads".
- otterley 2mo agoAI slop. The content might be valuable but the framing makes it too painful to read. Please, folks, write with your own voice -- especially if it's for your business blog. It's good for you as an author (practice makes perfect) and it's good for your readers (whom you want to influence).
- ethin 2mo agoHow are these two DB engines even comparable other than at the edges? They handle two completely separate workload types: one is more a general-purpose DB engine and the other is specifically for columnar datasets, analytics and the like -- of course a hand-tuned DB engine is going to destroy SQLite on any reasonable benchmark: SQLite wouldn't be optimized for that hand-tuned use-case whereas something like DuckDB is.
- coldbrewed 2mo agoSQLite is _the_ tool of choice for local SQL databases with minimal overhead. If you needed a single file DB for an OLAP workload, SQLite was still the best option even if the technology wasn't an ideal fit. Duckdb is exciting specifically because SQLite/duckdb aren't comparable; we can stop shoehorning OLAP into an OLTP database. I ran into this myself; I tried using SQLite to store the results of whole-internet rDNS scan and a count() over the entire DB could take 8 minutes. I used the wrong DB for the job and the narrative around SQLite/duckdb is around reckoning with perfectly reasonable limitations and tradeoffs that SQLite made.
- ethin 2mo agoSure, but this is what I'm getting at. :) SQLite is a general-purpose engine that is absolutely amazing at what it does and where it's used (and in many cases even where it isn't appropriate). But TFA was about comparing the two which just doesn't work because one is hand-tuned for a specific use-case while the other is not.
- datadrivenangel 2mo agoWith indexes SQLite is very very fast even for aggregation at medium scales. And DuckDB is reasonably fast for even single record writes. ~1000x slower than SQLite, but that's still pretty fast if you're only doing a few hundred writes per second or batching.
- ethin 2mo agoAgreed. And DuckDB certainly has a very nice API. But I wouldn't use it for a general dataset (think document storage).
- shubhamjain 2mo agoDespite its obvious advantages, the biggest drawback of DuckDB is its concurrency model [1]. If a process opens a database in read-write mode, it acquires an exclusive lock on the file. This prevents even simple read operations from other processes as long as the writer remains open. Maybe there's a simple workaround I haven't come across, but I found it to be quite a productivity killer. So yes, all these benchmarks are great, but it wasn't so fun working with DuckDB when I had to close duckdb cli, just so a query in another script could run. [1]: https://duckdb.org/docs/current/connect/concurrency https://duckdb.org/docs/current/connect/concurrency
- datadrivenangel 2mo agoQuack is now kind of a workaround for that limitation, as you can have a process with the lock offer read access to other processes. It's not perfect, but for that specific use case it's pretty good.
- coldbrewed 2mo agoDuckdb has a server mode[1] which might alleviate some of those pain points. SQLite is a bit more precise in that only a single connection can write to the DB which provides more concurrency but still has pain points. For a single file DB either choice seems justifiable to manage complexity. [1]: https://duckdb.org/2026/05/12/quack-remote-protocol https://duckdb.org/2026/05/12/quack-remote-protocol
- biophysboy 2mo agoYou can have multiple read only processes. But yes, concurrent read mode and write mode is blocked
- threatofrain 2mo agoThis is actually the biggest reason I don't automatically choose sqlite for small project databases. I've built a lot of toy utilities that crossed a bare threshold of usefulness, and then suddenly it's not a toy anymore when people start using it. Now it's debt. Oops.
- deleted 2mo ago[deleted]
- deleted 2mo ago[deleted]
- oathvz 2mo agoThis is a bait and switch article. Compare apples and oranges, "oh btw look at our product".
- datadrivenangel 2mo agoThis is AI slop, but I really want to know more about their write patterns and how they were doing batches.
- deleted 2mo ago[deleted]
- lanstin 2mo agoDuckDB is modern but written in C++ and crashes in production more than SQLite, which is old and it doesn’t really crash. you have to build in resilience to use DuckDb.
- frollogaston 2mo agoWait seriously? That's a dealbreaker if true.
- lanstin 2mo agoI don’t use it myself but a friend of mine uses it at a startup selling a cheap snowflake proxy and yeah, especially with new releases.
- raro11 2mo ago> $16.49/month server [...] hetzner CCX13 That server is now $51.09 for those wondering See https://news.ycombinator.com/item?id=48540844 https://news.ycombinator.com/item?id=48540844
- crustycoder 2mo agoHeartening to see so many "This is an apple, that is an orange" comments. Spot on folks!
- k3liutZu 2mo agoI couldn't read the article as it read like AI. And I am fatigued by the AI style in all code comments, reviews, PRs etc :(
- biophysboy 2mo agoAs many others have said here, you should just think about transactional vs analytical as well as single user in-process vs multi-user client-server when making choices. That said, I do think duckdb has a wider range of use cases than people here might think. It can whip through fairly large datasets (I use it for ~1B row tables all the time)
- d1l 2mo agoThe AI slop is tiring man wtf. It’s so fucking lazy. The benchmark is comparing apples to oranges and doesn’t seem to be aware of it, and the way it’s written just reeks of LLM.
- adsharma 2mo agoSQLite can be replicated. rqlite, dqlite, litestream etc DuckDB can't be. PR was sent a year ago. Blocked on the same concurrency model issue in the other sub thread. Specifically on windows, the database can't read its own WAL file from a different thread in the same process. Love DuckDB for being permissively open source, great tech and performance!
- esafak 2mo agoHow about writing with sqlite and querying with duckdb-sqlite, using a replica or WAL, to avoid locks?
- deleted 2mo ago[deleted]
- monkepon 2mo agowait, it means you should not save all traces to Langfuse? what is the primary usecase of Langfuse? exporting some subset of all traces for some quality checks etc.?