7 ms·
Seems like a good time to recommend Apache Datafusion. It's designed to be a library, but works pretty well standalone as well. There's a CLI version, but also
by tormeh 22d ago
Seems like a good time to recommend Apache Datafusion. It's designed to be a library, but works pretty well standalone as well. There's a CLI version, but also Python and Java bindings, as well as a Rust library, of course.
From my own experience I can say it integrates far better into your Rust app than DuckDB does.
Well over 100 monthly contributors, too.
- colinmarc 22d agoCame here to post the same thing. DataFusion is fantastic.
- tobilg 22d agoYou can even use DuckLake with Datafusion: https://github.com/tobilg/datafusion-ducklake-provider https://github.com/tobilg/datafusion-ducklake-provider
- cbility 21d agohttps://github.com/datafusion-contrib/datafusion-ducklake https://github.com/datafusion-contrib/datafusion-ducklake is more established, and linked to directly from the ducklake docs
- asa400 22d agoIs Datafusion related to Polars at all? I ask because you mentioned Rust integration.
- hocuspocus 21d agoNo, it's a competing ecosystem, even though both are built on Apache Arrow, and their dataframe APIs might look similar on the surface. DataFusion is being used as a building block for a growing number of databases and data processing engines in Rust, as it offers the necessary primitives.
- asa400 21d agoAwesome, thank you for the detail.
- hobofan 22d agoDatafusion has all of the individual pieces, especially if you just want to data analysis, but is far from an embedded database. Indexes, transactions, a first-class storage format are all things that come included with DuckDB, that you won't have with Datafusion.
- andriy_koval 21d agoindexes are very weak in duckdb, need to fit memory.
- 1egg0myegg0 21d agoThis fall, DuckDB will support larger than memory indexes! I'm really excited about it. https://duckdb.org/2026/08/17/duckdb-20-highlights#7-storage-format-v20 https://duckdb.org/2026/08/17/duckdb-20-highlights#7-storage...
- andriy_koval 21d agothey don't say "this fall", they say later this year, I think it is some parallel effort, and it is actually going for few years already (rewriting index logic to have them in mmapped buffers and not in memory), finger crossed they will finish it finally.
- dtnm 21d agoIt's juts not as user friendly as DuckDB.