4 ms·

Thanks, I gathered that much. I am curious what kind of stuff people are using it for, especially in their personal projects.

• domper 23d ago
  One great use case I've found: You can find monthly archives of every reddit comment on Academic torrents. These are huge NDJSON files compressed to .zst, named like 'RC_2026-01.zst'. The size is ~60GB compressed, 350GB+ uncompressed, per month. Most of the size is taken by the actual comment text. But I was only interested in calculating how many unique commenters subreddits have in a month so I only wanted to extract a few fields from it and discard the rest. If you use traditional tools like pandas or load the data to a database and then query it, you would quickly run out of RAM or storage space, especially when doing it on a basic laptop like I was. But with DuckDB it's just this: ``` SELECT lower(subreddit) AS subreddit, author FROM read_json(['RC_2026-01.zst', 'RC_2026-02.zst']) ``` DuckDB automatically handles decompression on the fly, figures out the schema, manages RAM so you won't OOM and so on. And even on my laptop that query takes like, 2 minutes? Which is super impressive to me.
  • Fervicus 22d ago
    Oh, that's cool. And that's exactly the kind of answer I was looking for. So thanks a lot!