7 ms·
Do the CSV files allow quoted newlines? If yes, what's the trick to avoid checking the whole file too find out whether a newline is quoted or a record separator
by ilyagr 1mo ago
Do the CSV files allow quoted newlines? If yes, what's the trick to avoid checking the whole file too find out whether a newline is quoted or a record separator when reading it from the middle in an async thread?
- xxs 1mo agomost likely - they read large chunks and decode them
- pdet 1mo agoDuckDB uses a speculative parallel CSV parsing technique. The basic idea is that the parser speculates about the state the CSV parser is in at a random byte (e.g., whether it is inside a quoted field) and tries to figure out where the next row starts based on that.There are validation steps during finalization as well, to ensure the parser did not got anything wrong in its speculation. I've never gotten around to writing a blog post about it, but I go quite in-depth on the technique in this presentation: https://www.youtube.com/watch?v=YrqSp8m7fmk https://www.youtube.com/watch?v=YrqSp8m7fmk (Disclaimer: I'm the author of the blog post and also the developer who implemented the entire CSV parser in DuckDB.)
- abirch 1mo agoThank you so much. I love DuckDB!
- hilariously 1mo agoBurning my points to say dude that's sick, parsing CSVs is hell I am genuinely going to watch this thank you.
- _zoltan_ 1mo agothere isn't any upvote limit on HN.
- hilariously 1mo agoNo, but its genuinely looked down upon to make frivolous comments.
- pletnes 1mo agoYou’re officially my hero, the duckdb csv parser is the fastest I could find and helped save a data project with hundreds of gigabytes of csv pain.
- ilyagr 1mo ago[flagged]
- quag 1mo agoThank you. I love using duckdb and frequently query directly from zstd compressed files (and write out new ones). They’re easy to inspect using visidata and make outer use of. I found the talk interesting and enjoyable.
- orthoxerox 1mo agoIn general, yes. But Hive, for example, doesn't support embedded newlines or record separators that are not newlines, which means that a lot of older CSV files containing big data do not allow quoted newlines.