Thanks :-) I've not done much profiling on DuckDB and what the overhead is - i.e. after the data is parsed how much memory is used. Would be really interesting

Thanks :-) I've not done much profiling on DuckDB and what the overhead is - i.e. after the data is parsed how much memory is used. Would be really interesting to push it to the limit - or to explore not loading the entire file in, but only reading the relevant parts, but again that probably requires a conversion first, e.g. to parquet or some other column based storage format.