7 ms·
My takeaway is that csv has some undefined behaviours, and it takes up space. I like that everyone knows about .csv files, and it's also completely human reada
by aronhegedus 2y ago
My takeaway is that csv has some undefined behaviours, and it takes up space.
I like that everyone knows about .csv files, and it's also completely human readable.
So for <100mb I would still use csv.
- nradov 2y agoIf both parties implement RFC 4180 and use a consistent character set encoding then I don't think there are actually any undefined behaviors. But in practice a lot of implementations are simply broken, including those from major tech companies that ought to know better.
- senknvd 2y agoI don't think RFC 4180 differentiates between an empty string and a null value. As long as you add a check that all string columns are free of empty values before writing you should be good. I think in polars it's df.filter(pl.col(pl.Utf8).str.len_bytes() == 0).shape[0] == 0 although there's probably a better way to write this.
- nradov 2y agoWell I would consider differentiation between empty string versus null as simply being out of scope for CSV rather than undefined behavior. It was never intended as a complete database dump format.
- thayne 2y agoAnd the application doesn't try to convert the cells into non-string data types like numbers, dates, etc.