6 ms·
Have you benchmarked this against pickling those data files? In our experience, parquet's overhead isn't worth it for smaller data files.
by lsorber 6y ago
Have you benchmarked this against pickling those data files? In our experience, parquet's overhead isn't worth it for smaller data files.
- EdwardDiego 6y agoI'm not surprised, Parquet's columnar encoding and compression won't really kick in significantly for smaller files.
- kylebarron 6y agoBut with pickling you can only read the data in Python.
- alfalfasprout 6y agoI just did some benchmarks and it's pretty similar for small files. The difference would only be noticeable if you're serializing a ton of small files.
- lsorber 6y agoHuh, makes a pretty big difference for us. We were using pandas' built-in to_parquet though, which seems to suffer from some overhead.
- cbsmith 6y agoIf pickling is what is working best for you, it can't be much data.