5 ms·
I’d like to see a 1 trillion row challenge.
by worldwidelies 3y ago
I’d like to see a 1 trillion row challenge.
- tbragin 3y agoOne exists https://blog.coiled.io/blog/1trc.html https://blog.coiled.io/blog/1trc.html
- genewitch 3y agofive minutes! with some silliness groupby("name").agg(the math stuff). So their testbed (coiled, a cluster manager?) is meant to do exactly what the challenge requires. and it was data out of S3. one thing that i caught there was that parquet serves the same purpose as CSV but for much bigger datasets and now i want to learn more about it.