mwc360

4 ms·

1.
by mwc360 2y ago
OSS Delta support deletion vectors. The problem is that the OSS Deltalake (based on Delta-rs) python library does not and this prevents engines like DuckDB and Polars from writing to suck tables. I'm pretty sure DV is in OSS since 3.1
2.
by mwc360 2y ago
Very cool! This seems like fantastic functionality and would make it super easy to migrate small Spark workloads to DuckDB :)
3.
by mwc360 2y ago
Ritchie - thx for graciously correcting some things I got wrong, will get it corrected!
4.
by mwc360 2y ago
FYI I had V-Order and Optimzed Write disabled in the benchmark. The only wrote diff was that I enabled deletion vectors in Spark since it’s supported which the other two don’t.
5.
by mwc360 2y ago
Author here: that’s exactly what I was trying to communicate but you said it better :)
6.
by mwc360 2y ago
Miles Cole here: I’d love to see Daft on Ray become more widely used. Same Dataframe API and run it in either single or multi-machine mode. The only thing I don’t love about it today is that their marketing is a bit misleading. Daft is dist
7.
by mwc360 2y ago
Author of the blog here: fair point. Pretty much every published benchmark has an agenda that ultimately skews the conclusion. I did my best here to be impartial, I.e I fully designed the benchmark and each test prior to running code on any
8.
by mwc360 2y ago
Miles Cole here… thx for the correction, another reader just notes this as well. I’ll get this corrected tomorrow and possibly retest after verifying I have spill set up. Thx!
9.
by mwc360 2y ago
Hi - Miles Cole here… I used lazy APIs where available. I.e. everything up to write_delta() is lazy in the Polars (Mod) variant. Yeah I was debating whether to share all of the source code. I may share a portion of it soon.