thedood

6 ms·

1.
by thedood 2y ago
As noted in the bottom of the design doc at https://github.com/ray-project/deltacat/blob/main/deltacat/c... , we also improved the runtime efficiency of compaction from O(nlogn) to O(n). However, a l
2.
by thedood 2y ago
Good to see you here! It's been great working with Daft to further improve data processing on Ray, and the early results of incorporating Daft into the compactor have been very impressive. Also agree with the overall sentiment here tha
3.
by thedood 2y ago
Some of DeltaCAT's goals and use cases have been discussed in this 2022 talk: https://youtu.be/M3pZDp1zock?t=4676 Today, our immediate next goal for DeltaCAT is to ensure that the compactor, and similar procedures for
4.
by thedood 2y ago
As alluded to in the blog post, Ray+Parquet+Iceberg is the next frontier we'd like to make our compactor and similar procedures available on in open source so that the community can start bringing similar benefits for their Iceberg wor
5.
by thedood 2y ago
From the blog post, the largest individual Ray cluster that was observed running a production compaction job in Q1 had 26,846 vCPUs and ~210TiB of RAM. This is roughly equivalent to a Ray cluster composed of 839 r5.8xlarge EC2 nodes (w&#x2F
6.
by thedood 2y ago
This is a specialized ETL use-case - similar to taking a single SQL query and creating a dedicated distributed application tailored to run only that query. The lower-level primitives in Ray Core (tasks and actors) are general purpose enough
7.
by thedood 2y ago
We wrote the compactor in Python but, as noted in my previous response to quadrature, most of the performance sensitive code is written in C++ and Rust (but still invoked from Python).
8.
by thedood 2y ago
Some of the initial differentiators are described at the bottom of our design doc at https://github.com/ray-project/deltacat/blob/main/deltacat/c... . But yes, controlling file I/O was also an
9.
by thedood 2y ago
Hi Narhem - one of the devs that worked on the migration here. The data volume, and subsequent compute power required to process it, is actually one of the things that led us to Ray (or Ray Core specifically) since it had the distributed co
10.
by thedood 2y ago
Hi mannyv - one of the devs that worked on the migration here. It has been a pretty long project - approached with caution due to the criticality of keeping our BI datasets healthy - but the preliminary results produced year-over-year kept