andygrove

36 ms·

1.
by andygrove 1y ago
I've just started exploring adding OpenTelemetry support to the Comet subproject of DataFusion. I'm excited to see the integration with Apache Arrow (Rust) and potentially DataFusion in the future.
2.
by andygrove 1y ago
Congrats on the launch! I contributed to the NVIDIA Spark RAPIDS project for ~4 years and for the past year have been contributing to DataFusion Comet, so I have some experience in Spark acceleration and I have some questions! 1. Given the
3.
by andygrove 2y ago
No.
4.
by andygrove 2y ago
I wish I'd known about the search issue. I do see a new opportunity for Ballista. By leveraging all of the Spark-compatible operators and expressions being built in Comet, it would be able to support a wider range of queries much more
5.
by andygrove 2y ago
Yes, Ballista failed to gain traction. I think that one of the challenges was that it only supported a small subset of Spark, and there was too much work involved to try and get to parity with Spark. The Comet approach is much more pragmati
6.
DataFusion Comet: Apache Spark Accelerator(github.com)
107 points by andygrove 2y ago|25 comments
7.
by andygrove 4y ago
Original author of DataFusion/Ballista here. Having alamb and others from InfluxData involved has been a huge help in driving the project forward and helping build an active community behind the project. It is genuinely hard to keep up
8.
by andygrove 4y ago
I will update these old pages on my blog and redirect them!
9.
by andygrove 4y ago
I think it is worth pointing out that this tool does support querying Delta Lake (the author of ROAPI is also a major contributor the native Rust implementation of Delta Lake). Delta Lake certainly supports transactions, so ROAPI can query
10.
Apache Arrow DataFusion 8.0.0 Release(arrow.apache.org)
8 points by andygrove 4y ago|0 comments
11.
by andygrove 5y ago
FWIW, my dealer is not adding any markup. Here is the email I received from them explaining the process: "We finally received an update from Ford regarding the Lightning! This will be a completely unique process from Ford and they will
12.
Apache Arrow DataFusion (Rust query engine) now has an online user guide(arrow.apache.org)
8 points by andygrove 5y ago|0 comments
13.
by andygrove 5y ago
The Apache Spark project is many many years ahead of DataFusion & Ballista with more than a decade of work from more than 1,700 contributors and is going strong. I don't see DataFusion as a competitor to Spark since it is specifica
14.
by andygrove 5y ago
There is no support for Plasma. There is a TableProvider API for custom file formats and there is built-in support for CSV, Parquet, and JSON.
15.
by andygrove 5y ago
Ballista started out as a separate project and was donated in April 2021. They currently share a release schedule (but have different versioning) and this was the first release of DataFusion to include the Ballista crate. My hope is that Ba
16.
by andygrove 5y ago
The best "Getting Started" documentation right now is that on docs.rs - https://docs.rs/ballista/0.5.0/ballista/ This demonstrates using the Rust client (BallistaContext + DataFrame). There are alre
17.
by andygrove 5y ago
Yes. The Ballista crate (part of the arrow-datafusion repo) provides distributed query execution and the scheduler has a gRPC service. Flight is used internally as well but not directly exposed to users. There is also work in progress to ad
18.
by andygrove 5y ago
There is experimental support for distributed query execution with spill-to-disk between stages to support larger than memory datasets. This is implemented in the Ballista crate, which extends DataFusion. https://github.com/
19.
by andygrove 5y ago
That doesn't surprise me at all. I have politely requested that the project considers renaming to avoid this. https://github.com/datafuselabs/datafuse/issues/654
20.
A New Development Workflow for Arrow's Rust Implementation(arrow.apache.org)
5 points by andygrove 5y ago|0 comments
21.
Ballista Distributed Compute Donated to Apache Arrow(arrow.apache.org)
15 points by andygrove 5y ago|0 comments
22.
by andygrove 6y ago
I thought this was about me until I got to the word Rich in the title.
23.
by andygrove 6y ago
Ah, yes, that makes sense. I can see how this could have been misread.
24.
by andygrove 6y ago
There is also a more recent blog post which perhaps led to the project being posted here (I am guessing). https://andygrove.io/2021/01/ballista-2021/
25.
by andygrove 6y ago
The project uses stable Rust. Which version are you trying to compile with?
26.
by andygrove 6y ago
Apache Arrow is a specification for in-memory columnar data, IPC format + Flight protocol, with implementations in a number of languages. Some of the implementations contain code to perform computations on the in-memory data. Some of the im
27.
by andygrove 6y ago
There are contributors and committers in the Arrow project working to resolve this. We recently removed specialization from the core Arrow crate and we plan on doing the same for the Parquet crate.
28.
Apache Arrow 2.0.0 Rust Highlights(arrow.apache.org)
2 points by andygrove 6y ago|0 comments
29.
by andygrove 6y ago
Arrow is a memory format and optimized for efficient vectorized processing in memory. Although it is possible to persist the Arrow format to disk, it is more common to use Parquet.
30.
by andygrove 6y ago
There is also https://github.com/ballista-compute/ballista (shameless plug for my own project)

More ›