alamb

6 ms·

1.
Embedding a Tantivy Index in Parquet(github.com)
1 points by alamb 1y ago|1 comments
2.
by alamb 1y ago
This demo extends a Parquet file by embedding a Tantivy full-text search index inside it. A custom DataFusion TableProvider implementation uses the embedded full-text index to optimize wildcard LIKE predicates.
3.
by alamb 1y ago
> Note that the readers of Parquet need to be aware of any metadata to exploit it. But if not, nothing changes The one downside of this approach, which is likely obvious, but I haven't seen mentioned is that the resulting parquet fi
4.
by alamb 1y ago
> That is, start with Wild West and define specs as needed Yes this is my personal hope as well -- if there are new index types that are widespread, they can be incorporated formally into the spec However, changing the spec is a non triv
5.
by alamb 1y ago
We are actively working on supporting extension types. The mechanism is likely to be using the Arrow extension type mechanism (a logical annotation on top of existing Arrow types https://arrow.apache.org/docs/format&#x2
6.
by alamb 1y ago
See also related blog: https://datafusion.apache.org/blog/2025/04/10/fastest-tpch-g...
7.
by alamb 2y ago
Specifically, DataFusion is faster when querying parquet directly. Most of the leaderboard of ClickBench is for database specific file formats (that you first have to load the data into)
8.
by alamb 2y ago
I think you would pick DataFusion over DuckDB if you want to customize it substantially. Not just with user defined functions (which are quite easy to write in DataFusion and are very fast), but things like * custom file formats (e.g. Spira
9.
by alamb 2y ago
BTW here is a fun exercise that takes this idea to the extreme. Who can build a custom file format that gets the best ClickHouse performance (on DataFusion): https://github.com/apache/datafusion/issues/13448
10.
by alamb 2y ago
In general, if you can partition your datasets on your predicate column, sorting is likely the best option For example when you have a predicate like, `where id = 'fdhah-4311-ddsdd-222aa'` sorting on the `id` column will help How
11.
by alamb 2y ago
It would be amazing if the code for working with arrow on GPUs could be made open source -- I think that would drive a significant amount of adoption
12.
by alamb 2y ago
So great to see another project built on DataFusion @!
13.
by alamb 3y ago
The Apache Arrow PMC is pleased to announce the donation of the Comet project, a native Spark SQL Accelerator built on Apache Arrow DataFusion.
14.
Apache Arrow DataFusion Comet(arrow.apache.org)
6 points by alamb 3y ago|1 comments
15.
by alamb 3y ago
CMU's database courses are online and excellent: https://15445.courses.cs.cmu.edu/spring2024/ https://15721.courses.cs.cmu.edu/spring2023/
16.
by alamb 3y ago
BTW you can see a version of what an industrial strength query optimizer / execution engine looks like in Rust https://arrow.apache.org/datafusion/ (can also use it in your own projects) It is quite similar to wha
17.
Pg_analytics: Transforming Postgres into a Fast Analytical Database(docs.paradedb.com)
10 points by alamb 3y ago|3 comments
18.
DataWeb: Virtual Data Unsiloing(github.com)
1 points by alamb 3y ago|0 comments
19.
by alamb 3y ago
The following paper describes some of the tradeoffs between different formats Deep Dive into Common Open Formats for Analytical DBMSs https://www.vldb.org/pvldb/vol16/p3044-liu.pdf
20.
by alamb 3y ago
I do think it was important for duckdb to put out a new version of the results as the earlier version of that benchmark [1] went dormant with a very old version of duckdb with very bad performance, especially against polars. [1] https:&#x2
21.
by alamb 3y ago
DuckDB is a great piece of software if you are If you are looking for a query engine implemented in a safe language (Rust) I definitely suggest checking out DataFusion. It is comparable to DuckDB in performance, has all the standard built i
22.
by alamb 4y ago
Here is another blog post that offers some perspective on the growth of Arrow over the intervening years and future directions: https://www.datawill.io/posts/apache-arrow-2022-reflection/
23.
by alamb 4y ago
For completeness, FlightSQL[1] (as mentioned elsewhere in this thread) aims to provide such an HTTP based protocol https://arrow.apache.org/blog/2019/10/13/introducing-arrow-f...
24.
by alamb 4y ago
Ah -- got it! This is the beauty of aligning ourselves with technologies like Arrow, Parquet and DataFusion. We can share as well as benefit from the efforts of the broader community
25.
by alamb 4y ago
Author here -- it is "free" in the sense that all the effort we put into DataFusion flows directly into IOx. But we do put a lot of effort into DataFusion
26.
Rust Apache Arrow and Parquet 16.0.0 Highlights(arrow.apache.org)
2 points by alamb 4y ago|0 comments
27.
by alamb 4y ago
Rust for the win!
28.
by alamb 5y ago
Here is an announcement of the 5.0 release: https://news.ycombinator.com/item?id=28290616
29.
by alamb 5y ago
Security, performance and async parquet reader for the rustlang implementation of Apache Arrow
30.
Apache Arrow Rust Implementation Highlights(arrow.apache.org)
2 points by alamb 5y ago|1 comments

More ›