MrPowers

5 ms·

1.
2025: The Year of 1,000 DataFusion-Based Systems(influxdata.com)
2 points by MrPowers 10mo ago|0 comments
2.
Managing spatial tables in Data Lakehouses with Iceberg(sedona.apache.org)
2 points by MrPowers 10mo ago|0 comments
3.
Is physical world AI the future of autonomous machines?(therobotreport.com)
1 points by MrPowers 10mo ago|0 comments
4.
by MrPowers 1y ago
Rust is a good language for performant computing in general, but especially for data projects because there are so many great OSS data libraries like DataFusion and Arrow. SedonaDB currently supports SQL, Python, R, and Rust APIs. We can s
5.
by MrPowers 1y ago
You can generate the dataset with the instructions in this readme: https://github.com/apache/sedona-spatialbench/tree/main Here are the queries: https://github.com/apache/sedona-spatialbe
6.
by MrPowers 1y ago
The "DuckDB is probably the most important geospatial software of the last decade" post has a nice related discussion: https://news.ycombinator.com/item?id=43881468
7.
by MrPowers 1y ago
There is a project called GeoPolars: https://github.com/geopolars/geopolars From the README: > Update (August 2024): GeoPolars is blocked on Polars supporting Arrow extension types, which would allow GeoPolars to pe
8.
by MrPowers 1y ago
SedonaDB builds on libraries in the Rust ecosystem, like Apache DataFusion, to provide users with a nice geospatial DataFrame experience. It has functions like ST_Intersects that are common in spatial libraries, but not standard in most Da
9.
SedonaDB: A new geospatial DataFrame library written in Rust(sedona.apache.org)
197 points by MrPowers 1y ago|49 comments
10.
Benefits of Apache Iceberg for geospatial data analysis(wherobots.com)
16 points by MrPowers 1y ago|1 comments
11.
by MrPowers 2y ago
IMO, it would have been better to donate the repos to a shared org and motivate the community to continue maintaining them. But pretty awesome this individual is retiring from programming / taking a sabbatical. There is nothing wrong
12.
by MrPowers 2y ago
> A Data Lakehouse is fine but what benefit does it give you over a much more simple solution of ETL/ELTing the data in batches (weekly, daily, hourly, etc) and letting it sit in some kind of DB. Lots of engines like Polars, PyTorch
13.
Delta Lake vs. Data Lakes – what's the difference?(delta.io)
1 points by MrPowers 2y ago|2 comments
14.
by MrPowers 2y ago
Lots of Spark workloads are executed with the C++ Photon engine on the Databricks platform, so we ironically have partially moved back to C++. Disclosure: I work for Databricks.
15.
by MrPowers 2y ago
The OP is the original creator of Ballista, so he's well aware of the project. Ballista is much less mature than Spark and needs a lot of work. It's awesome they're making Spark faster with Comet.
16.
Pros and cons of Hive-style partitioning(delta.io)
1 points by MrPowers 2y ago|0 comments
17.
Pandas API on Spark(spark.apache.org)
1 points by MrPowers 2y ago|0 comments
18.
by MrPowers 3y ago
I love Medellin and lived there for many years, but the air quality is terrible and getting worse. You can talk with any locals and they say that the climate is noticeably different than it was in the past. Medellin is surrounded by mounta
19.
Delta Lake Rust implements ACID transactions(delta-io.github.io)
1 points by MrPowers 3y ago|0 comments
20.
by MrPowers 3y ago
I work at Databricks, but am pretty much just an OSS nerd, mainly focusing on Delta Rust recently: https://github.com/delta-io/delta-rs I did some keyword research and wrote this post cause lots of folks are doing sear
21.
by MrPowers 3y ago
Delta Live Tables are a Databricks feature and aren't related to Delta Lake. Can't you just setup a cron job to vacuum periodically?
22.
by MrPowers 3y ago
Yea, comparing Delta Lake to Iceberg is more apt, but I've been shying away from that content cause I don't wanna flamewar. Another poster is asking for this post tho, so maybe I should write it. I don't really see how Delta
23.
by MrPowers 3y ago
Lots of Parquet files in the same directory are typically referred to as a "Parquet table". Yes, Parquet can be compressed with zip, but snappy is much more common because it's splittable. Parquet tables can be registered in
24.
by MrPowers 3y ago
Yea, that's exactly what Delta Lake does. All the table metadata is stored in a Parquet file (it's initially stored in JSON files, but eventually compacted into Parquet files). These tables are sometimes so huge that the table m
25.
by MrPowers 3y ago
Yea, it is fair feedback. I respect the Iceberg team & their work. I've been shying away from that post cause I don't wanna start a flamewar, but I will reflect on this and reconsider. Thank you.
26.
by MrPowers 3y ago
Lots of organizations have Parquet data lakes and are considering switching to Delta Lake. Converting a Parquet table to a Delta table is an in-place, cheap computation. You can just add the Delta Lake metadata to an existing Parquet table
27.
by MrPowers 3y ago
I actually wrote this. I thought it was going to be part of the post description and didn't realize it was going to be a comment.
28.
by MrPowers 3y ago
Yea, Spark works best with "right-sized" files. Let's suppose you have a data lake with 40,000 Parquet files. You need to list the files before you can read the data. This can take a few minutes. I've worked on data l
29.
by MrPowers 3y ago
Looking at this now. * Delta Lake supports merge-on-read via deletion vectors: https://delta.io/blog/2023-07-05-deletion-vectors/ * Why doesn't Delta Lake have efficient bulk load? Lots of the biggest datase
30.
by MrPowers 3y ago
Yep, it is re-inventing database systems and you raise a great question. At first glance, it seems like Delta Lake is inferior to a database. Most databases support multi-table transactions and Delta Lake only support transactions for sing

More ›