ritchie46

5 ms·

1.
by ritchie46 6mo ago
Rust engineer Polars TYPE: full-time LOCATION: Amsterdam REMOTE: Optional, Based in the Netherlands is preferred VISA: No DESCRIPTION: Polars is the company founded from the Polars OSS project. The company is building a distributed query en
2.
by ritchie46 8mo ago
Polars does that for you.
3.
by ritchie46 8mo ago
Python 3.14 "spawns" by default. However, this is not a Polars issue. Using "fork" can leave ANY MUTEX in the system process invalid (a multi-threaded query engine has plenty of mutexes). It is highly unsafe and has the
4.
by ritchie46 11mo ago
> but quite a bit of pandas code will run as-is with polars I highly doubt this. Aside from dataframe generation and series assignment, almost everything in the API surface is different. Strictness is also not something you can transplan
5.
by ritchie46 1y ago
We have full iceberg read support. We have done some preliminary work for iceberg write support. I think we will ship that once we have decided which Catalog we will add. The iceberg write API is intertwined with that.
6.
by ritchie46 1y ago
He means that he wants our Rust library as easy as our Python lib. Which I understand as our focus has been mostly on Python. It is where most of our userbase is and it is very hard for us to have a stable Rust API as we have a lot of inter
7.
by ritchie46 1y ago
With Polars Cloud you don't have to choose those either. You can pick cpu/memory and we will offer autoscaling in a few months. Cluster configuration is optional if you want this control. Anyhow, this doesn't have much to do
8.
by ritchie46 1y ago
We also target ad-hoc analysis. If your data doesn't fit on your laptop, you can spin up a larger box or a cluster and run interactive queries.
9.
by ritchie46 1y ago
On-premises is in the works. We expect this in a couple of months. Currently it is managed on AWS only.
10.
by ritchie46 1y ago
Not right now. Our current SQLContext locally inspects schema's to convert the SQL to Polars LazyFrames (DSL). However, this should happend during IR-resolving. E.g. the SQL should translate directly to Polars IR, and not LazyFrames. T
11.
by ritchie46 1y ago
I am not an expert on Spark RDDs, but AFAIK they are a more low-level data structure that offer resilience and a lower level map-reduce API. Polars Cloud maps the Polars API/DSL to distributed compute. This is more akin to Spark's
12.
by ritchie46 1y ago
You don't have to. Passing cpu and memory works as well. pc.ComputeContext{ cpus=4, memory=16 } We are working on a minimal cluster and auto-scaling based on the query.
13.
by ritchie46 1y ago
Your billing partner is AWS. Polars' markup is on your AWS bill.
14.
by ritchie46 1y ago
Hi, I am the original author and CEO of Polars. We are not focused on SQL at this time and provide a DataFrame native API. Polars cloud will for the moment only support our DataFrame API. SQL might come later on the roadmap, but since this
15.
by ritchie46 1y ago
Polars | Rust Engineer | Amsterdam ONSITE / HYBRID At Polars we're building a fast distributed query engine for Polars DataFrames. Our mission is scale DataFrame processing and offer a modern verstatile API to process data fast an
16.
by ritchie46 1y ago
Polars | Rust engineers | DevOps | FrontEnd | Hybrid Polars is built on the foundation of a vibrant and active open-source community, and we embrace that philosophy in how we run our company. We trust talented people to do their best work w
17.
by ritchie46 2y ago
Yes, you can run `pc.remote(my_udf, schema)` Where `def my_udf() -> DataFrame` We link the appropiate Python version at cluster startup.
18.
by ritchie46 2y ago
import polars.col as C C.blah
19.
by ritchie46 2y ago
Disclosure, I am the author of Polars and this post. The difference with Ibis is that Polars cloud will also manage hardware. It is similar to Modal in that sense. You don't have to have a running cluster to fire a remote query. The ot
20.
by ritchie46 2y ago
The cluster runs in your own VPC.
21.
by ritchie46 2y ago
Disclosure, I wrote this post. The compute-plane (cluster) will run in your own VPC.
22.
by ritchie46 2y ago
Nothing. Polars -> pola.rs The Polars name and a hint to the .rs file extension.
23.
by ritchie46 2y ago
Great! A small correction on your post. Polars does have SQL suppor. It isn't the main usecase, so it isn't as good as that of Spark and DuckDB, but it does exist and is being improved on.
24.
by ritchie46 2y ago
I do think code should be shared when you are benchmarking. He could be using Polars' eager API for instance, which would not be apples to apples.
25.
by ritchie46 2y ago
*COMPANY*: [polars]( https://pola.rs/ ) *TYPE*: full-time *LOCATION*: Amsterdam *REMOTE*: Optional, Based in the Netherlands is preferred *VISA*: No *DESCRIPTION*: Polars is the company founded from the Polars OSS project. Th
26.
by ritchie46 2y ago
I don't trust their benchmarks. I ran their benchmarks source locally on my machine TPCH scale 10. Polars was orders of magnitudes faster and didn't SIGABORT at query 10 (I wasn't OOM). (.venv) [fireducks] ritchie46 &#
27.
by ritchie46 2y ago
A DataFrame is our in memory table. A LazyFrame is a compute plan that can have DataFrames as source. The engine is what executes our plans and materializes a result. This is plural as we are building a new one.
28.
by ritchie46 2y ago
They have different implications for us. An abort due to an OOM isn't a bug in our program, as SEGFAULT is a serious bug we want to fix.
29.
by ritchie46 2y ago
Do you mean segfault or OOM? I am not aware of Polars segfaulting on high memory pressure. If it does segfault, would you mind opening an issue? Some context; Polars is building a new streaming engine that will eventually be ready to run th
30.
by ritchie46 2y ago
DuckDB has their own string type (quite similar to this) that deviates from Arrow (large)-string type, so it had to do a copy anyway. Nothing has changed on that front.

More ›