9 ms·

Pre-Release of Polars 2.0

403 points by komape 14d ago

benrutter 14d ago
> We don’t aim to make a big feature release of Polars 2.0. In fact we hope it to be a boring experience for you. The reason we bump this major version is that we can get rid of design decisions made in the past that currently block us and then we want to change defaults to more sensible settings that will benefit a greater audience I know this take reveals me as a very dull person, but I love seeing projects take semver seriously like this! Version bumps should really be about removing deprecated cruft rather than shiny new features. I've used polars for a while now, and their focus on stability was a big part if convincing me to make the jump initially!
- Bluestein 14d ago
  "Tranquil development" (vs. "hype-driven shipping") :)
- dist-epoch 14d ago
  That being said Polars is one of the few Python libraries from the hundreds I use that I need to read the notes of every minor release (eg 1.44 -> 1.45), because they tend to frequently deprecate, remove or change features.
  - duskdozer 14d ago
    It sounds like they should be on a version much higher than 2.x then.
    - sanderjd 13d ago
      Deprecating without breaking is fine in a minor version under semver.
      - tialaramex 13d ago
        There's also always a "strictness" curve. Rust has specific rules where you could write something which doesn't break those rules but does break people who've done something very specific At some point you reach the Hyrum's Law / Spacebar Heating edge where sure, there's no reasonable technical definition which concludes your change "broke" people but somebody's solution used to work and now it doesn't and they're angry about that. To give an example that's only slightly less ridiculous than spacebar heating, maybe somebody depended on being able to parse a file from your library, you re-designed the library, there are no API changes but they're parsing your source code and now their approach doesn't work. Generally, libraries with fewer and lighter users tend to be more lax, maybe they forgot to make a structure opaque, that's a break but none of their users notice, or they forgot to make an error type non-exhaustive, but everybody using their type had a default clause so no harm done when they add a new value. Hyrum means that as you get bigger and more important, some day, the spacebar heating problem happens and too bad. A language (like Rust) can help you forestall that day somewhat, and reduce the blast radius when it does, but it is inevitable, Hyrum's law wasn't something you can outright prevent from happening, it's a factor for engineers to consider when designing systems, like misuse resistance.
        sanderjd 13d ago
        Yeah. Agreed.
      - 0x457 13d ago
        Did you miss "remove or change features." ?
        sanderjd 13d ago
        No, I was just making a narrower point about one aspect of the comment.
  - esafak 13d ago
    A library is a collection of features, and any of them could have breaking changes. That's why semver is insufficient. It would be good to have a standardized way to indicate breaking changes in components, like changesets.
- nicce 14d ago
  > Version bumps should really be about removing deprecated cruft rather than shiny new features. Can there be deprecated cruft without new features? :-D
  - kvdveer 14d ago
    Ideally: no. All new shiny new features shouldn't have waited for the (N+1).0 version, they should already have been part of the (N).(M) version. In practice, the removing the deprecated cruft will remove blockers for some new features, but that should be rare.
  - SkiFire13 14d ago
    Yes, the features don't need to be added immediately.
  - illiac786 13d ago
    Good question. I guess sometimes stuff becomes unnecessary due to external factors and not due to new features.
- altern8 14d ago
  Aren't major versions supposed to indicate breaking changes..? That's how I thought semantic versioning worked
  - theptip 13d ago
    Concretely, TFA lists a bunch of input validation that is being made more strict in the default configuration.
  - suddenlybananas 14d ago
    >The reason we bump this major version is that we can get rid of design decisions made in the past that currently block us and then we want to change defaults to more sensible settings that will benefit a greater audience I don't know how to read this sentence other than "there are breaking changes we want to make"
    - altern8 14d ago
      I see. Just making sure I had it right :-)
      - winniewinnie 13d ago
        [flagged]
    - magicmicah85 14d ago
      The migration guide does say there are breaking changes, but the interpretation I have is "this won't have new features but allows us to develop new features".
  - cassianoleal 14d ago
    Not every product uses SemVer
    - winniewinnie 13d ago
      But Polars does: https://docs.pola.rs/development/versioning/ https://docs.pola.rs/development/versioning/ > Polars adheres to the semantic versioning specification: And it does have breaking changes in 2.0. The original asker presumably missed that. E: On the other hand, that whole page on versioning seems inconsistent.
rfgplk 14d ago
[flagged]
- dbdr 14d ago
  Why?
  - irpap 14d ago
    I assume because “land” is a word Claude would choose.
    - mgaunard 14d ago
      Claude's wording (and knowledge) is based on what competent senior engineers would say.
      - tecleandor 14d ago
        None of the competent senior engineers I know say "not this, but that" every two sentences.
        pepperoni_pizza 14d ago
        But what about load-bearing seams that have provenance?
        mgaunard 13d ago
        Those are all precise and meaningful words that are very useful in software design.
      - blitzar 14d ago
        I really hope I don't have to work with these "competent senior engineers" anytime soon — my codebase couldn't handle it. On the plus side I would spend all day hearing I am "absolutely right" from a superior being.
bobson_dugnutt5 14d ago
I love polars. Did a lot of evangelizing in work to get people to give up pandas in favor of it.
- mgaunard 14d ago
  Both have terrible syntax that make SQL look like the most readable thing ever.
  - condwanaland 14d ago
    Could not agree less. Ive always found SQL an unreadable mess but tools like polars and dplyr are such elegant ways to manipulate data. Pandas is a mess though.
    - world2vec 14d ago
      There's no way SQL is more unreadable than polars. IMO it's the other way around.
      - benrutter 14d ago
        > There's no way SQL is more unreadable than polars. IMO it's the other way around. I think on basic queries, SQL is really nice, but when stuff gets more complex, with a bunch of CTEs, let alone functions requiring loops, it becomes pretty obtuse.
      - sanderjd 13d ago
        I would say that it is easier to decompose polars (and all dataframe api) queries and to build them up from pieces than it is to do the same with sql. Any time I find myself writing more than five or so lines of sql, or especially building a sql string in parts with logic, I wish I had a dataframe api instead. But the reverse is also somewhat true, that simpler and explicit expressions are nicer with sql.
  - fzumstein 14d ago
    I tend to agree. SQL may have been harder to write in the past (worse autocomplete than pandas/polars), but now that AI is writing the code, SQL is usually much easier to read. So DuckDB is another interesting alternative to pandas.
arn3n 14d ago
The decision to default to the streaming engine is really interesting. My intuition is that this would be slower than other data frame operations that are more parallelizable with batch processing, because streaming engines necessarily process rows sequentially. Is my intuition off/am I overestimating how much auto-parallelization polars does?
- orlp 14d ago
  Streaming here has a different meaning than perhaps what you're used to. It's not referring to online processing where you maintain aggregates/state while an endless stream of data comes in. The name was chosen early on to contrast with the old execution model, which was essentially all-data-in-memory, column-at-a-time. That engine still exists, we use it as a fallback mechanism for things that aren't supported yet in the new engine (or if you explicitly ask for `engine="in-memory"`). The new execution model first constructs a computational graph of nodes which communicate in streams of in-cache batches (morsels) of data, meaning the full dataset will never be held in memory if not necessary. This was called the streaming engine for that reason in an early prototype and the name stuck. In hindsight I do admit the naming choice is somewhat confusing.
  - arn3n 14d ago
    Cool, thanks for the explanation!
  - sanderjd 13d ago
    When you say "in-cache batches", you mean that this cache is on disk? Is that only the case when data is quite large? (Or a more general question: What is the best resource for me to read about how the streaming engine and cache work?)
    - orlp 13d ago
      Well... once my recent work on out-of-core lands the batch could be on disk when we run out of memory budget ;) But no, that's not what I meant. I meant that the batch is meant to be of a size that fits in your CPU cache. This can be a huge throughput improvement as each bit of data stays in cache as it moves from data source to sink. Compare this to column-at-a-time execution: by the time you start the next operation on this column the start of the column will be out of cache again, meaning you operate at RAM speed (or worse, disk speed) rather than cache speed. I gave a (fairly surface-level) talk on the streaming engine a bit over a year ago: https://pola.rs/posts/talk-polars-meetup-1-streaming-engine/ https://pola.rs/posts/talk-polars-meetup-1-streaming-engine/.
bluebarbet 14d ago
What does this project have to do with Serbia? Are the developers in Belgrade?
- mkl 14d ago
  It's just a play on the name, and it's pretty common. claude.ai has nothing to do with Anguilla, John Romero's rome.ro has nothing to do with Romania, twitch.tv has nothing to do with Tuvalu, etc.
  - bluebarbet 14d ago
    Indeed, and Bit.ly has nothing to do with Libya, nor Lemmy.ml with Mali (both failed states). I posit that domain hacking is an ugly, shortsighted, unserious habit that we should drop.
Kydlaw 14d ago
Happy to see activity around Polars. This has been my go-to library for data processing due to the enhanced ergonomics compared to Pandas and SQL. But they were a bit quiet lately, and I started looking more and more into DuckDB recently… until the recent acquisition of DuckLab by AWS
- f311a 14d ago
  I've been using clickhouse-local for quite some time, instead of DuckDB. There is also chDB. After using pandas for 10 years, I favor SQL now, for some reason.
  - mihaelm 14d ago
    Maybe because it's like a swiss army knife for data work, regardless of whether you need it for OLTP or OLAP workloads. Having different SQL dialects is a bit annoying, but the base is the same more or less, so switching doesn't come at too big of a cost.
  - efromvt 14d ago
    As a huge duckdb fan, I'd love to see chDB to get proper windows support - that would make it real competition (having WASM coverage is already a big step) which would be good for the space as a whole.
  - Kydlaw 13d ago
    I haven't tried chDB yet, but I heard about it. Thank you for reminding me of that option. I use SQL in data pipelines and processing that is going to require interoperability. But for data exploration, I usually prefer Polars (imo it is easier to work with text, semi-structured data, etc.)
  - geysersam 13d ago
    Interesting, why do you typically prefer clickhouse local to duckdb?
    - f311a 13d ago
luciana1u 14d ago
[flagged]
trombonechamp 14d ago
Is there a reason besides performance that maintain_order=False by default? I ask because polars is used in many scientific data analysis pipelines, and non-deterministic behaviour is a well-documented source of bugs in scientific computing (e.g. https://pmc.ncbi.nlm.nih.gov/articles/PMC6919963/ https://pmc.ncbi.nlm.nih.gov/articles/PMC6919963/). The new default requires users to keep the implementation details of the API in their head while determining whether code is correct or not. This is tricky with scientific computing because the correct answer is not known in advance, so bugs can slide by and silently give incorrect results.
- winniewinnie 14d ago
  Perhaps to get better results on benchmarks.
  - Bootvis 13d ago
    Or you know, just better performance for people that know how to use their tool of choice.
    - winniewinnie 13d ago
      Wouldn't such people just pass in maintain_order=False ?
      - Bootvis 13d ago
        That’s an API design question. What is the more common use case. You seem to suggest they did it for benchmarking reasons only. They could use the option there themselves without changing the default so that is unlikely to be the motivation.
- hopfenspergerj 13d ago
  It's standard sql behavior, users always specify the ordering they want as part of the query.
lmeyerov 13d ago
Moving towards streaming and generally out-of-core is great We recently added a Polars backend to GFQL (cypher graph queries on dataframes, no DB needed), both CPU and GPU mode, and super impressive. Noticeable improvements vs pandas/cudf, and enabled GFQL to beat out popular systems on more categories like low-latency, not just big datasets: https://www.graphistry.com/blog/cypher-on-polars-cpu-gpu-graph-engine https://www.graphistry.com/blog/cypher-on-polars-cpu-gpu-gra...
perrygeo 13d ago
For me, the superpower of polars is production stability. Pandas tends to push all problems to runtime, with all sorts of hidden heuristics. Particularly around column types and missing values. It's very hard to know if you've tested all the edge cases. The only way to test your code is to throw all variations of data at it. Fine if you're sitting at a notebook and have the patience to validate and "clean" the data on its behalf. Not so fine if you get paged at 3am because your data pipeline failed when it expected an int column but got float. Polars is more strict by default and front-loads costs through its planner. The resulting apps are noticeably more stable in production. You can test code and reasonable assurance that it will work on data in the wild. I don't really have any interest in the API ergonomics or syntax - both are fine. It's all about how they deal with data variation at runtime. Can you write general code that doesn't break on variants? Pandas, not a chance. Polars, absolutely! Bonus round: polars has a Rust API too, the compiler can effectively prove that your program handles every edge case. It's common to write rust polars apps that run unattended for years.
- sigseg1v 13d ago
  Do people actually use languages where it doesn't do any compile time checks on the API in 2026? Why would developers put up with the lack of that. I'm not in this ecosystem but what you are describing sounds like the bare minimum to me that should be table stakes.
  - perrygeo 13d ago
    I agree. But the data science/engineering space is enamored with Python. Makes good sense when doing interactive work. Makes no sense in production. But by that point, most developers consider it a sunk cost and just keep their Python apps limping along rather than rewriting. That's why polars is a great option. Start prototyping in python, then a relatively easy port to a Rust app when the need hits.
    - maleldil 13d ago
      You can still have extensive "compile-time" checking with linters and static type checkers. This makes Polars even better in typed Python, as it type-checks much better than the dynamic mess that is Pandas.
bbstats 13d ago
Bothering me like crazy that "Use instead: .cat.to(dtype) for int → categorical, .cat.physical() for categorical → int." doesn't give the requisite code example!
- thijsn 13d ago
  https://docs.pola.rs/releases/upgrade/2/#disable-casting-from-integers-to-categoricals-and-from-categoricals-to-integers https://docs.pola.rs/releases/upgrade/2/#disable-casting-fro... You can find it in the migration guide. Let me know if you miss anything, if you'd like you can make an issue and I'll make sure to get to it before the 2.0 release!
dtj1123 13d ago
> as soon as their ready. So strange that it's now relatively normal to see a typo and think 'oh cool, a human wrote this, I can take this seriously' rather than 'oh dear, they can't spell, I can't take this seriously'.
wodenokoto 13d ago
>Enums/Categoricals <> integers. >pl.Series([None, 1, 0, 2], dtype=pl.UInt32).cast(pl.Enum(["a", "b", "c"])) ># ComputeError: casting from u32 to enum is not supported. >Use instead: .cat.to(dtype) for int → categorical, .cat.physical() for >categorical → int. Always show the correct way of doing things. I have no idea what the correct is here, and I don't really see what benefit this change in API brings.

403 points by komape 14d ago

> We don’t aim to make a big feature release of Polars 2.0. In fact we hope it to be a boring experience for you. The reason we bump this major version is that we can get rid of design decisions made in the past that currently block us and then we want to change defaults to more sensible settings that will benefit a greater audience I know this take reveals me as a very dull person, but I love seeing projects take semver seriously like this! Version bumps should really be about removing deprecated cruft rather than shiny new features. I've used polars for a while now, and their focus on stability was a big part if convincing me to make the jump initially!

Bluestein 14d ago
"Tranquil development" (vs. "hype-driven shipping") :)
dist-epoch 14d ago
That being said Polars is one of the few Python libraries from the hundreds I use that I need to read the notes of every minor release (eg 1.44 -> 1.45), because they tend to frequently deprecate, remove or change features.
- duskdozer 14d ago
  It sounds like they should be on a version much higher than 2.x then.
  - sanderjd 13d ago
    Deprecating without breaking is fine in a minor version under semver.
    - tialaramex 13d ago
      There's also always a "strictness" curve. Rust has specific rules where you could write something which doesn't break those rules but does break people who've done something very specific At some point you reach the Hyrum's Law / Spacebar Heating edge where sure, there's no reasonable technical definition which concludes your change "broke" people but somebody's solution used to work and now it doesn't and they're angry about that. To give an example that's only slightly less ridiculous than spacebar heating, maybe somebody depended on being able to parse a file from your library, you re-designed the library, there are no API changes but they're parsing your source code and now their approach doesn't work. Generally, libraries with fewer and lighter users tend to be more lax, maybe they forgot to make a structure opaque, that's a break but none of their users notice, or they forgot to make an error type non-exhaustive, but everybody using their type had a default clause so no harm done when they add a new value. Hyrum means that as you get bigger and more important, some day, the spacebar heating problem happens and too bad. A language (like Rust) can help you forestall that day somewhat, and reduce the blast radius when it does, but it is inevitable, Hyrum's law wasn't something you can outright prevent from happening, it's a factor for engineers to consider when designing systems, like misuse resistance.
      - sanderjd 13d ago
        Yeah. Agreed.
    - 0x457 13d ago
      Did you miss "remove or change features." ?
      - sanderjd 13d ago
        No, I was just making a narrower point about one aspect of the comment.
- esafak 13d ago
  A library is a collection of features, and any of them could have breaking changes. That's why semver is insufficient. It would be good to have a standardized way to indicate breaking changes in components, like changesets.
nicce 14d ago
> Version bumps should really be about removing deprecated cruft rather than shiny new features. Can there be deprecated cruft without new features? :-D
- kvdveer 14d ago
  Ideally: no. All new shiny new features shouldn't have waited for the (N+1).0 version, they should already have been part of the (N).(M) version. In practice, the removing the deprecated cruft will remove blockers for some new features, but that should be rare.
- SkiFire13 14d ago
  Yes, the features don't need to be added immediately.
- illiac786 13d ago
  Good question. I guess sometimes stuff becomes unnecessary due to external factors and not due to new features.
altern8 14d ago
Aren't major versions supposed to indicate breaking changes..? That's how I thought semantic versioning worked
- theptip 13d ago
  Concretely, TFA lists a bunch of input validation that is being made more strict in the default configuration.
- suddenlybananas 14d ago
  >The reason we bump this major version is that we can get rid of design decisions made in the past that currently block us and then we want to change defaults to more sensible settings that will benefit a greater audience I don't know how to read this sentence other than "there are breaking changes we want to make"
  - altern8 14d ago
    I see. Just making sure I had it right :-)
    - winniewinnie 13d ago
      [flagged]
  - magicmicah85 14d ago
    The migration guide does say there are breaking changes, but the interpretation I have is "this won't have new features but allows us to develop new features".
- cassianoleal 14d ago
  Not every product uses SemVer
  - winniewinnie 13d ago
    But Polars does: https://docs.pola.rs/development/versioning/ https://docs.pola.rs/development/versioning/ > Polars adheres to the semantic versioning specification: And it does have breaking changes in 2.0. The original asker presumably missed that. E: On the other hand, that whole page on versioning seems inconsistent.

rfgplk 14d ago

[flagged]

dbdr 14d ago
Why?
- irpap 14d ago
  I assume because “land” is a word Claude would choose.
  - mgaunard 14d ago
    Claude's wording (and knowledge) is based on what competent senior engineers would say.
    - tecleandor 14d ago
      None of the competent senior engineers I know say "not this, but that" every two sentences.
      - pepperoni_pizza 14d ago
        But what about load-bearing seams that have provenance?
        mgaunard 13d ago
        Those are all precise and meaningful words that are very useful in software design.
    - blitzar 14d ago
      I really hope I don't have to work with these "competent senior engineers" anytime soon — my codebase couldn't handle it. On the plus side I would spend all day hearing I am "absolutely right" from a superior being.

bobson_dugnutt5 14d ago

I love polars. Did a lot of evangelizing in work to get people to give up pandas in favor of it.

mgaunard 14d ago
Both have terrible syntax that make SQL look like the most readable thing ever.
- condwanaland 14d ago
  Could not agree less. Ive always found SQL an unreadable mess but tools like polars and dplyr are such elegant ways to manipulate data. Pandas is a mess though.
  - world2vec 14d ago
    There's no way SQL is more unreadable than polars. IMO it's the other way around.
    - benrutter 14d ago
      > There's no way SQL is more unreadable than polars. IMO it's the other way around. I think on basic queries, SQL is really nice, but when stuff gets more complex, with a bunch of CTEs, let alone functions requiring loops, it becomes pretty obtuse.
    - sanderjd 13d ago
      I would say that it is easier to decompose polars (and all dataframe api) queries and to build them up from pieces than it is to do the same with sql. Any time I find myself writing more than five or so lines of sql, or especially building a sql string in parts with logic, I wish I had a dataframe api instead. But the reverse is also somewhat true, that simpler and explicit expressions are nicer with sql.
- fzumstein 14d ago
  I tend to agree. SQL may have been harder to write in the past (worse autocomplete than pandas/polars), but now that AI is writing the code, SQL is usually much easier to read. So DuckDB is another interesting alternative to pandas.

arn3n 14d ago

The decision to default to the streaming engine is really interesting. My intuition is that this would be slower than other data frame operations that are more parallelizable with batch processing, because streaming engines necessarily process rows sequentially. Is my intuition off/am I overestimating how much auto-parallelization polars does?

orlp 14d ago
Streaming here has a different meaning than perhaps what you're used to. It's not referring to online processing where you maintain aggregates/state while an endless stream of data comes in. The name was chosen early on to contrast with the old execution model, which was essentially all-data-in-memory, column-at-a-time. That engine still exists, we use it as a fallback mechanism for things that aren't supported yet in the new engine (or if you explicitly ask for `engine="in-memory"`). The new execution model first constructs a computational graph of nodes which communicate in streams of in-cache batches (morsels) of data, meaning the full dataset will never be held in memory if not necessary. This was called the streaming engine for that reason in an early prototype and the name stuck. In hindsight I do admit the naming choice is somewhat confusing.
- arn3n 14d ago
  Cool, thanks for the explanation!
- sanderjd 13d ago
  When you say "in-cache batches", you mean that this cache is on disk? Is that only the case when data is quite large? (Or a more general question: What is the best resource for me to read about how the streaming engine and cache work?)
  - orlp 13d ago
    Well... once my recent work on out-of-core lands the batch could be on disk when we run out of memory budget ;) But no, that's not what I meant. I meant that the batch is meant to be of a size that fits in your CPU cache. This can be a huge throughput improvement as each bit of data stays in cache as it moves from data source to sink. Compare this to column-at-a-time execution: by the time you start the next operation on this column the start of the column will be out of cache again, meaning you operate at RAM speed (or worse, disk speed) rather than cache speed. I gave a (fairly surface-level) talk on the streaming engine a bit over a year ago: https://pola.rs/posts/talk-polars-meetup-1-streaming-engine/ https://pola.rs/posts/talk-polars-meetup-1-streaming-engine/.

bluebarbet 14d ago

What does this project have to do with Serbia? Are the developers in Belgrade?

mkl 14d ago
It's just a play on the name, and it's pretty common. claude.ai has nothing to do with Anguilla, John Romero's rome.ro has nothing to do with Romania, twitch.tv has nothing to do with Tuvalu, etc.
- bluebarbet 14d ago
  Indeed, and Bit.ly has nothing to do with Libya, nor Lemmy.ml with Mali (both failed states). I posit that domain hacking is an ugly, shortsighted, unserious habit that we should drop.

Kydlaw 14d ago

Happy to see activity around Polars. This has been my go-to library for data processing due to the enhanced ergonomics compared to Pandas and SQL. But they were a bit quiet lately, and I started looking more and more into DuckDB recently… until the recent acquisition of DuckLab by AWS

f311a 14d ago
I've been using clickhouse-local for quite some time, instead of DuckDB. There is also chDB. After using pandas for 10 years, I favor SQL now, for some reason.
- mihaelm 14d ago
  Maybe because it's like a swiss army knife for data work, regardless of whether you need it for OLTP or OLAP workloads. Having different SQL dialects is a bit annoying, but the base is the same more or less, so switching doesn't come at too big of a cost.
- efromvt 14d ago
  As a huge duckdb fan, I'd love to see chDB to get proper windows support - that would make it real competition (having WASM coverage is already a big step) which would be good for the space as a whole.
- Kydlaw 13d ago
  I haven't tried chDB yet, but I heard about it. Thank you for reminding me of that option. I use SQL in data pipelines and processing that is going to require interoperability. But for data exploration, I usually prefer Polars (imo it is easier to work with text, semi-structured data, etc.)
- geysersam 13d ago
  Interesting, why do you typically prefer clickhouse local to duckdb?
  - f311a 13d ago

luciana1u 14d ago

[flagged]

trombonechamp 14d ago

Is there a reason besides performance that maintain_order=False by default? I ask because polars is used in many scientific data analysis pipelines, and non-deterministic behaviour is a well-documented source of bugs in scientific computing (e.g. https://pmc.ncbi.nlm.nih.gov/articles/PMC6919963/ https://pmc.ncbi.nlm.nih.gov/articles/PMC6919963/). The new default requires users to keep the implementation details of the API in their head while determining whether code is correct or not. This is tricky with scientific computing because the correct answer is not known in advance, so bugs can slide by and silently give incorrect results.

winniewinnie 14d ago
Perhaps to get better results on benchmarks.
- Bootvis 13d ago
  Or you know, just better performance for people that know how to use their tool of choice.
  - winniewinnie 13d ago
    Wouldn't such people just pass in maintain_order=False ?
    - Bootvis 13d ago
      That’s an API design question. What is the more common use case. You seem to suggest they did it for benchmarking reasons only. They could use the option there themselves without changing the default so that is unlikely to be the motivation.
hopfenspergerj 13d ago
It's standard sql behavior, users always specify the ordering they want as part of the query.

lmeyerov 13d ago

Moving towards streaming and generally out-of-core is great We recently added a Polars backend to GFQL (cypher graph queries on dataframes, no DB needed), both CPU and GPU mode, and super impressive. Noticeable improvements vs pandas/cudf, and enabled GFQL to beat out popular systems on more categories like low-latency, not just big datasets: https://www.graphistry.com/blog/cypher-on-polars-cpu-gpu-graph-engine https://www.graphistry.com/blog/cypher-on-polars-cpu-gpu-gra...

perrygeo 13d ago

For me, the superpower of polars is production stability. Pandas tends to push all problems to runtime, with all sorts of hidden heuristics. Particularly around column types and missing values. It's very hard to know if you've tested all the edge cases. The only way to test your code is to throw all variations of data at it. Fine if you're sitting at a notebook and have the patience to validate and "clean" the data on its behalf. Not so fine if you get paged at 3am because your data pipeline failed when it expected an int column but got float. Polars is more strict by default and front-loads costs through its planner. The resulting apps are noticeably more stable in production. You can test code and reasonable assurance that it will work on data in the wild. I don't really have any interest in the API ergonomics or syntax - both are fine. It's all about how they deal with data variation at runtime. Can you write general code that doesn't break on variants? Pandas, not a chance. Polars, absolutely! Bonus round: polars has a Rust API too, the compiler can effectively prove that your program handles every edge case. It's common to write rust polars apps that run unattended for years.

sigseg1v 13d ago
Do people actually use languages where it doesn't do any compile time checks on the API in 2026? Why would developers put up with the lack of that. I'm not in this ecosystem but what you are describing sounds like the bare minimum to me that should be table stakes.
- perrygeo 13d ago
  I agree. But the data science/engineering space is enamored with Python. Makes good sense when doing interactive work. Makes no sense in production. But by that point, most developers consider it a sunk cost and just keep their Python apps limping along rather than rewriting. That's why polars is a great option. Start prototyping in python, then a relatively easy port to a Rust app when the need hits.
  - maleldil 13d ago
    You can still have extensive "compile-time" checking with linters and static type checkers. This makes Polars even better in typed Python, as it type-checks much better than the dynamic mess that is Pandas.

bbstats 13d ago

Bothering me like crazy that "Use instead: .cat.to(dtype) for int → categorical, .cat.physical() for categorical → int." doesn't give the requisite code example!

thijsn 13d ago
https://docs.pola.rs/releases/upgrade/2/#disable-casting-from-integers-to-categoricals-and-from-categoricals-to-integers https://docs.pola.rs/releases/upgrade/2/#disable-casting-fro... You can find it in the migration guide. Let me know if you miss anything, if you'd like you can make an issue and I'll make sure to get to it before the 2.0 release!

dtj1123 13d ago

> as soon as their ready. So strange that it's now relatively normal to see a typo and think 'oh cool, a human wrote this, I can take this seriously' rather than 'oh dear, they can't spell, I can't take this seriously'.

wodenokoto 13d ago

>Enums/Categoricals <> integers. >pl.Series([None, 1, 0, 2], dtype=pl.UInt32).cast(pl.Enum(["a", "b", "c"])) ># ComputeError: casting from u32 to enum is not supported. >Use instead: .cat.to(dtype) for int → categorical, .cat.physical() for >categorical → int. Always show the correct way of doing things. I have no idea what the correct is here, and I don't really see what benefit this change in API brings.