9 ms·

Is there a reason besides performance that maintain_order=False by default? I ask because polars is used in many scientific data analysis pipelines, and non-de

Is there a reason besides performance that maintain_order=False by default? I ask because polars is used in many scientific data analysis pipelines, and non-deterministic behaviour is a well-documented source of bugs in scientific computing (e.g. https://pmc.ncbi.nlm.nih.gov/articles/PMC6919963/ https://pmc.ncbi.nlm.nih.gov/articles/PMC6919963/). The new default requires users to keep the implementation details of the API in their head while determining whether code is correct or not. This is tricky with scientific computing because the correct answer is not known in advance, so bugs can slide by and silently give incorrect results.

winniewinnie 14d ago
Perhaps to get better results on benchmarks.
- Bootvis 14d ago
  Or you know, just better performance for people that know how to use their tool of choice.
  - winniewinnie 14d ago
    Wouldn't such people just pass in maintain_order=False ?
    - Bootvis 14d ago
      That’s an API design question. What is the more common use case. You seem to suggest they did it for benchmarking reasons only. They could use the option there themselves without changing the default so that is unlikely to be the motivation.
hopfenspergerj 14d ago
It's standard sql behavior, users always specify the ordering they want as part of the query.
- Xunjin 14d ago
  Often people new to databases and SQL thinks ordering is implicit, I have teach dozens of juniors that believe this is the default. I always wondered why that is the thought process...
  - fwip 14d ago
    Because out of the first 100 queries they ran, 99 came back in order and they didn't notice that the 100th didn't.
Permik 14d ago
This is a tricky field, the problem is not actually the non-determinism of the processing algorithms, but implicit ordering of the data.[1] The implicit ordering of the data is a footgun that -- as seen in the paper -- has already claimed victims. Using algorithms that don't need to upkeep the ordinality requirement in every operation will definitely move the library to a better direction and make future data modeling better and more explicit. [1] Aha, now I see why language models use this so frequently and why it might be overrepresented in the data. This is a perfect way to move the blame from the person you're responding to, if they're mistaken. They probably have a super, super overtuned "politeness" gym using sentiment analysis that tries to reword answers to not blame the misunderstandings of the person. Then this blame shifting unfortunately gets re-used as this super, super common phrase.
- winniewinnie 14d ago
  By "implicit ordering", do you mean "implicitly assumed that the data is ordered a certain way"? Since if that assumption of data being sorted a certain way is broken on some systems and not others, the result might be both non-deterministic (which could be a bug if the result is not allowed to be non-deterministic, but may or may not be a bug regarding the algorithm's assumptions) as well as a bug if the algorithm's assumptions requires it to be sorted a certain way. > Using algorithms that don't need to upkeep the ordinality requirement in every operation will definitely move the library to a better direction and make future data modeling better and more explicit. How would the library "make future data modeling ... more explicit" if this is a change to a default, which is implicit?
  - Lvl999Noob 14d ago
    > How would the library "make future data modeling ... more explicit" if this is a change to a default, which is implicit? It would become more explicit because where the order matters, you will now see a `maintain_order=True` where previously, you couldn't say whether it actually needed the fixed order or not.
  - Permik 14d ago
    Yeah, by implicit ordering, I mean an underspecified pipeline which has an undeclared dependency on some of the inputs being sorted. Unfortunately this is why the code in the cited paper worked on some systems and not others as the os.listdir() call didn't guarantee sorted results and thus the assumed invariants were broken. Requiring code to be explicit about its invariants by default is just plain good design and makes things more robust. And when being more exact about invariants, you can reap the algorithmic benefits.
sanderjd 14d ago
Is "non-deterministic" the right description for this? I read it as describing an implementation where ordering is not preserved, but deterministically. Is that a misreading?
- dhampi 14d ago
  For example, polars internally hashes rows for some operations in ways that affect the ultimate ordering. They do not guarantee stability of hashing algorithm or seed across versions and platforms. (No complaints here! I agree ordering shouldn’t matter unless you make it explicit.)
nemothekid 14d ago
If the correctness of my program depends on the ordering of data (for operations that would otherwise be commutative), that seems like that should be something explicit rather than implicit. I'm not sure if I agree that "hidden setting actually keeps your data correct" is something that should be the default.
0cf8612b2e1e 14d ago
This behavior has repeatedly frustrated me. I am writing some new transformation, want to see the results, and my first few sentinel rows are nowhere to be seen because they have been shuffled. I do not think of a dataframe as a set, but an ordered collection of rows. My source csv had the rows in this order and I want that maintained unless I choose maximum performance.
- geysersam 14d ago
  Why not slap an order_by on the end of the query and you're all good? If that's expensive maybe you can add a limit clause to your source.
  - 0cf8612b2e1e 14d ago
    Sure, I can change what I get, I just disagree that my ordered table is not guaranteed to stay ordered. Neither R nor Pandas will shuffle results by default.
dash2 14d ago
I once persuaded the dplyr maintainers not to do an update that might re-order rows after a filter(). I think the human tendency to think of database rows as existing in a fixed, given order, which will only be changed explicitly, is deep.
jorelfermin 13d ago
Unfortunately (for your case) the ordering of group_by, join, and unique all run in parallel hash aggregation across the threads so the output order comes about by how it gets partitioned across cores. Which is why you can get different order of rows depending on the machine even when you have the same set of data and polars version (this has happened to me). To fix you can set maintain_order=True or probably better an explicit sort whereever you save or compare / diff the output.

winniewinnie 14d ago

Perhaps to get better results on benchmarks.

Bootvis 14d ago
Or you know, just better performance for people that know how to use their tool of choice.
- winniewinnie 14d ago
  Wouldn't such people just pass in maintain_order=False ?
  - Bootvis 14d ago
    That’s an API design question. What is the more common use case. You seem to suggest they did it for benchmarking reasons only. They could use the option there themselves without changing the default so that is unlikely to be the motivation.

hopfenspergerj 14d ago

It's standard sql behavior, users always specify the ordering they want as part of the query.

Xunjin 14d ago
Often people new to databases and SQL thinks ordering is implicit, I have teach dozens of juniors that believe this is the default. I always wondered why that is the thought process...
- fwip 14d ago
  Because out of the first 100 queries they ran, 99 came back in order and they didn't notice that the 100th didn't.

Permik 14d ago

This is a tricky field, the problem is not actually the non-determinism of the processing algorithms, but implicit ordering of the data.[1] The implicit ordering of the data is a footgun that -- as seen in the paper -- has already claimed victims. Using algorithms that don't need to upkeep the ordinality requirement in every operation will definitely move the library to a better direction and make future data modeling better and more explicit. [1] Aha, now I see why language models use this so frequently and why it might be overrepresented in the data. This is a perfect way to move the blame from the person you're responding to, if they're mistaken. They probably have a super, super overtuned "politeness" gym using sentiment analysis that tries to reword answers to not blame the misunderstandings of the person. Then this blame shifting unfortunately gets re-used as this super, super common phrase.

winniewinnie 14d ago
By "implicit ordering", do you mean "implicitly assumed that the data is ordered a certain way"? Since if that assumption of data being sorted a certain way is broken on some systems and not others, the result might be both non-deterministic (which could be a bug if the result is not allowed to be non-deterministic, but may or may not be a bug regarding the algorithm's assumptions) as well as a bug if the algorithm's assumptions requires it to be sorted a certain way. > Using algorithms that don't need to upkeep the ordinality requirement in every operation will definitely move the library to a better direction and make future data modeling better and more explicit. How would the library "make future data modeling ... more explicit" if this is a change to a default, which is implicit?
- Lvl999Noob 14d ago
  > How would the library "make future data modeling ... more explicit" if this is a change to a default, which is implicit? It would become more explicit because where the order matters, you will now see a `maintain_order=True` where previously, you couldn't say whether it actually needed the fixed order or not.
- Permik 14d ago
  Yeah, by implicit ordering, I mean an underspecified pipeline which has an undeclared dependency on some of the inputs being sorted. Unfortunately this is why the code in the cited paper worked on some systems and not others as the os.listdir() call didn't guarantee sorted results and thus the assumed invariants were broken. Requiring code to be explicit about its invariants by default is just plain good design and makes things more robust. And when being more exact about invariants, you can reap the algorithmic benefits.

sanderjd 14d ago

Is "non-deterministic" the right description for this? I read it as describing an implementation where ordering is not preserved, but deterministically. Is that a misreading?

dhampi 14d ago
For example, polars internally hashes rows for some operations in ways that affect the ultimate ordering. They do not guarantee stability of hashing algorithm or seed across versions and platforms. (No complaints here! I agree ordering shouldn’t matter unless you make it explicit.)

nemothekid 14d ago

If the correctness of my program depends on the ordering of data (for operations that would otherwise be commutative), that seems like that should be something explicit rather than implicit. I'm not sure if I agree that "hidden setting actually keeps your data correct" is something that should be the default.

0cf8612b2e1e 14d ago

This behavior has repeatedly frustrated me. I am writing some new transformation, want to see the results, and my first few sentinel rows are nowhere to be seen because they have been shuffled. I do not think of a dataframe as a set, but an ordered collection of rows. My source csv had the rows in this order and I want that maintained unless I choose maximum performance.

geysersam 14d ago
Why not slap an order_by on the end of the query and you're all good? If that's expensive maybe you can add a limit clause to your source.
- 0cf8612b2e1e 14d ago
  Sure, I can change what I get, I just disagree that my ordered table is not guaranteed to stay ordered. Neither R nor Pandas will shuffle results by default.

dash2 14d ago

I once persuaded the dplyr maintainers not to do an update that might re-order rows after a filter(). I think the human tendency to think of database rows as existing in a fixed, given order, which will only be changed explicitly, is deep.

jorelfermin 13d ago

Unfortunately (for your case) the ordering of group_by, join, and unique all run in parallel hash aggregation across the threads so the output order comes about by how it gets partitioned across cores. Which is why you can get different order of rows depending on the machine even when you have the same set of data and polars version (this has happened to me). To fix you can set maintain_order=True or probably better an explicit sort whereever you save or compare / diff the output.