10 ms·

I agree with this 100%. The creator of duckdb argues that people using pandas are missing out of the 50 years of progress in database research, in the first 5 m

by mrtimo 1y ago

I agree with this 100%. The creator of duckdb argues that people using pandas are missing out of the 50 years of progress in database research, in the first 5 minutes of his talk here [1]. I've been using Malloy [2], which compiles to SQL (like Typescript compiles to Javascript), so instead of editing a 1000 line SQL script, it's only 18 lines of Malloy. I'd love to see a blog post comparing a pandas approach to cleaning to an SQL/Malloy approach. [1] https://www.youtube.com/watch?v=PFUZlNQIndo https://www.youtube.com/watch?v=PFUZlNQIndo [2] https://www.malloydata.dev/ https://www.malloydata.dev/

esafak 1y ago
Have you used Malloy in a pipeline, e.g., with Airflow? If so, how was the experience?
fumeux_fume 1y ago
In the same talk, Mark acknowledges that "for data science workflows, database systems are frustrating and slow." Granted DuckDB is an attempt to fix that, most data scientists don't get to choose what database the data is stored in.
- willvarfar 1y ago
  (I use duckdb to query data stored in parquet files)
  - mrtimo 1y ago
    Same. But, I use Malloy which uses duckdb to query data stored in hundreds of parquet files (as if they were one big file).
    - willvarfar 1y ago
      I haven't looked at Mallory, but I do regularly scan lots of parquet files using wildcards etc from duckdb. Its a neat builtin duckdb feature.
orlp 1y ago
> The creator of duckdb argues that people using pandas are missing out of the 50 years of progress in database research, in the first 5 minutes of his talk here. That's pandas. Polars builds on much of the same 50 years of progress in database research by offering a lazy DataFrame API which does query optimization, morsel-based columnar execution, predicate pushdown into file I/O, etc, etc. Disclaimer: I work for Polars on said query execution.
- entropicdrifter 1y ago
  Just wanted to say I'm a huge fan of your work. Been using Polars for my team's main project for years and it just keeps getting better.
- phailhaus 1y ago
  The DataFrame interface itself is the problem. It's incredibly hard to read, write, debug, and test. Too much work has gone into reducing keystrokes rather than developing a better tool.
  - dev_l1x_be 1y ago
    Not sure what you mean by this. The table concept is the same age as computers. Here is a table, do something with it -> this is the high level df api. All the functions make sense, what is hard to read, write or debug here? I have used Polars to process 600M of xml files (with a bit of a hack) and the polars part of the code is readable with minimal comments. Polars has a better api than pandas, at least the intent is easier to understand. (lazyness, yay)
    - phailhaus 1y ago
      The problem with the dataframe API is that whenever you want to change a small part of your logic, you usually have to rethink and rewrite the whole solution. It is too difficult to write reusable code. Too many functions that try to do too many things with a million kwargs that each have their own nuances. This is because these libraries tend to favor fewer keystrokes over composable design. So the easy stuff is easy and makes for pretty docs, but the hard stuff is obnoxious to reason through. This article explains it pretty well: https://dynomight.net/numpy/ https://dynomight.net/numpy/

by mrtimo 1y ago

esafak 1y ago

Have you used Malloy in a pipeline, e.g., with Airflow? If so, how was the experience?

fumeux_fume 1y ago

In the same talk, Mark acknowledges that "for data science workflows, database systems are frustrating and slow." Granted DuckDB is an attempt to fix that, most data scientists don't get to choose what database the data is stored in.

willvarfar 1y ago
(I use duckdb to query data stored in parquet files)
- mrtimo 1y ago
  Same. But, I use Malloy which uses duckdb to query data stored in hundreds of parquet files (as if they were one big file).
  - willvarfar 1y ago
    I haven't looked at Mallory, but I do regularly scan lots of parquet files using wildcards etc from duckdb. Its a neat builtin duckdb feature.

orlp 1y ago

> The creator of duckdb argues that people using pandas are missing out of the 50 years of progress in database research, in the first 5 minutes of his talk here. That's pandas. Polars builds on much of the same 50 years of progress in database research by offering a lazy DataFrame API which does query optimization, morsel-based columnar execution, predicate pushdown into file I/O, etc, etc. Disclaimer: I work for Polars on said query execution.

entropicdrifter 1y ago
Just wanted to say I'm a huge fan of your work. Been using Polars for my team's main project for years and it just keeps getting better.
phailhaus 1y ago
The DataFrame interface itself is the problem. It's incredibly hard to read, write, debug, and test. Too much work has gone into reducing keystrokes rather than developing a better tool.
- dev_l1x_be 1y ago
  Not sure what you mean by this. The table concept is the same age as computers. Here is a table, do something with it -> this is the high level df api. All the functions make sense, what is hard to read, write or debug here? I have used Polars to process 600M of xml files (with a bit of a hack) and the polars part of the code is readable with minimal comments. Polars has a better api than pandas, at least the intent is easier to understand. (lazyness, yay)
  - phailhaus 1y ago
    The problem with the dataframe API is that whenever you want to change a small part of your logic, you usually have to rethink and rewrite the whole solution. It is too difficult to write reusable code. Too many functions that try to do too many things with a million kwargs that each have their own nuances. This is because these libraries tend to favor fewer keystrokes over composable design. So the easy stuff is easy and makes for pretty docs, but the hard stuff is obnoxious to reason through. This article explains it pretty well: https://dynomight.net/numpy/ https://dynomight.net/numpy/