6 ms·
DuckDB and Polars are my bets in the Python data-wrangling space. I grew tired of Pandas' weird-ass API.
by mgradowski 5y ago
DuckDB and Polars are my bets in the Python data-wrangling space. I grew tired of Pandas' weird-ass API.
- elforce002 5y agoI don't know DuckDB but polars could dethrone pandas. We're planning on using it to create our pipeline. Ibis-project is another solution if anyone wants to check it out.
- mgradowski 5y agoHuh, even though I would prefer a universal SQL layer, ibis looks quite nice.
- elforce002 5y agoI mean I haven't heard about DuckDB.
- spaniard89277 5y agoI haven't touched pandas in months, but I also found quite tiring to deal with pandas. Does your setup allow for an end-to-end solution? I mean, can I sink time into that setup and feel like I have everything I need to for regular data-wrangling? I'm sure Pandas is amazing, but as a newbie I found myself doing many transformation logic with python data structures because it's just so much easier. Maybe I'm dumb but going around the docs sometimes was like :/
- closed 5y agoAuthor of the post and siuba here. I'm pretty interested in exploring supporting polars as a backend, and if it works well supporting versions of the SQL backends that translate to SQL based on the polars method API :). (I haven't really used it, but it looks promising)
- mrtranscendence 5y agoHey, I love siuba. Haven't had a chance to use it much but it scratches an itch for me. For years I've grumbled about how Python isn't flexible enough to accommodate tidyverse style libraries, as it lacks pipes and lazy evaluation (or macros), but siuba has managed to be very nice to use. Maybe someday Python'll get a macro system ...
- sweezyjeezy 5y agoI would love to switch to something else, but it feels like pandas is lingua-franca in data science now, to switch puts a burden on everyone else.
- mytherin 5y agoYou can use DuckDB as a processing engine on top of Pandas [1], while continuing to use Pandas as a data storage/data interchange format. [1] https://duckdb.org/2021/05/14/sql-on-pandas.html https://duckdb.org/2021/05/14/sql-on-pandas.html
- mgradowski 5y agoThat's what I do at $dayjob whenever I have to do windowing &c. Figuring out this stuff in Pandas is a waste of time. Before I discovered DuckDB, I would re-learn the API every damn time. I came up with a little utility function, which you can implement yourself :) ``` def sqldf(df: DataFrame, query: str) -> DataFrame: ... ```
- kristjansson 5y agoYears of unpicking others use of Rs sqldf (which by default used to copy the entire data frame to a SQLite db, run the query, the copy the result set back) when they complained their R code was to slow has taught me a visceral, negative to the name and pattern. Glad to to see duckDB delivering, finally, on the promise of running SQL against in-memory dataframes
- mgradowski 5y agoTIL there's an actual 'botched' library with the same name; I actually came up with it independently on a lazy office afternoon :^)
- mgradowski 5y agoI like interface-only packages in the Julia ecosystem e.g. Tables.jl enables the development of several packages for querying tabular data that work across many concrete implementations; Plots.jl separates the high-level plotting interface from the plotting backend.
- anonymousDan 5y agoYes I used it for the first time in ages recently and I have to say I found the whole thing a mess. There are about 5 ways to do everything.