chrisjc

7 ms·

1.
by chrisjc 1y ago
Any thought about a pass-through server for various non-local databases? It should be extremely simple for databases that support ADBC (for example Snowflake, PostgreSQL). For others it might just be a matter of mapping DDL, DML, DQL, etc t
2.
by chrisjc 2y ago
duckdb -cmd 'CREATE TABLE my_data AS FROM READ_PARQUET($$data.parquet$$)' -ui `duckdb -ui sqlitedb.db` should work bc duckdb can read sqlite files. If it doesn't autoload extension, you can INSTALL/LOAD in to your ~&#x2F
3.
by chrisjc 2y ago
duckdb -ui pre_seeded_db.db duckdb -ui -init startup.sql where startup.sql contains various statements/commands like `.open pre_seeded_db.db` Alternatively place statements/commands in `~/.duckdbrc` and just ru
4.
by chrisjc 2y ago
> In the worst-case scenario, they would be 2-3 years behind the cutting edge, which is not mission-critical. It's also worth considering how such an event might affect the US and allies. Would it slowdown, perhaps even halt certain
5.
by chrisjc 2y ago
Haven't tried it out yet, but the release notes look very promising esp relating to Arrow interactions - Push dynamically generated join filters through UNION, UNNEST and AGGREGATE - Fix arrow table filters - [Arrow] Fix scan of an obj
6.
by chrisjc 2y ago
Perhaps I don't fully understand what you're saying CREATE TABLE person (name VARCHAR, age BIGINT); INSERT INTO person VALUES ('Alice', 37), ('Ana', 35), ('Bob', 41), ('Bea', 25);
7.
by chrisjc 2y ago
I've been eagerly awaiting this for a couple of months now. And I've long wondered why there hasn't been such an implementation/extension for Flight especially as there are extensions for mysql, postgres, sqlite... Only
8.
by chrisjc 2y ago
For read-oriented interactions... ADBC --sql--> Flight SQL Server --flight--> Flight Servers (plural) --> datasources or SELECT * FROM <relation> WHERE ... where the "relation" is a collection of A
9.
by chrisjc 2y ago
Thank you for all the work you guys do. The Arrow ecosystem is just absolutely incredible. My few gripes related to interop with duckdb are related to Arrow scanning/pushdowns. And this extends to interop with other projects like pyice
10.
by chrisjc 2y ago
And all the Arrow parts work together quite nicely. ADBC client --> Flight SQL (duckdb/whatever) --> Flight --> ? The result highlights your exact point: why take your data and transpose it twice? It's quite an e
11.
by chrisjc 2y ago
Since you're using python, have you looked into sqlglot? I think it has some pretty-print options. https://github.com/tobymao/sqlglot
12.
by chrisjc 2y ago
Maybe something like this? https://duckdb.org/2024/10/02/pyodide.html
13.
by chrisjc 2y ago
Probably also worth mentioning that DuckDB can interact with SQLite dbs. https://duckdb.org/docs/extensions/sqlite.html https://duckdb.org/docs/guides/database_integration/sqlite.h..
14.
by chrisjc 2y ago
Probably not exactly what you mean or asking for, but the work Motherduck is doing looks promising. https://motherduck.com/blog/differential-storage-building-bl... Hopefully it finds its way into duckdb's repo som
15.
by chrisjc 2y ago
Ibis looks very promising. There are also other ways to use both. https://duckdb.org/docs/guides/python/polars.html All of this dataframe compatibility is awesome. (much thanks to Arrow and others)
16.
by chrisjc 2y ago
Perhaps not exactly what you're talking about, but maybe? (unsure bc the with statements are sometimes called "temp tables") https://duckdb.org/docs/sql/query_syntax/with#cte-materializ... Obvi
17.
by chrisjc 2y ago
Although only experimental and probably off topic to the discussion, it's worth mentioning DuckDB also provides a Spark API implementation. https://duckdb.org/docs/api/python/spark_api And while on the s
18.
by chrisjc 2y ago
Not the original parent, so unsure of their use-case. But I've seen the approach where some/basic development can be done on duckdb, before making its way to dev/qa/prd. Something like your project might enable grabbing
19.
by chrisjc 2y ago
Real awesome project. This would be even better if some of the differential storage functionality makes its way into duckdb. https://motherduck.com/blog/differential-storage-building-bl... Unsure if it's even poss
20.
by chrisjc 2y ago
Perhaps similar to https://github.com/duckdb/dbt-duckdb , but SQLMesh instead of DBT obviously.
21.
by chrisjc 2y ago
It should be just as easy as adding: authenticator="externalbrowser" Adding a Snowflake connection configuration option that allowed for standard Snowflake connection conventions might be a good option. That way you could
22.
by chrisjc 2y ago
Yes, DuckDB does do Iceberg. https://duckdb.org/docs/extensions/iceberg
23.
by chrisjc 2y ago
Interested to hear why you switched from DataFusion?
24.
by chrisjc 2y ago
DuckDB is really for OLAP/Columnar (analytical workloads) whereas SQLite is more for OLTP/Row-based (transactional). When you start running aggregations/windows over large amounts of data, you'll soon see the difference
25.
by chrisjc 2y ago
Via the AWS extension https://duckdb.org/docs/guides/network_cloud_storage/s3_impo... https://duckdb.org/docs/extensions/aws
26.
by chrisjc 2y ago
DuckDB can do some pushdowns to certain file formats like parquet, but every release seems to be getting better and better at doing it. Parquet pushdowns combined with Hive structuring is a pretty good combination. There are some HTTP and M
27.
by chrisjc 2y ago
DuckDB? https://duckdb.org/2024/03/29/external-aggregation.html https://duckdb.org/2021/06/25/querying-parquet.html If your DB is mysql or postgres, then you could read a strea
28.
by chrisjc 2y ago
In my experience, it's always come down to either using a specific database type for the wrong job, or using a database type the wrong way. - Treating an OLAP like an OLTP. - Using a document store for relational data bc managing relat
29.
by chrisjc 2y ago
Keep an eye on this link. New support is being added every so often. https://github.com/davidgasquez/awesome-duckdb?tab=readme-ov...
30.
by chrisjc 2y ago
Though it certainly takes care of the usual "We reached out to ???, but they didn't respond" statement.

More ›