mslot

9 ms·

1.
by mslot 1mo ago
Makes sense, we just shipped it https://www.linkedin.com/posts/craigkerstiens_barely-over-2-...
2.
by mslot 1mo ago
The challenge is converting primary key updates/deletes to row offsets in a columnar table. That requires maintaining an expensive mapping or doing expensive scans, and is not something you want Postgres itself to do. It's also t
3.
by mslot 11mo ago
In principle, Postgres has an infinite number of possible types :). pg_lake maps types into their Parquet equivalent and otherwise stores as text representation, there are a few limitations like very large numerics. https://githu
4.
by mslot 11mo ago
(1) We've thought about it, no current plans. We'd ideally reimplement DuckLake in Postgres directly such that we can preserve Postgres transaction boundaries, rather than reuse the Ducklake implementation that would run in a sepa
5.
by mslot 11mo ago
It's the same team and same project :). Crunchy Data was acquired by Snowflake.
6.
by mslot 11mo ago
When we first developed pg_lake at Crunchy Data and defined GTM we considered whether it could be a Snowflake competitor, but we quickly realised that did not make sense. Data platforms like Snowflake are built as a central place to collect
7.
by mslot 11mo ago
Yes, just COPY table TO 's3://mybucket/data.parquet' Or COPY table TO STDOUT WITH (format 'parquet') if you need it on the client side.
8.
by mslot 11mo ago
Definitely similar goals, from the Mooncake author: https://news.ycombinator.com/item?id=43298145 I think pg_mooncake is still relatively early stage. There's a degree of maturity to pg_lake resulting from our team&#x2
9.
by mslot 11mo ago
DuckLake is pretty cool, and we obviously love everything the DuckDB is doing. It's what made pg_lake possible, and what motivated part of our team to step away from Microsoft/Citus. DuckLake can do things that pg_lake cannot do w
10.
by mslot 11mo ago
You can use it as a read layer for for specific metadata JSON URL or a table in a REST catalog. The latter got merged quite recently, not yet in docs.
11.
by mslot 11mo ago
I gave a talk on that at Data Council, then still discussing the pg_lake extensions as part of Crunchy Data Warehouse. https://youtu.be/HZArjlMB6W4?si=BWEfGjMaeVytW8M1 Also, nicer recording from POSETTE: https://
12.
by mslot 11mo ago
There are Postgres roles for read/write access to the S3 object that DuckDB has access to. Those roles can create tables from specific files or at specific locations, and can then assign more fine-grained privileges to other Postgres r
13.
by mslot 11mo ago
You could say With DuckLake, the query frontend and query engine are DuckDB, and Postgres is used as a catalog in the background. With pg_lake, the query frontend and catalog are Postgres, and DuckDB is used as a query engine in the backgro
14.
Postgres, Dbt, and Iceberg: Scalable Data Transformation(crunchydata.com)
4 points by mslot 2y ago|0 comments
15.
by mslot 2y ago
We first launched it as "Crunchy Bridge for Analytics" in April last year. At the time, it could mostly query/import/export Parquet/CSV/JSON. Our goal was to build a data warehouse, but getting early feedback w
16.
by mslot 2y ago
It's a similar idea, but Crunchy Data Warehouse was built by several founding engineers of Citus, which lets us speedrun through it :) It's a generally available (very solid) product powering some large production workloads, with
17.
by mslot 2y ago
I created pg_incremental because I keep running into the same challenge in PostgreSQL: You have a table of raw event data that you insert into, either individual rows when the event happens, or batches of events in other systems. You then m
18.
by mslot 2y ago
Not really. S3 tables provides storage & catalog for Iceberg tables, but is not a query/transaction engine. We'll definitely look for ways to take advantage of it as it matures. I wrote a bit more about it at https:/&#x2
19.
by mslot 2y ago
Azure is not supported as a backend in pg_parquet right now, but shouldn't be hard to add (contributions welcome!) https://github.com/CrunchyData/pg_parquet It would not be safe to let any user access object stora
20.
by mslot 2y ago
Fun fact, I created pg_azure_storage :)
21.
by mslot 2y ago
(Marco from Crunchy Data) With PostgreSQL extensions, we find it's most effective to have single-purpose modular extensions. For instance, I created pg_cron a few years ago, and it's on basically every PostgreSQL service because i
22.
Postgres Powered by DuckDB: The Modern Data Stack in a Box(crunchydata.com)
1 points by mslot 2y ago|0 comments
23.
by mslot 2y ago
Nice! Seems to be pretty well-crafted.
24.
by mslot 2y ago
I think it comes down to every aspect of the DBMS being optimized differently. For instance, UDFs in DuckDB have vectors as input and output, while that would be confusing and unnecessary in PostgreSQL.
25.
Running TPC-H Queries on Iceberg Tables from PostgreSQL(crunchydata.com)
5 points by mslot 2y ago|0 comments
26.
by mslot 2y ago
Nice comparison! Worth noting that pg_cron is available on almost all managed PostgreSQL services. Also, many thanks to Devrim Gunduz and Christoph Berg for providing community packages. I wrote pg_cron with the intention of keeping it as s
27.
Syncing Postgres Partitions to Your Data Lake in Crunchy Bridge for Analytics(crunchydata.com)
1 points by mslot 2y ago|0 comments
28.
Syncing Postgres Partitions to Parquet in S3 in Crunchy Bridge for Analytics(crunchydata.com)
8 points by mslot 2y ago|0 comments
29.
by mslot 2y ago
Crunchy Bridge is similar to RDS. It runs on EC2 and is a mature managed service with features such as VPC peering, and now analytics, so you could consider it as an alternative.
30.
by mslot 2y ago
Crunchy Bridge is a managed PostgreSQL service by Crunchy Data available on AWS, Azure, and GCP. Bridge for Analytics is a special instance/cluster type in Crunchy Bridge with additional extensions and infrastructure for querying data

More ›