zhousun

7 ms·

1.
by zhousun 1y ago
author here. I would actually second you. My core belief is "it is possible to build true database-like functionalities on top of iceberg", but it is definitely not 'easier' than building them directly in a db (in fact,
2.
by zhousun 1y ago
there's actually a great read, cursor started with a distributed OLTP solution: yugabyte, and then fall back to RDS...
3.
by zhousun 1y ago
Using SQL as catalog is not new (iceberg supports JDBC catalog from the very beginning). The main difference is to store metadata and stats also directly in SQL databases, which makes perfect sense for smaller scale data. In fact we were do
4.
by zhousun 1y ago
DataFile(parquet) is not enough for table with update/delete, (they are part of iceberg "metadata"). for CDC from OLTP use-cases, the pattern involves rapidly marking rows as deleted/ insert new rows and optimizing small
5.
by zhousun 1y ago
Glab to see more 'postgres-native' full-text search implementation. Alternative solutions (lucene/ tantivy) are both designed for 'immutable segments' (indexing immutable files), so marrying them with postgres heap
6.
by zhousun 2y ago
lol spot-on comment and stay tuned for our v2 :) The focus of mooncake is to be a columnar storage engine, that natively integrate with pg, allowing writing from pg, replicating from pg, and reading by pg using pg_mooncake. We want people
7.
by zhousun 2y ago
Yep what I want say is the line between the two designs is indeed very blur. Logical replication with mooncake will try to create a columnar version of a postgres heap table, that can be readable within postgres (using pg_mooncake); or outs
8.
by zhousun 2y ago
Yea this is indeed a repeated pattern we saw people requesting (filter on many columns) and we are trying to solve with pg_mooncake. If you are interested, feel free to join mooncake-devs.slack.com to chat more about your use case.
9.
by zhousun 2y ago
people tried to run spark (better hadoop) and failed lol. https://github.com/ClickHouse/ClickBench/pull/139
10.
by zhousun 2y ago
Thanks for the comment but you are mixing some terminologies. The core idea of mooncake is to built upon open columnar format + substitutable vectorized engine, while natively integrate with Postgres. So it is indeed closer to BigQuery (esp
11.
by zhousun 2y ago
hydra/pg_duckdb embeds duckdb to query existing data on S3. So it is kind of targeting a completely different use case (someone already prepares and shared a dataset and you just want to query it). pg_mooncake (&crunchyData) is imp
12.
by zhousun 2y ago
zhou from mooncake labs here. Good point! Normally for postgres extension it won't be solvable, but for mooncake it is actually not the case! The core idea of mooncake is to built upon open columnar format + substitutable vectorized en
13.
by zhousun 2y ago
transactions are also managed by postgres as if they are native table, so that you don't need to worry about coordinating commits between postgres and the S3 data.
14.
by zhousun 2y ago
Interesting enough, the mooncake team was building SingleStore before, and so far it is the best production-ready HTAP system. One lesson I really learned is, people don't want to switch their system-of-record OLTP system. Spoiler-aler
15.
by zhousun 2y ago
Zhou from mooncake labs here. Mooncake is built upon open-table formats and substitutable query engines. So it don't need to be just a postgres extension. PG_mooncake will stay open-source under MIT, for small devs where everything fit
16.
by zhousun 2y ago
Hi, Zhou From Mooncake labs here. Love your work on PeerDB and it's inspiring the evolvement of pg_mooncake (logical replication will be the killing feature for V2) The core idea of mooncake is to built upon open columnar format + subs
17.
by zhousun 2y ago
The only datastack iceberg (or lakehouse) will never replace is OLTP systems, for high-concurrency updates optimistic concurrency control & object store is simply a no go. Iceberg out-of-the-box is "NOT" good at streaming use
18.
WTH are Amazon S3 Tables? And why they are NOT useful for enterprises(mooncake.dev)
5 points by zhousun 2y ago|0 comments
19.
by zhousun 2y ago
It's such an honor our https://pgmooncake.com/ is covered in the review! A little sad Andy didn't share more of his thoughts on the intersection between Data and AI, and how that's going to evolve.