exAspArk

6 ms·

1.
Show HN: Telio – AI agents for call/text support, built on sandboxed lakehouses(gettelio.com)
2 points by exAspArk 8mo ago|0 comments
2.
Show HN: BemiDB – Open-source data warehouse with zero-ETL(bemidb.com)
13 points by exAspArk 1y ago|2 comments
3.
Product Analytics Queries Without Database Meltdown(blog.bemi.io)
3 points by exAspArk 1y ago|0 comments
4.
by exAspArk 1y ago
There are a few different approaches. The main categories, from simplest to most complex: 1) Read replicas with copied data. The most straightforward, allowing using the same SQL syntax and tooling. Examples: Postgres read replica and BemiD
5.
Cloud Data Analytics Is a Scam(blog.bemi.io)
4 points by exAspArk 2y ago|0 comments
6.
Data Analytics with PostgreSQL: The Ultimate Guide(blog.bemi.io)
2 points by exAspArk 2y ago|0 comments
7.
BemiDB – Zero-ETL Analytics on Postgres(bemidb.com)
5 points by exAspArk 2y ago|1 comments
8.
by exAspArk 2y ago
Hey HN! We’re Evgeny and Arjun. We’ve built a managed version of BemiDB that syncs with your existing PostgreSQL database and gives you fast analytical queries without heavy ETL pipelines. BemiDB Cloud automatically replicates your data int
9.
by exAspArk 2y ago
Our plan is to make BemiDB work with dbt by leveraging the Postgres-compatibility (supported dbt adapters https://docs.getdbt.com/docs/trusted-adapters ). So it should be possible to transform data from Postgres or dire
10.
by exAspArk 2y ago
This is a great DIY setup. We're hoping to compress this stack and simplify it down to a single binary
11.
by exAspArk 2y ago
Yes! BemiDB natively supports two storage layers, a local disk and S3 (we assumed that most people would choose this in production environments to simplify management). When I query Iceberg tables stored on SSD, it works superfast.
12.
by exAspArk 2y ago
Our initial approach was to implement periodic full table re-syncing. We're starting to work on CDC with logical replication for incremental syncing. Here is our roadmap https://github.com/BemiHQ/BemiDB#future-road
13.
by exAspArk 2y ago
Sorry, we haven't benchmarked it against ClickHouse yet. Our initial point of reference was just Postgres
14.
by exAspArk 2y ago
We haven't tested this with 1TB Postgres databases yet, assuming that most companies operating at this scale already built analytics data pipelines :) I'm curious if you currently move the data from this Postgres to somewhere else
15.
by exAspArk 2y ago
Our initial approach is to do full table re-syncs periodically. Our next step is to enable incremental data syncing by supporting insert/update/delete according to the Iceberg spec. In short, it'd produce "diff" Par
16.
by exAspArk 2y ago
That's why our current approach is to build missing or not fully functional features ourselves to move fast. For example, DuckDB performs reads from Iceberg tables not according to the spec, can't perform writes, etc.
17.
by exAspArk 2y ago
Great ideas! We'll keep this suggestion related to read/write separation in mind. We started with a simple unified solution, but we'll keep iterating, listening and addressing any feedback :)
18.
by exAspArk 2y ago
My few cents: - Compute and storage separation simplifies managing a system making compute "ephemeral" - Compute resources can be scaled separately without worrying about scaling storage - Object storage provides much higher durab
19.
by exAspArk 2y ago
I'd say that querying data from S3 is not ideal when low-latency queries are required. Generally, there could be a few roundtrip requests to fetch metadata (JSON, Avro) and data (Parquet) files, which may lead to around 1s or so latenc
20.
by exAspArk 2y ago
Oh, interesting, thanks for sharing it!
21.
by exAspArk 2y ago
Good point. For more complex scenarios, people would still be able to implement, for example, a Medallion Architecture to progressively improve data quality and structure. Because it is Postgres- and Iceberg-compatible (db and data), it&#x2
22.
by exAspArk 2y ago
Haha, it's awesome for isolating project environments (languages, databases, etc.) without using docker
23.
by exAspArk 2y ago
Would you be able to share how you implemented "bulk-appended using a separate infrastructure" at a high level?
24.
by exAspArk 2y ago
Iceberg for the win! We actually separate Read/Write paths. BemiDB reads by levering DuckDB as a query engine. And it writes to Iceberg completely separately from DuckDB. I'm curious if that's what you imagined.
25.
by exAspArk 2y ago
Exactly! You can run it on any server connecting to any Postgres, without installing custom extensions (AWS Aurora supports only a limited number of extensions https://docs.aws.amazon.com/AmazonRDS/latest/AuroraPos
26.
by exAspArk 2y ago
Our philosophy in general is to go to a more open license over time (vs the other direction). So we might consider other more permissive OSI-approved licenses. Would you be able to share why AGPL license is a no-go for you? I'm genuine
27.
by exAspArk 2y ago
Got it, thanks for sharing it! We'll try to look into DBT-3 and the indexes it creates to test with SF10
28.
by exAspArk 2y ago
Interesting. I haven't used the DBT-3 kit, does it add any indexes? I manually added these Postgres indexes https://github.com/BemiHQ/BemiDB/blob/main/benchmark/data/cr... to reduce the ma
29.
by exAspArk 2y ago
Thank you, please give it a try! Great questions: 1. We currently don't support multi-dimensional arrays, but we plan to add support for such complex data structures. 2. Would you be able to share what type of user-defined functions ar
30.
by exAspArk 2y ago
Full table re-syncing is our initial solution. Using Postgres logical replication is next on our roadmap!

More ›