AdamProut

6 ms·

1.
by AdamProut 6d ago
I'm curious why the test needed so many router hosts: 512 shards, each with one Postgres primary each on an r8g.16xlarge 480 Neki routers, each on its own 8xlarge instance That's ~250K queries/sec per router which seems lowis
2.
by AdamProut 6d ago
yeah, this is a definitely a "best case" workload for a sharded database. Single row reads on the key used to shard with no hotspots (no shard to shard network traffic at all).
3.
by AdamProut 13d ago
I'll admit to not reading your code. It's hard to keep up with all the different databases launched in the last decade. I have doubts that a global readwrite lock around a hashtable makes for a good general purpose storage system
4.
by AdamProut 13d ago
what kind of maintenance? I think one of the maybe less talked about benefits of distributed SQL is support for nearly transparent rolling upgrades of the database with very little impact to a running workload. Spanner is best in class at
5.
by AdamProut 13d ago
The intro section is a good summary of why distributed SQL databases (Spanner, roach, Yugabyte, TiDB) haven't taken off in the market in the same way as say distributed data warehouses have (Snowflake, Databricks, FabricDW, Clickhouse,
6.
Agents Are Better Testers Than We Are(medium.com)
1 points by AdamProut 5mo ago|1 comments
7.
by AdamProut 5mo ago
Agents should be driving up the quality of software if we focus them on finding and fixing bugs.
8.
by AdamProut 2y ago
Very cool! pg_duckdb itself is missing fully integrated storage - it can query data stored externally (say in S3) in delta/iceberg formats, but it can't write out data in those formats via transactional writes to PG tables (insert
9.
by AdamProut 2y ago
Had a similar thought. Azure Postgres has something similar to pg_parquet (pg_azure_storage), but we're looking into replacing it with pg_duckdb assuming the extension continues to mature. It would be great if the Postgres community c
10.
by AdamProut 2y ago
Checksums can detect a torn page, but not always repair them. It's likely a good part of the database page is gone (i.e., an amount of data that matches the disk / file system atomic write unit size is probably missing). Torn pag
11.
by AdamProut 2y ago
Maybe good to mention torn pages somewhere too? Both MySQL and Postgres jump through some hoops to both detect them and repair them [1][2]. So, even the scenario in the post where fsync is used to harden writes, the database still needs to
12.
by AdamProut 2y ago
I'm not sure why you're getting downvoted. I think the pressure to appease Gartner usually starts when companies bring in CEOs whose primary background is enterprise sales. They tend to over value magic quadrant positioning (in
13.
Does MongoDB Support ACID Transactions?(medium.com)
1 points by AdamProut 3y ago|2 comments
14.
by AdamProut 3y ago
Based on the writings of 2 Turing awards winners for work on databases; No MongoDB doesn't have ACID transactions.
15.
by AdamProut 3y ago
Control theory is also used by databases (probably not as often as it should be). It's great for "self tuning" [1], for example tuning the various cache sizes a database has to maximize throughput under changing workload cond
16.
by AdamProut 3y ago
RE: "Distributed SQL offers elasticity (no downtime resize"). I'm not sure this is as much of an advantage of distributed databases vs single host databases anymore. Some of the tech to move virtual machines between machines
17.
by AdamProut 3y ago
Databricks has kept their Photon[1][2] query engine for Spark closed sourced thus far. Unless EMR has made equivalent changes to the Spark runtime they use Databricks should be much faster. Photon brings the standard vectorized execution t
18.
by AdamProut 3y ago
yeah, that is part of the trade off. Using an ephemeral SSD (for a database) means the database needs to have another means of making the data durable (replication, storing data in S3, etc.). There are AWS instance types (I3en) with large
19.
by AdamProut 3y ago
Its a popular design for SQL Data warehouses. I think almost all of them (snowflake, redshift, etc.) store cold data in S3 and hot data on local disk[1][2]. It works well if the data is stored as immutable files (i.e., A log structure mer
20.
by AdamProut 3y ago
Yeah for workloads with any long running write transactions a single writer design is a pretty big limitation. Say some long running data load (or a big bulk deletion) running along with some faster high throughput key value writes - the bi
21.
by AdamProut 3y ago
This may depend on if your use case is only vector search in isolation (ANN lookups). In this scenario pgvector is potentially not the best option ( https://ann-benchmarks.com/ ) That said, using pgvector (or using other
22.
by AdamProut 4y ago
Yep, SingleStoreDB (formerly MemSQL) is not open source (probably never will be), but it does have many paying customers who have had workloads in production for over a decade at this point. Also, isn't taking a bet on a very recently
23.
by AdamProut 4y ago
Most SQL Analytical databases don't discourage joins (think BigQuery, Redshift, Snowflake, etc.) and all of them are columnstores. I think discouraging joins is something very specific to Clickhouse, Druid, Pinot and others that have v
24.
by AdamProut 4y ago
Yep, for writes network bandwidth usage is independent of separation of storage of compute in some sense. Any database that provides high availability is writing over the network somewhere before it acks a transaction committed. It matter
25.
by AdamProut 4y ago
There are hybrid designs for separation of storage and compute that are aimed at mixed workloads[1]. They avoid writes to remote storage on transaction commit (i.e., act like a shared nothing databases for commits, but still push data asy
26.
by AdamProut 4y ago
Columnstores can do row level access by trading off a bit in terms of compression. If you organize the columnstore files as an LSM tree and use incremental compression schemes (so you don't have to compress too many more rows then the
27.
by AdamProut 4y ago
I read the article. And yes, this was probably a mistake to engage with you on it. Yandex N.V. (A dutch holding company of Yandex Russia) is listed as an investor of Clickhouse Inc. This is all I'm stating.
28.
by AdamProut 4y ago
Because half truths like your statements bother me? Yandex is a large investor in clickhouse Inc. (at last as far as public record shows). Have they forced Yandex to divest their holdings?
29.
by AdamProut 4y ago
Yandex (Google of Russia) is a large share holder of Clickhouse Inc[1]. Yes, this maybe a European Yandex subsidiary.. I really don't care to dig, but I think its dishonest to say the company doesn't have Russian backers. Has Cli
30.
by AdamProut 4y ago
Almost no one does "official" TPC results these days (maybe other then Oracle and some of the Chinese vendors). Most other cloud DWs have public TPC-H or TPC-DS results that are easily googlable. Clickhouse is missing for a reaso

More ›