Sirupsen

5 ms·

1.
Rust zero-cost abstractions vs. SIMD(turbopuffer.com)
24 points by Sirupsen 7mo ago|4 comments
2.
SpacetimeDB: A Short Technical Review(strn.cat)
9 points by Sirupsen 7mo ago|0 comments
3.
by Sirupsen 7mo ago
Yep! Other than random reads (~p99=200ms on larger ranges), it's essential to get good download performance of a single file. A single (range) request can "only" drive ~500 MB/s, so you need multiple offsets. https:&#x2
4.
by Sirupsen 7mo ago
Due to the batching, this will only consume a few million class B per month. They are $5/million
5.
by Sirupsen 7mo ago
(cofounder of tpuf here) We don't have a relational database, otherwise that would work great for a queue! You can imagine us continuing to iterate here to Step 5, Step 6, ... Step N over time. The tradeoff of each step is complexity,
6.
A distributed queue in a single JSON file on object storage(turbopuffer.com)
171 points by Sirupsen 7mo ago|55 comments
7.
Vectorized MAXSCORE over WAND, especially for long LLM-generated queries(turbopuffer.com)
4 points by Sirupsen 9mo ago|0 comments
8.
by Sirupsen 10mo ago
Our query planner has that built in! We've spent a lot of time making high recall with any selectivity in the fitler work.
9.
by Sirupsen 2y ago
The short of it is that building a database on top of object storage has generally required a complicated, distributed system for consensus/metadata. CAS makes it possible to build these big data systems without any other dependencies.
10.
by Sirupsen 2y ago
To avoid any dependencies other than object storage, we've been making use of this in our database (turbopuffer.com) for consensus and concurrency control since day one. Been waiting for this since the day we launched on Google Cloud
11.
Amazon S3 Adds Put-If-Match (Compare-and-Swap)(aws.amazon.com)
524 points by Sirupsen 2y ago|160 comments
12.
by Sirupsen 2y ago
It works great. We’ve had SPANN in production since October of 2023 at https://turbopuffer.com/
13.
by Sirupsen 2y ago
tpuf’s ANN index uses a variant of SPFresh, yup. These are the only two production implementations I am aware of. I don’t think it is in production at MSFT yet
14.
by Sirupsen 2y ago
Ya, the world needed S3 to become fully consistent. This didn't happen until end of 2020!
15.
by Sirupsen 2y ago
Yeah, thinking about this more I now understand Clickhouse to be more of an operational warehouse similar to Materialize, Pinot, Druid, etc. if I understand correctly? So bunching with BigQuery/Snowflake/Trino/Databricks... w
16.
Turbopuffer: Fast search on object storage(turbopuffer.com)
379 points by Sirupsen 2y ago|64 comments
17.
by Sirupsen 3y ago
Most production storage systems/databases built on top of S3 spend a significant amount of effort building an SSD/memory caching tier to make them performant enough for production (e.g. on top of RocksDB). But it's not easy t
18.
by Sirupsen 3y ago
Emil if you email me at info@turbopuffer.com I can let you into the alpha :)
19.
by Sirupsen 3y ago
While I never owned a VanMoof, I read they painted pictures of flat screen TVs on the shipping boxes after months of issues with bikes being damaged by the time they got to customers. Clever!
20.
by Sirupsen 4y ago
Are you aware of a good write-up on how Clickhouse/other columnar databases do the intersection?
21.
by Sirupsen 4y ago
MySQL can do skip-scans https://dev.mysql.com/doc/refman/8.0/en/range-optimization.h...
22.
by Sirupsen 4y ago
Ideally, I would add three graphs to the post: (1) Table size on the x-axis, and time on the y-axis for index merge vs composite index (2) Number of columns on the x-axis, and time on the y-axis for both (3) Number of final matches on the x
23.
by Sirupsen 4y ago
You're absolutely right! I forgot to move this around when I updated the article's structure. This is only relevant when doing the index merge. The article has been updated
24.
by Sirupsen 4y ago
Excellent, thank you! I'll add that to the article.
25.
by Sirupsen 4y ago
I've added this to the article, thanks! Composite index (int64, int64): ~70 MiB in Postgres, ~350 MiB in MySQL Single index (int64): ~70 MiB in Postgres, ~240 MiB in MySQL If you assume the majority of an index are index entries of (in
26.
Index Merges vs. Composite Indexes in Postgres and MySQL(sirupsen.com)
175 points by Sirupsen 4y ago|38 comments
27.
by Sirupsen 4y ago
You might be very right about that! It's been a while since I did the SSD benchmarks. Glad to hear it's most likely entirely accurate at 4 GiB/s then!
28.
by Sirupsen 4y ago
`madvise(2)` doesn't matter _that_ much in my experience with [1] on modern Linux kernels. SSD just can't read _quite_ as quickly as memory in my testing. Sure, SSD will be able to re-read a lot into ram, analogous to how memory r
29.
by Sirupsen 4y ago
Yes, sequential I/O bandwidth is closing the gap to memory. [1] The I/O pattern to watch out for, and the biggest reason why e.g. databases do careful caching to memory, is that _random_ I/O is still dreadfully slow. I/O
30.
by Sirupsen 4y ago
We spent a fair amount of time trying to get this executing fast on SQL. However, with spreadsheets you recursively compute a lot of values. You can do this in SQL with recursive CTEs, but it’s slow, they’re not optimized for this. They als

More ›