6 ms·
Turbopuffer: Fast search on object storage
- drodgers 2y agoI love the object-storage-first approach; it seems like such a natural fit for the could.
- softwaredoug 2y agoHaving worked with Simon he knows his sh*t. We talked a lot about what the ideal search stack would look when we worked together at Shopify on search (him more infra, me more ML+relevance). I discussed how I just want a thing in the cloud to provide my retrieval arms, let me express ranking in a fluent "py-data" first way, and get out of my way My ideal is that turbopuffer ultimately is like a Polars dataframe where all my ranking is expressed in my search API. I could just lazily express some lexical or embedding similarity, boost with various attributes like, maybe by recency, popularity, etc to get a first pass (again all just with dataframe math). Then compute features for a reranking model I run on my side - dataframe math - and it "just works" - runs all this as some kind of query execution DAG - and stays out of my way.
- snthpy 2y agoCould you give an example of what you mean by _fluent "py-data" first way_ ? You mean like a fluent API like `data.transform().filter()...` , that sort of thing?
- bkitano19 2y ago+1, had the fortune to work with him at a previous startup and meetup in person. Our convo very much broadened my perspective on engineering as a career and a craft, always excited to see what he's working on. Good luck Simon!
- cmcollier 2y agoUnrelated to the core topic, I really enjoy the aesthetic of their website. Another similar one is from Fixie.ai (also, interestingly, one of their customers).
- itunpredictable 2y agoThis website rocks
- nsguy 2y agoYeah! fast, clean, cool, unique.
- swyx 2y agowhat does fixie do these days?
- xarope 2y agoYes, I like the turboxyz123 animation and contrast to the minimalist website (reminds me of the zen garden with a single rock). I think people forget nowadays in their haste to add the latest and greatest react animation, that too much noise is a thing.
- k2so 2y agoThis was my first thought too, after reading through their blog. This feels like a no-frills website made by an engineer, who makes things that just work. The documentation is great, I really appreciate them putting the roadmap front and centre.
- deleted 2y ago[deleted]
- 5- 2y agoindeed! what a nice, minimal page... that comes with ~1.6mb of javascript.
- bigbones 2y agoSounds like a source-unavailable version of Quickwit? https://quickwit.io/ https://quickwit.io/
- fulmicoton 2y agoQuickwit is targetting logs: - it does not do vector search. It can rank docs using BM25, but usually people just want to sort by timestamp. - its does not use an SSD cache. Quickwit reads directly into the object storage. - it is append-only (you can't modify documents) - it scales really well and typically shines on the 1TB .. 100PB range - it has a Elastic search compatible API.
- pushrax 2y agoLSM tree storage engine vs time series storage engine, similar philosophy but different use cases
- singhrac 2y agoMaybe I misunderstood both products but I think neither Quickwit or Turbopuffer is either of those things intrinsically (though log structured messages are a good fit for Quickfit). I think Quickwit is essentially Lucene/Elasticsearch (i.e. sparse queries or BM25) and Turbopuffer does vector search (or dense queries) like say Faiss/Pinecone/Qdrant/Vectorize, both over object storage.
- pushrax 2y agoIt's true that turbopuffer does vector search, though it also does BM25. The biggest difference at a low level is that turbopuffer records have unique primary keys, and can be updated, like in a normal database. Old records that were overwritten won't be returned in searches. The LSM tree storage engine is used to achieve this. The LSM tree also enables maintenance of global indexes that can be used for efficient retrieval without any time-based filter. Quickwit records are immutable. You can't overwrite a record (well, you can, but overwritten records will also be returned in searches). The data files it produces are organized into a time series, and if you don't pass a time-based filter it has to look at every file.
- vidar 2y agoCan you compare to S3 Athena (ELI5)?
- CyberDildonics 2y agoSounds like a filesystem with attributes in a database.
- eknkc 2y agoIs there a good general purpose solution where I can store a large read only database in s3 or something and do lookups directly on it? Duckdb can open parquet files over http and query them but I found it to trigger a lot of small requests reading bunch of places from the files. I mean a lot. I mostly need key / value lookups and could potentially store each key in a seperate object in s3 but for a couple hundred million objects.. It would be a lot more managable to have a single file and maybe a cacheable index.
- jiggawatts 2y ago> trigger a lot of small requests reading bunch of places from the files. I mean a lot. That’s… the whole point. That’s how Parquet files are supposed to be used. They’re an improvement over CSV or JSON because clients can read small subsets of them efficiently! For comparison, I’ve tried a few other client products that don’t use Parquet files properly and just read the whole file every time, no matter how trivial the query is.
- eknkc 2y agoThis makes sense but the problem I had with duckdb + parquet is it looks like there is no metadata caching so each and every query triggers a lot of requests. Duckdb can query a remote duckdb database too, in that case it looks like there is caching. Which might be better. I wonder if anyone actually worked on a specific file format for this use case (relatively high latency random access) to minimize reads to as little blocks as possible.
- jiggawatts 2y agoSounds like a bug or missing feature in DuckDB more than an issue with the format
- imiric 2y agoClickHouse can also read from S3. I'm not sure how it compares to DuckDB re efficiency, but it worked fine for my simple use case.
- mjlxyz 2y ago[flagged]
- bean_salad_123 2y ago[flagged]
- omneity 2y ago> In 2022, production-grade vector databases were relying on in-memory storage This is irking me. pg_vector has existed from before that, doesn't require in-memory storage and can definitely handle vector search for 100m+ documents in a decently performant manner. Did they have a particular requirement somewhere?
- jbellis 2y agoHave you tried it? pgvector performance falls off a cliff once you can't cache in ram. Vector search isn't like "normal" workloads that follow a nice pareto distribution.
- omneity 2y agoTried and deployed in production with similar sized collections. You only need enough memory to load the index, definitely not the whole collection. A typical index would most likely fit within a few GBs. And even if you need dozens of GBs of RAM it won’t cost nearly as much as $20k/month as the article surmises.
- lyu07282 2y agoHow do you get to "a few GBs"? A hundred million embeddings, if you have 4 byte floats 1024 dimensions would be >400 GB alone.
- omneity 2y agoI did say the index, not the embeddings themselves. The index is a more compact representation of your embeddings collection, and that's what you need in memory. One approach for indexing is to calculate centroids of your embeddings. You have multiple parameters to tweak, that affect retrieval performance as well as the memory footprint of your indexes. Here's a rundown on that: https://tembo.io/blog/vector-indexes-in-pgvector https://tembo.io/blog/vector-indexes-in-pgvector
- yamumsahoe 2y agounsure if they are comparable, but is this and quickwit comparable?
- hipadev23 2y agoThat’s some woefully disappointing and incorrect metrics (read and write latency are both sub-second, storage medium would be “ Memory + Replicated SSDs”) you’ve got for Clickhouse there, but I understand what you’re going for and why you categorized it where you did.
- endisneigh 2y agoSlightly relevant - do people really want article recommendations? I don’t think I’ve ever read an article and wanted a recommendation. Even with this one - I sort of read it and that’s it; no feeling of wanting recommendations. Am I alone in this? In any case this seems like a pretty interesting approach. Reminds me of Warpstream which does something similar with S3 to replace Kafka.
- nh2 2y ago> $3600.00/TB/month It doesn't have to be that way. At Hetzner I pay $200/TB/month for RAM. That's 18x cheaper. Sometimes you can reach the goal faster with less complexity by removing the part with the 20x markup.
- TechDebtDevin 2y agoI will likely never leave Hetzner.
- AYBABTME 2y ago200$/TB/month for raw RAM, not RAM that's presented to you behind a usable API that's distributed and operated by someone else, freeing you of time. It's not particularly useful to compare the cost of raw unorganized information medium on a single node, to highly organized information platform. It's like saying "this CPU chip is expensive, just look at the price of this sand".
- kirmerzlikin 2y agoAFAIU, 3600$ is also a price for "raw RAM" that will be used by your common database via sys calls and not via a "usable API operated by someone else"
- hodgesrm 2y ago> It's not particularly useful to compare the cost of raw unorganized information medium on a single node, to highly organized information platform. Except that it does prompt you to ask what you could do to use that cheap compute and RAM. In the case of Hetzner that might be large caches that allow you to apply those resources on remote data whilst minimizing transfer and API costs.
- formerly_proven 2y agoYou seem to be quoting the highest figure from the article out of context as-if that is their pricing, but the opposite is the case. > $3600.00/TB/month (incumbents) > $70.00/TB/month (turbopuffer) That's still 3x cheaper than your number and it's a SaaS API, not just a piece of rented hardware.
- yawnxyz 2y agocan't wait for the day the get into GA!
- cdchn 2y agoThe very long introductory page has a ton of very juicy data in it, even if you don't care about the product itself.
- zX41ZdbW 2y agoA correction to the article. It mentions Warehouse BigQuery, Snowflake, Clickhouse ≥1s Minutes For ClickHouse, it should be: read latency <= 100ms, write latency <= 1s. Logging, real-time analytics, and RAG are also suitable for ClickHouse.
- Sirupsen 2y agoYeah, thinking about this more I now understand Clickhouse to be more of an operational warehouse similar to Materialize, Pinot, Druid, etc. if I understand correctly? So bunching with BigQuery/Snowflake/Trino/Databricks... wasn't the right category (although operational warehouses certainly can have a ton of overlap) I left that category out for simplicity (plenty of others that didn't make it into the taxonomy, e.g. queues, nosql, time-series, graph, embedded, ..)
- arnorhs 2y agoThis looks super interesting. I'm not that familiar with vector databases. I thought they were mostly something used for RAG and other AI-related stuff. Seems like a topic I need to delive into a bit more.
- solatic 2y agoIs it feasible to try to build this kind of approach (hot SSD cache nodes sitting in front of object storage) with prior open-source art (Lucene)? Or are the search indexes themselves also proprietary in this solution? Having witnessed some very large Elasticsearch production deployments, being able to throw everything into S3 would be incredible. The applicability here isn't only for vector search.
- francoismassot 2y agoIf you don't need vector search and have very large Elasticsearch deployment, you can have a look at Quickwit, it's a search engine on object storage, it's OSS and works for append-only datasets (like logs, traces, ...) Repo: https://github.com/quickwit-oss/quickwit https://github.com/quickwit-oss/quickwit
- rohitnair 2y agoElasticsearch and OpenSearch already support S3 backed indices. See features like https://opensearch.org/docs/latest/tuning-your-cluster/availability-and-recovery/snapshots/searchable_snapshot/ https://opensearch.org/docs/latest/tuning-your-cluster/avail... The files in S3 are plain old Lucene segment files (just wrapped in OpenSearch snapshots which provide a way to track metadata around those files).
- francoismassot 2y agoBut you don’t have fast search on those files stored on object storage.
- rohitnair 2y agoYes, there is a cold start penalty but once the data is cached, it is equivalent to disk backed indices. There is also active work being done to improve the performance, example https://github.com/opensearch-project/OpenSearch/issues/13806 https://github.com/opensearch-project/OpenSearch/issues/1380...