7 ms·
Vector indexing all of Wikipedia on a laptop
- gfourfour 2y agoMaybe I’m missing something but I’ve created vector embeddings for all of English Wikipedia about a dozen times and it costs maybe $10 of compute on Colab, not $5000
- emmelaich 2y agoGot any details?
- gfourfour 2y agoNothing too crazy, just downloading a dump, splitting it into manageable batch sizes, and using a lightweight embedding model to vectorize each article. Using the best GPU available on colab it takes maybe 8 hours if I remember correctly? Vectors can be saved as NPY files and loaded into something like FAISS for fast querying.
- j0hnyl 2y agoHow big is the resulting vector data?
- gfourfour 2y agoLike 8 gb roughly
- abetusk 2y agoThis probably deserves its own article and might be of interest to the HN community.
- gfourfour 2y agoI will probably make a post when I launch my app! For now I’m trying to figure out how I can host the whole system for cheap because I don’t anticipate generating much revenue
- solarengineer 2y agoI'm interested in hearing about what you will be hosting. Would Digital Ocean or Hetzner meet your needs?
- gfourfour 2y agoI was going to use ec2 and s3, should I look at digital ocean or hetzner instead?
- riku_iki 2y agoWhat is the end task(e.g. RAG, or just vector search for question answering), are you satisfied with results in terms of quality?
- gfourfour 2y agoThe end result is a recommendation algorithm, so basically just vector similarity search (with a bunch of other logic too ofc). The quality is great, and if anything a little bit of underfitting is desirable to avoid the “we see you bought a toilet seat, here’s 50 other toilet seats you might like” effect.
- thomasfromcdnjs 2y agoDid you chunk the articles? If so, in what way?
- gfourfour 2y agoYes. I split the text into sentence and append sentences to a chunk until the max context window is reached. The context window size is dynamic for each article so that each chunk is roughly the same size. Then I just do a mean pool of the chunks for each article.
- thomasfromcdnjs 2y agoThanks for the answer. Wikipedia has a lot of tables so I was wondering if content-aware sentence chunking would be good enough for Wikipedia. https://www.pinecone.io/learn/chunking-strategies/ https://www.pinecone.io/learn/chunking-strategies/
- gfourfour 2y agomwparserfromhell can parse the text content without including tables
- hivacruz 2y agoDid you do use the same method, i.e. split by chunks each article and vectorize each chunk?
- gfourfour 2y agoYes
- dudus 2y agoThat's the only way to do it. You can't index the whole thing. The challenge is chunking. There are several different algorithms to chunk content for vectorization with different pros and cons.
- minimaxir 2y agoYou can do much bigger chunks with models that support RoPE embeddings, such as nomic-embed-text-1.5 which has a 8192 context length: https://huggingface.co/nomic-ai/nomic-embed-text-v1.5 https://huggingface.co/nomic-ai/nomic-embed-text-v1.5 In theory this would be an efficiency boost but the performance math can be tricky.
- qudat 2y agoAs far as I understand it, context length degrades llm performance, so just because an llm "supports" a large context length it basically just clips a top and bottom chunk and skips over the middle bits.
- rahimnathwani 2y agoWhy would you want chunks that big for vector search? Wouldn't there be too much information in each chunk, making it harder to match a query to a concept within the chunk?
- nostrebored 2y agoThe problem is that often semantic meaning depends on state multiple paragraphs or sections away. This is a coarse way to tackle that
- DataDaemon 2y agoHow?
- bunderbunder 2y agoThis is covering 300+ languages, not just English, and it's specifically using Cohere's Embed v3 embeddings, which are provided as a service and currently priced at US$0.10 per million tokens [1]. I assume if you're running on Colab you're using an open model, and possibly a relatively lighter weight one as well? [1]: https://cohere.com/pricing https://cohere.com/pricing
- traverseda 2y agoThis is pretty early in the game to be relying on proprietary embeddings, don't you think? If if they are 20% better, blink and there will be a new normal. It's insane to me that someone, this early in the gold rush, would be mining in someone else's mine, so to speak
- bunderbunder 2y agoI have no idea. But that wasn't the question I was answering. It was, "how does the article's author estimate that would cost $5000?" And I think that's how. Or at least, that gets to a number that's in the same ballpark as what the author was suggesting. That said, first guess, if you do want to evaluate Cohere embeddings for a commercial application, using this dataset could be a decent basis for a lower-cost spike.
- jbellis 2y agoYes, that is how I came up with that number.
- janalsncm 2y agoIt’s not just that. Embeddings aren’t magic. If you’re going to be creating embeddings for similarity search, the first thing you need to ask yourself is what makes two vectors similar such that two embeddings should even be close together? There are a lot of related sources of similarity, but they’re slightly different. And I have no idea what Cohere is doing. Additionally, it’s not clear to me how queries can and should be embedded. Queries are typically much shorter than their associated documents, so they typically need to be trained jointly. Selling “embeddings as a service” is a bit like selling hashing as a service. There are a lot of different hash functions. Cryptographic hashes, locality sensitive hashes, hashes for checksum, etc.
- janalsncm 2y agoAlso, if you’re spending $5000 to compute embeddings, why are you indexing them on a laptop?
- syllogistic 2y agoHe's not though because cohere stuck the already embedded dataset on huggingface https://huggingface.co/datasets/Cohere/wikipedia-22-12-en-embeddings https://huggingface.co/datasets/Cohere/wikipedia-22-12-en-em...
- bytearray 2y agoDo you have a link to the notebook?
- gfourfour 2y agoNo haha just a rats nest of a bunch of notebooks
- traverseda 2y ago>Disable swap before building the index. Linux will aggressively try to cache the index being constructed to the point of swapping out parts of the JVM heap, which is obviously counterproductive. In my test, building with swap enabled was almost twice as slow as with it off. This is an indication to me that something has gone very wrong in your code base.
- dekhn 2y agoAs a workaround, you can mlock your process which should prevent the application pages from being evicted by swap.
- jbellis 2y agoFWIW this is what Cassandra does on startup, but it didn't seem worth it to go to the trouble of dealing with Unsafe for a demo project.
- nextaccountic 2y agoCan't mlock be wrapped out in a safe API?
- jbellis 2y agoProbably, but I'm not aware of any Java library that provides that functionality.
- papercrane 2y agoIf you don't mind using a preview feature you should be able to use the foreign function API to call mlockall without any unsafe code. Otherwise, JNA is probably the easiest way, and how Cassandra does it. https://docs.oracle.com/en/java/javase/21/core/calling-c-library-function-foreign-function-and-memory-api.html https://docs.oracle.com/en/java/javase/21/core/calling-c-lib... https://github.com/java-native-access/jna https://github.com/java-native-access/jna
- tjake 2y agoYou can demo this here: https://jvectordemo.com:8443/ https://jvectordemo.com:8443/ GH Project: https://github.com/jbellis/jvector https://github.com/jbellis/jvector
- deleted 2y ago[deleted]
- hot_gril 2y agoHow many dimensions are in the original vectors? Something in the millions?
- jbellis 2y ago1024 per vector x 41M vectors
- hot_gril 2y ago1024-dim vectors would fit into pgvector in Postgres, which can do cosine similarity indexing and doesn't require everything to fit into memory. Wonder how the performance of that would compare to this.
- jbellis 2y agoIt's been a while since I read the source to pgvector but at the time it was a straightforward HNSW implementation that implicitly assumes your index fits in page cache. Once that's not true, your search performance will fall off a cliff. Which in turn means your insert performance also hits a wall since each new vector requires a search. I haven't seen any news that indicates this has changed, but by all means give it a try!
- hot_gril 2y agoThanks, I didn't know that. Last time I was dealing with these kinds of problems, pgvector didn't exist yet.
- isoprophlex 2y ago$5000?! I indexed all of HN for ... $50 I think. And that's tens of millions of posts.
- xandrius 2y agoTo be fair Wikipedia has over 60 million pages and this is for 300+ languages. But yeah, the value shows that they might not be using the cheapest service out there.
- criddell 2y agoHow are you using that index?
- isoprophlex 2y agohttps://www.searchhacker.news/ https://www.searchhacker.news/ A tool that (hopefully) surfaces interesting HN discussion threads; I wanted an excuse to investigate (hybrid) full text and vector search at a substantial scale beyond toy datasets. Sadly (well not really) I changed jobs soon after building the first version. Life caught up and I never got around to adding more features and polishing up the frontend (eg. the broken back button Ideas for new features are very welcome :)
- peter_l_downs 2y ago> JVector, the library that powers DataStax Astra vector search, now supports indexing larger-than-memory datasets by performing construction-related searches with compressed vectors. This means that the edge lists need to fit in memory, but the uncompressed vectors do not, which gives us enough headroom to index Wikipedia-en on a laptop. It's interesting to note that JVector accomplishes this differently than how DiskANN described doing it. My understanding (based on the links below, but I didn't read the full diff in #244) is that JVector will incrementally compress the vectors it is using to construct the index; whereas DiskANN described partitioning the vectors into subsets small enough that indexes can be built in-memory using uncompressed vectors, building those indexes independently, and then merging the results into one larger index. OP, have you done any quality comparisons between an index built with JVector using the PQ approach (small RAM machine) vs. an index built with JVector using the raw vectors during construction (big RAM machine)? I'd be curious to understand what this technique's impact is on the final search results. I'd also be interested to know if any other vector stores support building indexes in limited memory using the partition-then-merge approach described by DiskANN. Finally, it's been a while since I looked at this stuff, so if I mis-wrote or mis-understood please correct me! - DiskANN: https://dl.acm.org/doi/10.5555/3454287.3455520 https://dl.acm.org/doi/10.5555/3454287.3455520 - Anisotropic Vector Quantization (PQ Compression): https://arxiv.org/abs/1908.10396 https://arxiv.org/abs/1908.10396 - JVector/#168: How to support building larger-than-memory indexes https://github.com/jbellis/jvector/issues/168 https://github.com/jbellis/jvector/issues/168 - JVector/#244: Build indexes using compressed vectors https://github.com/jbellis/jvector/pull/244 https://github.com/jbellis/jvector/pull/244
- jbellis 2y agoThat's correct! I've tested the build-with-compression approach used here with all the datasets in JVector's Bench [1] and there's near zero loss in accuracy. I suspect that the reason the DiskANN authors used the approach they did is that in 2019 Deep1B was about the only very large public dataset around, and since the vectors themselves are small your edge lists end up dominating your memory usage. So they came up with a clever solution, at the cost of making construction 2.5x as expensive. (Educated guess: 2x is from adding each vector to multiple partitions and the extra 50% to merge the results.) So JVector is just keeping edge lists in memory today. When that becomes a bottleneck we may need to do something similar to DiskANN but I'm hoping we can do better because it's frankly a little inelegant. [1] https://github.com/jbellis/jvector/blob/main/jvector-examples/src/main/java/io/github/jbellis/jvector/example/Bench.java https://github.com/jbellis/jvector/blob/main/jvector-example...
- localhost 2y agoThis is a giant dataset of 536GB of embeddings. I wonder how much compression is possible by training or fine-tuning a transformer model directly using these embeddings, i.e., no tokenization/decoding steps? Could a 7B or 14B model "memorize" Wikipedia?
- arnaudsm 2y agoIn expert topics, is vector search finally competitive with BM25-like algorithms? Or do we still need to mix the 2 together ?
- jbellis 2y agoColBERT gives you best of both worlds. https://arxiv.org/abs/2004.12832 https://arxiv.org/abs/2004.12832 https://thenewstack.io/overcoming-the-limits-of-rag-with-colbert/ https://thenewstack.io/overcoming-the-limits-of-rag-with-col...
- danaugrs 2y agoLoosely related: https://www.quantamagazine.org/computer-scientists-invent-an-efficient-new-way-to-count-20240516/ https://www.quantamagazine.org/computer-scientists-invent-an...
- jl6 2y agoThe source files appear to include pages from all namespaces, which is good, because a lot of the value of Wikipedia articles is held in the talk page discussions, and these sometimes get stripped from projects that use Wikipedia dumps.
- worldsayshi 2y agoI'm curious what the main value you see in the talk pages? I almost never look at them myself.
- jl6 2y agoThey’re not so interesting for mundane topics, but for anything remotely controversial, they are essential for understanding what perspectives aren’t included in the article.
- lilatree 2y ago“… turning an O(log N) search per segment into O(N) overall.” Can someone explain why?
- StrangeDoctor 2y agoWhen it’s all in memory you get to amortize the cost of the initial load. Or just pay it when it’s not part of the hot path. When it’s segmented, you’re doing that because memory is full and you need to read in all the segments you don’t have. That’ll completely overwhelm the log n of the search you still get
- jbellis 2y agoI was trying to make the point that the dominant factor becomes linear instead of logarithmic, but more accurately it's O(S log N) = O(N log N) because S (number of segments) is proportional to N (number of vectors).
- StrangeDoctor 2y agoAh yeah that’s what I wanted to write but I guess I didn’t want to put words in your mouth, and stuck to what I could be certain about happening. We do all this work to throw away the unneeded bits in one situation and when comparing it to a slightly different situation go “huh some of that garbage would be kinda nice here”
- HammadB 2y ago"The obstacle is that until now, off-the-shelf vector databases could not index a dataset larger than memory, because both the full-resolution vectors and the index (edge list) needed to be kept in memory during index construction. Larger datasets could be split into segments, but this means that at query time they need to search each segment separately, then combine the results, turning an O(log N) search per segment into O(N) overall." How is a log N search over S segments O(N)?
- thfuran 2y agoDoesn't doubling N double S?
- jbellis 2y agoI was trying to make the point that the dominant factor becomes linear instead of logarithmic, but more accurately it's O(S log N) = O(N log N) because S is proportional to N.
- deleted 2y ago[deleted]
- HammadB 2y agoSure, I see. I think this is an area where complexity analysis doesn’t lead to useful information. To be more correct it’s O(N/C log C) where C is the capacity of a segment. In this case you can ignore 1/C and log C as constant. So now sure, you actually just have O(N). But this is not super useful as it says that a segmented hnsw approach and brute force approach are the same - when this is really not the case in practice. Also O(N log N) > O(N) so I’m not sure why we would ever do anything with segmentation according to that analysis if it were correct.
- jbellis 2y ago> I’m not sure why we would ever do anything with segmentation according to that analysis if it were correct. What's your alternative when you can't build an index larger than C?
- m3kw9 2y agoHe should have asked HN on the cheapest way to embed Wikipedia before starting
- jbellis 2y agoI'm baffled that so many people fixate on the estimated cost and miss the fact that it's a public dataset. As in, free.
- syllogistic 2y agoIt's in the first sentence of the article too :)
- m3kw9 2y agoGetting the embeddings ain’t free
- burgerrito 2y agoI made a side project that uses Wikipedia recently too, and found out that there are database dump available to be downloaded: https://en.wikipedia.org/wiki/Wikipedia:Database_download https://en.wikipedia.org/wiki/Wikipedia:Database_download
- Atotalnoob 2y agoWhy is the author listing himself as datastax cto? He isn’t according the Wikipedia, my friend who works there, and their company website. https://www.datastax.com/our-people https://www.datastax.com/our-people That’s kind of weird
- deleted 2y ago[deleted]
- metadat 2y agoWhat are you talking about? The datastax site lists it: > SANTA CLARA, Calif. – September 28, 2020 – DataStax today announced that DataStax Co-Founder and CTO Jonathan Ellis will deliver a keynote address at ApacheCon @Home 2020 https://www.datastax.com/press-release/datastax-co-founder-and-cto-jonathan-ellis-keynote-apachecon-2020-open-source-cloud https://www.datastax.com/press-release/datastax-co-founder-a.... As an aside, I'm an ApacheCon presenter but there was no press release about the hot excitement of my involvement. Maybe next time :)
- Atotalnoob 2y agoThat’s from 2024. They aren’t the cto of datastax currently
- metadat 2y agoMaybe it's the highest rank they achieved and still trying to capitalize on it. So what? Should they now say "Unemployed"? There has to be more interesting things to discuss than this. P.s. I think you meant 2020.
- lemarchr 2y agoSee https://www.datastax.com/our-people/jonathan-ellis https://www.datastax.com/our-people/jonathan-ellis Wikipedia lists them as a founder. Perhaps their author bio is outdated, or Wikipedia is. Not sure about your friend.
- 2y ago
- Khelavaster 2y agoThis is how Microsoft powered it's academic paper search in 2016, before rolling it into Bing in 2020!
- opdahl 2y agoWould be interesting if you could try implementing the Cohere Reranker into this. Should be fairly easy, and could lead to quite a bit of performance gain.
- issafram 2y agoWould a docker container help running it on Windows?
- noufalibrahim 2y agoWhat are the good solutions in this space? Vector databases I mean. Mostly for semantic search across various texts. I have a few projects I'd like to work on. For typical web projects, I have a "go to" stack and I'd like to add something sensible for vector based search to that.
- StrauXX 2y agoIn my experience its usually easiest to use a vector store extension for an off-the-shelf database like postgres (pgvector is nice). That way you don't have to manage another, rapidly changing, service and you can easily combine queries on the vectors with regular columns, join them and so on.
- jbellis 2y agoJVector (the index used in TFA) is available as a service with a friendly API from DataStax. https://www.datastax.com/products/datastax-astra https://www.datastax.com/products/datastax-astra [article author, I work on JVector and Astra]
- noufalibrahim 2y agoNice. I wanted to try something out on a machine before moving to hosted soclutions.
- riku_iki 2y agoCould you tell how scalable JVector is? How many vectors it can handle, like millions, billions, hundreds of billions?
- anonymousDan 2y agoHow do embeddings created by state of the art open source models compare to the free embeddings mentioned in the article? Would they actually cost 5k to create given a reasonable local GPU setup?
- Mathnerd314 2y ago> Enough RAM to run a JVM with 36GB of heap space Are there laptops like that? Maybe an upgraded MacBook, but I have been looking for Windows/Linux laptops and they generally top out at 32GB. I checked Lenovo's website and everything with 64GB and up is not called a laptop but a "mobile workstation".