7 ms·
Systematically Improving Your RAG
- DerCommodore 2y ago[flagged]
- satisfice 2y agoNot a lot of content, here.
- socksy 2y agoI’m always suspicious with low-signal articles written about LLM based systems, as I suspect that the crowd involved are very trigger happy in using an LLM to write human facing text. Maybe not what’s happening in this case, but it’s what springs to mind.
- minimaxir 2y agoThe author is legitimately knowledgeable about LLM/RAG systems and develops open-source tooling around it. But yes, this isn't a good HN submission without detail.
- jxnlco 2y agohow can i make it better? this was a quick post that i wrote up after a 30 minute call with someone, mostly notes to take in preparation for a bigger talk im giving.
- minimaxir 2y agoBullet points in general aren't good for HN discussion even if it's on an interesting/complex topic. Examples are good. A talk + written summary/transcript of the talk when it's made would be much better.
- jxnlco 2y agoI'll try to add more details!
- yumraj 2y ago> this was a quick post that i wrote up after a 30 minute call with someone Then you shouldn't have posted it on HN until there was more meat. Notes from a 30min call are OK for personal consumption, but not mass sharing as it is not useful in general and devalues personal brand. My 2 cents..
- timack 2y agoThis comment seems a bit harsh. My 2 cents.
- groby_b 2y agofwiw, I found this high value. I'm currently in the space of "conceptually, what do I need to think about in this space", and this was a great set of notes for me. Awesome if the author wants to flesh it out further, but sometimes raw knowledge is more than enough.
- jxnlco 2y agocheck out the other rag stuff https://jxnl.co/writing/category/rag/ https://jxnl.co/writing/category/rag/ all based on consulting calls and advisory work (i sell the implementation so sorry i don't post one click deploys)
- groby_b 2y agoOooh, nice - thank you for sharing those! (At least from my POV, that's already a boatload of value you're giving away for free!)
- hdlothia 2y agoSo the part of RAG that's tripping me up right now is vector search and familiarity scores. Does anyone have a good resource to learn more about this? I've been using this as a starter. https://developers.cloudflare.com/workers-ai/tutorials/build-a-retrieval-augmented-generation-ai/ https://developers.cloudflare.com/workers-ai/tutorials/build... I put in text but I feel like my conception of what should get high relevancy scorrs doesn't match the percentages that come out. The article talks about full text search and meta data so maybe that's the path I should be taking instead of vector search? Where would I store the Metadata in this case? A regular db? I wish articles like this would go into more details about the nitty gritty. But I appreciate high level overview in the article as well.
- jxnlco 2y agothanks! this was based on a 30 minute 'crash course on what things they need to look out fo' I'd recommend taking a look at lancedb as they support text, vectors, and sql. high relevancy scores are not percentages, they only make sense in 'ordering' but 0.7 does not mean relevant. but .9 vs .7 means 'maybe more relevant.
- PheonixPharts 2y agoOnce you have vector representations the "similarity" scores are just basic linear algebra. It's fundamentally no different than any other IR/recsys task. A good overview is chapter 6 of the Stanford NLP group's IR book [0]. Engineering LLMs still requires a good foundation in the basics of ML/NLP so it's worth the time to catch up a bit. 0. https://web.archive.org/web/20231207074155/https://nlp.stanford.edu/IR-book/html/htmledition/scoring-term-weighting-and-the-vector-space-model-1.html https://web.archive.org/web/20231207074155/https://nlp.stanf...
- hdlothia 2y agoThis is exactly what I was looking for. Thank you so much!
- danenania 2y agoThis all seems pretty sensible. Another area that would be nice to see addressed are strategies for balancing latency/cost/performance when data is frequently updated. I'm building a terminal-based AI coding tool[1] and have been thinking about how to bring RAG into the picture, as it clearly could add value, but the tradeoffs are tricky to get right. The options, as far as I can tell, are: - Re-embed lazily as needed at prompt-time. This should be the cheapest as it minimizes the number of embedding calls, but it's the most expensive in terms of latency. - Re-embed eagerly after updates (perhaps with some delay and throttling to avoid rapid-fire embed calls). Great for latency, but can get very expensive. - Some combination of the above two options. This seems to be what many IDE-based AI tools like GH Copilot are doing. An issue with this approach is that it's hard to ever know for sure what's updated in the RAG index and what's stale, and what exactly is getting added to context at any given time. I'm leaning toward the first option (lazy on-demand embedding) and letting the user decide whether the latency cost is worth it for their task vs. just manually selecting the exact context they want to load. 1 - https://github.com/plandex-ai/plandex https://github.com/plandex-ai/plandex
- 3abiton 2y agoAny benchmarks on performance for on-demand embeddings?
- Xenoamorphous 2y agoWhy is there so much buzz about RAG? Isn’t it basically a traditional search (either keyword based, vector based -embeddings have been around for years-, or a combination of both) where you take the top N results (usually not even full docs, but chunks due to context length limitations) and pass them to an LLM to regurgitate a response (hopefully without hallucinations), instead of simply listing the results right away? I think some implementations also ask the LLM to rewrite the user query to “capture the user intent”. What I’m missing here? What makes it so useful?
- dragonwriter 2y agoThe intent is usually not to simply regurgitate the results, but to augment the prompt with them to enable a better, focussed answer to the user question than either search or an LLM alone would provide.
- jxnlco 2y agonothing, all i really say is 'add monitoring, do topic clustering' which is how i did 'search' and 'recommendation' systems 1) are there filters we need to build 2) do we have inventory
- simonw 2y agoThat's exactly what it is, and it's useful because when it works it means you can ask a question and get an answer to your question, rather then having to read the documents and then answer that question yourself.
- ru552 2y ago*What makes it so useful? One example is in finance, you have a lot of 45 page PDFs laying around and you're pretty sure one of them has the Reg, or info you need. You aren't sure which so you open them one by one and do a search for a word, then jump through a bunch of those results and decide it's not this PDF. You do that till you find the "one". There are a non trivial amount of Executive level jobs that pretty much do this for half of their work week. RAG purports to let you search one time.
- 2y ago
- psynister 2y agoMost of this can be done automatically using https://vectorize.io https://vectorize.io It generates synthetic questions, tests different embedding models, chunking strategies, etc. You end up with clear data that shows you what will give you the optimal results for your RAG app: https://platform.vectorize.io/public/experiments/ca60ce85-26a4-4646-b68f-612748338bb3 https://platform.vectorize.io/public/experiments/ca60ce85-26...
- 7thpower 2y agoThis is a great intro. I am amazed how many people don’t use the LLMs to analyze the questions themselves and apply filters to avoid pulling back irrelevant documents in the first place. We run as many methods as practical in parallel (sql, vector, full text, other methods, etc.) and return the first one that meets our threshold. Vector search is almost never the winner relative to full text. Instead, I see a lot of people in sister companies using the most robust models they can find and having agents to do chain of thought, while their users are wondering when, if ever, they’ll get a response back.
- cpursley 2y agoThis sounds really interesting. Do you have any longer-form writeup on this approach (or could you point us towards related info)?
- 7thpower 2y agoI do not but my twitter handle is in my profile and I am always more than happy to hop on a call and share what I know. For reference our subject matter is engineering specs for high precision electronics manufacturing. We have ~100k products and a lot of them have identical documentation except for a few figures (which make all the difference in the world), so it’s a challenging use case that is very unforgiving. Totally doable though and the basis for a lot of capabilities we’ll be investing in moving forward. Happy to share as I think we’re ahead in a few areas but believe others will catch up and we’ve learned so much from others willing to share info, so we always try to pay forward.
- schmidt_fifty 2y ago> Vector search is almost never the winner relative to full text. Full text search is certainly the winner in the time dimension, but can it compete in quality? Presumably which method is likely to provide relevant results depends greatly on the query. Invoking LLMs to pre-process the query and select a retrieval method is going to be quite expensive compared to each of the search methods.
- 2y ago
- ofermend 2y agoBuilding RAG can be easy for a simple example, but it's much more nuanced than you might think when you try to do it at larger scale. With larger-scale real-world enterprise RAG-based applications, you soon realize the enormous time and effort required to experiment with all these levers to optimize the RAG pipeline: which vector DB to use and how, which embedding model to use, pure vector search or hybrid search, chunking strategies, and on and one... With Vectara's RAG-as-a-service (www.vectara.com) we try to help address exactly this issue: you get an optimized, high performance, secure and scalable RAG pipeline, so you don't need to go through this massive hyper-parameter tuning exercise. Yes, there are still some very useful levers you can experiment with, but only where it really matters.
- demilich 2y agoTry RAPTOR: https://arxiv.org/html/2401.18059v1 https://arxiv.org/html/2401.18059v1 An implementation: github.com/infiniflow/ragflow