6 ms·
In my experience the semantic/lexical search problem is better understood as a precision/recall tradeoff. Lexical search (along with boolean operators, exact ph
by whakim 10mo ago
In my experience the semantic/lexical search problem is better understood as a precision/recall tradeoff. Lexical search (along with boolean operators, exact phrase matching, etc.) has very high precision at the expense of lower recall, whereas semantic search sits at a higher recall/lower precision point on the curve.
- simonw 10mo agoYeah, that sounds about right to me. The most effective approach does appear to be a hybrid of embeddings and BM25, which is worth exploring if you have the capacity to do so. For most cases though sticking with BM25 is likely to be "good enough" and a whole lot cheaper to build and run.
- mips_avatar 10mo agoDepends on the app and how often you need to change your embeddings, but I run my own hybrid semantic/bm25 search on my MacBook Pro across millions of documents without too much trouble.
- echion 9mo agoCan you elaborate a bit on your setup if you have time?