6 ms·
Well the most standard approach is to use cross-encoders (e.g. something like Cohere Rerank) to give similarity scores between the query and the chunk, and then
by Weves 3y ago
Well the most standard approach is to use cross-encoders (e.g. something like Cohere Rerank) to give similarity scores between the query and the chunk, and then use these scores to update the ranking.
Our approach is to use an LLM (gpt-3.5-turbo for example), and to ask it explicitly "Is this chunk <CHUNK> useful for answering this query <QUERY>". We've found, while certainly a bit more expensive, the larger model size and greater understanding of the world allows this approach to yield significantly better results that the SOTA cross-encoders. It also allows us to ask the model to explain why it's useful, which can be really helpful for the user when determining if they should look deeper into a document (as opposed to the standard keyword-based highlighting which often isn't very useful when determining if a document actually has useful information for your query).
- sroussey 3y agoInteresting! Thanks for the explanation.
- yuhongsun 3y agoTwo other tidbits on this: 1. There's a difference between relevance and usefulness that cross-encoders cannot capture. Imagine a thread with a bunch of people complaining about an exception and each comment is another mention of the exception. Now imagine another thread with one mention of the exception at the top, and a bunch of people offering solutions. If you query for the exception, LLMs will find the second thread more useful, but cross encoders will find the first one more relevant. 2. LLMs/GenAI models don't output a single value. They can use the tokens they output to "reason" about the usefulness of a doc. Eg. Rerankers are like tiny LLMs that are only allowed to output "yes" or "no", but instead you can use an LLM to do chain-of-thought and finally decide at the end.