5 ms·

I'd be curious to hear what people think is state of the art of this kind of problem? I think the Cohere embeddings model v3 is very good, as it handles querie

I'd be curious to hear what people think is state of the art of this kind of problem? I think the Cohere embeddings model v3 is very good, as it handles queries and documents differently to embed them in the same vector space. Otherwise for a specific use case I guess dense retrieval (which is basically a problem specific fine tuned version of this approach) is the best way to go about it?

dmezzetti 3y ago
If you're talking about embeddings model metrics, the MTEB Leaderboard is a good resource: https://huggingface.co/spaces/mteb/leaderboard https://huggingface.co/spaces/mteb/leaderboard
- Layvier 3y ago
  Yes I had a look at it, I haven't tried e5-mistral-7b-instruct yet but I'll definitely give it a go. Is there such a leaderboard only focused on retrieval by any chance? I haven't found one so far
  - dmezzetti 3y ago
    The BEIR project might be what you're looking for: https://github.com/beir-cellar/beir/wiki/Leaderboard https://github.com/beir-cellar/beir/wiki/Leaderboard
  - viraptor 3y ago
    It's interesting how the massive model of e5-mistral has only marginal performance gains over the bge-base and similar ones. It could still be useful for the longer sentence length though.
    - mahjongmen1998 3y ago
      e5-mistral is essentially a distillation from gpt-4 to a smaller model. You can see here https://github.com/microsoft/unilm/blob/16da2f193b9c1dab0a692c6e4380bd43e70a40cd/e5/utils.py#L93 https://github.com/microsoft/unilm/blob/16da2f193b9c1dab0a69... they actually have custom prompts for each dataset being tested. Question would be, if you haven't seen the task before, what is a good prompt to prepend for your task? IMO e5-mistral is overfit to MTEB
  - pzar 3y ago
    https://huggingface.co/spaces/mteb/leaderboard https://huggingface.co/spaces/mteb/leaderboard see the retrieval tab
charcircuit 3y ago
>I'd be curious to hear what people think is state of the art of this kind of problem? I would assume Amazon's product suggestion would be SotA for reccomending a book based off of another. It is a recommondation system and while it uses semantic search there are many more ranking signals it uses.
- lukev 3y ago
  The problem with systems like that is they're going to be optimized for making Amazon the most money, not necessarily what's best for the user (unless the two happen to coincide.)
  - charcircuit 3y ago
    Money is a proxy for user value and it is in Amazon's best interest for the customer to continually act upon these recommendations. If Amazon fails to deliver user value customers will be less willing to continue paying.
    - lukev 3y ago
      lol. lmao.
    - boolemancer 3y ago
      > Money is a proxy for user value and it is in Amazon's best interest for the customer to continually act upon these recommendations. If Amazon fails to deliver user value customers will be less willing to continue paying. You're talking about the same Amazon that will blindly give me recommendations for other products that fill exactly the same niche as the one I just bought. "Oh, you just bought a generator? You probably need a second one too, right?"
teaearlgraycold 3y ago
> it handles queries and documents differently to embed them in the same vector space I assume this might be behind the state of the art, then. As of a year ago OpenAI managed to get an embeddings model that can do both without any special flag or dual model to treat queries and documents differently.
- mahjongmen1998 3y ago
  handling queries and documents differently leads to an increase in overall retrieval.

dmezzetti 3y ago

If you're talking about embeddings model metrics, the MTEB Leaderboard is a good resource: https://huggingface.co/spaces/mteb/leaderboard https://huggingface.co/spaces/mteb/leaderboard

Layvier 3y ago
Yes I had a look at it, I haven't tried e5-mistral-7b-instruct yet but I'll definitely give it a go. Is there such a leaderboard only focused on retrieval by any chance? I haven't found one so far
- dmezzetti 3y ago
  The BEIR project might be what you're looking for: https://github.com/beir-cellar/beir/wiki/Leaderboard https://github.com/beir-cellar/beir/wiki/Leaderboard
- viraptor 3y ago
  It's interesting how the massive model of e5-mistral has only marginal performance gains over the bge-base and similar ones. It could still be useful for the longer sentence length though.
  - mahjongmen1998 3y ago
    e5-mistral is essentially a distillation from gpt-4 to a smaller model. You can see here https://github.com/microsoft/unilm/blob/16da2f193b9c1dab0a692c6e4380bd43e70a40cd/e5/utils.py#L93 https://github.com/microsoft/unilm/blob/16da2f193b9c1dab0a69... they actually have custom prompts for each dataset being tested. Question would be, if you haven't seen the task before, what is a good prompt to prepend for your task? IMO e5-mistral is overfit to MTEB
- pzar 3y ago
  https://huggingface.co/spaces/mteb/leaderboard https://huggingface.co/spaces/mteb/leaderboard see the retrieval tab

charcircuit 3y ago

>I'd be curious to hear what people think is state of the art of this kind of problem? I would assume Amazon's product suggestion would be SotA for reccomending a book based off of another. It is a recommondation system and while it uses semantic search there are many more ranking signals it uses.

lukev 3y ago
The problem with systems like that is they're going to be optimized for making Amazon the most money, not necessarily what's best for the user (unless the two happen to coincide.)
- charcircuit 3y ago
  Money is a proxy for user value and it is in Amazon's best interest for the customer to continually act upon these recommendations. If Amazon fails to deliver user value customers will be less willing to continue paying.
  - lukev 3y ago
    lol. lmao.
  - boolemancer 3y ago
    > Money is a proxy for user value and it is in Amazon's best interest for the customer to continually act upon these recommendations. If Amazon fails to deliver user value customers will be less willing to continue paying. You're talking about the same Amazon that will blindly give me recommendations for other products that fill exactly the same niche as the one I just bought. "Oh, you just bought a generator? You probably need a second one too, right?"

teaearlgraycold 3y ago

> it handles queries and documents differently to embed them in the same vector space I assume this might be behind the state of the art, then. As of a year ago OpenAI managed to get an embeddings model that can do both without any special flag or dual model to treat queries and documents differently.

mahjongmen1998 3y ago
handling queries and documents differently leads to an increase in overall retrieval.