6 ms·
Instinctively this feels like a two phase problem - start with some machine translation into a single spoken language and index that, then when people are query
by jon-wood 22d ago
Instinctively this feels like a two phase problem - start with some machine translation into a single spoken language and index that, then when people are querying do the same thing. When returning search results show them in the original language.
- whilenot-dev 22d agoWhy not create indexes for multiple languages, as that would also avoid double translation issues (e.g. GER [query] → ENG [index] → GER [document])?
- j0selit0 21d agoyou would also need to maintain multiple indexes in multiple languages. I never had to do that - but I assume it's a pain
- whilenot-dev 19d agoIt's a matter of running a for-loop. You'd get faster response times (no query→index translation), but the storage requirements for the indexes would be larger.
- kaon_2 22d agoYes we've tried. It works. But jargon is hard. RAG with embeddings works all the same. The LLM doesn't mind receiving sources in Italian, french and German, and then outputting the answer in Japanese while providing the verbatim German jargon term in brackets
- jameshart 22d agoEmbedding search is effectively machine translation into a single common ‘language’ - embedding space - and then searching that; cleaner and less lossy than translating everything into English for searching, but harder to debug when it goes wrong.