6 ms·

>Yet the answers will get outdated quickly whilst the silicon is fixed. Bro is living in 2020 before rag was widely introduced.

mdp2021 1mo ago
> before rag was widely introduced And when did RAG start to work properly as a mature, reliable technology?
- fennecbutt 1mo ago
  I mean, it's pretty damn reliable now. Has been for simple high contrast Q&A for a while now. I've just been doing research and experiments for work related stuff. Typically we've used plain embeddings for a lot of high contrast documents aka discrete facts. However I've been working with a >1000 page document of complex procedures with incredibly low contrast where embedding falls flat. There's top down/graph searching, bottom up/embedded; alts like colbert, reranking, reasoning, search agents and now (though seemingly quite new) specific search agent models. Ultimately I found that a reasoning enabled search agent doing a hybrid of bottom up (with reranking) followed by top down, gave the absolute best results. Paired with Luna for cheaper and faster tokens it benchmarks pretty well even for vague references to procedures. I would imagine that search specific models just coming out are even better and I'll have to evaluate using these but for now the above works well for us. Having an agent get vector search results to use as anchors and then being able to explore the sections and subsections above that, then eventually digesting as much as is relevant (big context, cheap tokens) is amazing.
  - mdp2021 1mo ago
    Very interesting and very good (though, let us say, much more recent that ancient): only, it paradoxically does suggest that the computational cost (when compared either to a simple "search by vectors distance" or to the task failures that we can have witnessed by chief implementations*) strongly suggest the benefit of 1000x speed boosting, energy conservative Taalas board. *(Of course it has "always" worked well for «simple high contrast Q&A», ever since the base embeddings technology worked properly: that is almost by definition; it is on real world use cases, where the nuances of reality are present, that it failed miserably.)
paulryanrogers 1mo ago
I was told updates require replacing at least two layers of metal, though not whole thing. Was that not accurate? Can you say more?
- mdp2021 1mo ago
  You are talking about two different things. Yes, to update the blueprint for new models two layers will be updated. That is the NN. To instead update the data on which to operate you could use a RAG to query. (As in "the Pathfinder 2.0 NN is on the chip; the geodata is in the OpenGeoMaps dump-DB-nightly" - not really overlapping with LLM+RAG but may give an idea in a different scenario.)
- Closi 1mo ago
  You can change the prompt, and give the agent tools, and that keeps the agent up to date. The agent doesn't know the date, or know what hotels there are in Montreal, it sees: > System: You are an AI agent. The date is 11th August 2026. Your knowledge cut-off is March 2023. User is based in <date>. If you need to search for something to support the user say {search:<term>} and a list of options will be provided along with instructions on how to access. Or say {help} for a full list of commands. > User: Can you help me find hotels in Montreal for next weekend? The AI then interacts with the tools given in the base prompt, which can obviously be updated. So it then goes: > AI: Of course, let me search for that. {search: hotels in montreal for 16th August} > System: [Provides list of websites]. Say {read[n]} to read option or say {start subagent:<goal>) to register subagent. > AI: {start subagent: List hotels on booking.com available on 16th August}. [etc etc, then eventually] > AI: Yes I have searched for you and I found a few options! While you can't change embedded knowledge, a good model knowing that the date is 5th January 2040 can infer certain things (e.g. while it might not have been trained on certain deaths, it can probably guess that it should search before answering if it means a person would be 102 and their last information is from 2024)

mdp2021 1mo ago

> before rag was widely introduced And when did RAG start to work properly as a mature, reliable technology?

fennecbutt 1mo ago
I mean, it's pretty damn reliable now. Has been for simple high contrast Q&A for a while now. I've just been doing research and experiments for work related stuff. Typically we've used plain embeddings for a lot of high contrast documents aka discrete facts. However I've been working with a >1000 page document of complex procedures with incredibly low contrast where embedding falls flat. There's top down/graph searching, bottom up/embedded; alts like colbert, reranking, reasoning, search agents and now (though seemingly quite new) specific search agent models. Ultimately I found that a reasoning enabled search agent doing a hybrid of bottom up (with reranking) followed by top down, gave the absolute best results. Paired with Luna for cheaper and faster tokens it benchmarks pretty well even for vague references to procedures. I would imagine that search specific models just coming out are even better and I'll have to evaluate using these but for now the above works well for us. Having an agent get vector search results to use as anchors and then being able to explore the sections and subsections above that, then eventually digesting as much as is relevant (big context, cheap tokens) is amazing.
- mdp2021 1mo ago
  Very interesting and very good (though, let us say, much more recent that ancient): only, it paradoxically does suggest that the computational cost (when compared either to a simple "search by vectors distance" or to the task failures that we can have witnessed by chief implementations*) strongly suggest the benefit of 1000x speed boosting, energy conservative Taalas board. *(Of course it has "always" worked well for «simple high contrast Q&A», ever since the base embeddings technology worked properly: that is almost by definition; it is on real world use cases, where the nuances of reality are present, that it failed miserably.)

paulryanrogers 1mo ago

I was told updates require replacing at least two layers of metal, though not whole thing. Was that not accurate? Can you say more?

mdp2021 1mo ago
You are talking about two different things. Yes, to update the blueprint for new models two layers will be updated. That is the NN. To instead update the data on which to operate you could use a RAG to query. (As in "the Pathfinder 2.0 NN is on the chip; the geodata is in the OpenGeoMaps dump-DB-nightly" - not really overlapping with LLM+RAG but may give an idea in a different scenario.)
Closi 1mo ago
You can change the prompt, and give the agent tools, and that keeps the agent up to date. The agent doesn't know the date, or know what hotels there are in Montreal, it sees: > System: You are an AI agent. The date is 11th August 2026. Your knowledge cut-off is March 2023. User is based in <date>. If you need to search for something to support the user say {search:<term>} and a list of options will be provided along with instructions on how to access. Or say {help} for a full list of commands. > User: Can you help me find hotels in Montreal for next weekend? The AI then interacts with the tools given in the base prompt, which can obviously be updated. So it then goes: > AI: Of course, let me search for that. {search: hotels in montreal for 16th August} > System: [Provides list of websites]. Say {read[n]} to read option or say {start subagent:<goal>) to register subagent. > AI: {start subagent: List hotels on booking.com available on 16th August}. [etc etc, then eventually] > AI: Yes I have searched for you and I found a few options! While you can't change embedded knowledge, a good model knowing that the date is 5th January 2040 can infer certain things (e.g. while it might not have been trained on certain deaths, it can probably guess that it should search before answering if it means a person would be 102 and their last information is from 2024)