6 ms·
Giving GPT “Infinite” Knowledge
- Beltiras 3y agoI'm working on something where I need to basically add on the order of 150,000 tokens into the knowledge base of an LLM. Finding out slowly I need to delve into training a whole ass LLM to do it. Sigh.
- RhodesianHunter 3y agoOr, at this rate, just wait 6 months.
- v3ss0n 3y agohttps://deepai.org/publication/scaling-transformer-to-1m-tokens-and-beyond-with-rmt https://deepai.org/publication/scaling-transformer-to-1m-tok... Can this be implemented in current opensource models?
- akvadrako 3y agoCan't you use fine-tuning for this? A other option is to ask GPT to compress your tokens into a shorter prompt for itself.
- sudoapps 3y agoIf you are wondering what the latest is on giving LLM's access to large amounts of data, I think this article is a good start. Seems like this is a space where there will be a ton of innovation so interested to learn what else is coming.
- furyofantares 3y agoEmbeddings-based search is a nice improvement on search, but it's still search. Relative to ChatGPT answering on its training data, I find embeddings-based search to be severely lacking. The right comparison is to traditional search, where it becomes favorable. It has the same advantages search has over ChatGPT (being able to cite sources, being quite unlikely to hallucinate) and it has some of the advantages ChatGPT has over search (not needing exact query) - but in my experience it's not really in the new category of information discovery that ChatGPT introduced us to. Maybe with more context I'll change my tune, but it's very much at the whim of the context retrieval finding everything you need to answer the query. That's easy for stuff that search is already good at, and so provides a better interface for search. But it's hard for stuff that search isn't good at, because, well: it's search.
- sudoapps 3y agoAgreed, GPT answering based on its own training data has been the best experience by far (aside from hallucinations) and comparing against that is difficult. Embeddings might not even be the long term solution. I think it's still early to really know for certain but models are already getting better at interpreting with less overall training data so there are bound to be some new ideas.
- b33j0r 3y agoI’m sure many of you have tried generating epic conversations from history. With work and luck, I’ve read stuff way better than college. But 90% of the time, it’s two barely distinct personalities chatting back and forth: Me: Hey brian, what do you think of AI? Brian: It’s great! Me: I’m so glad we agree. Brian: Great, this increases the training weight of Brian agreeing with Brian to a much more accurate level! Me: Agree!
- b33j0r 3y agoMany points stated well. Agree. Now, I’m not certain of this, but I’m starting to get an intuition that duct-taping databases to an agent isn’t going to be the answer (I still kinda feel like hundreds of agents might be). But these optimizations are applications of technology stacks we already know about. Sometimes, this era of AI research reminds me of all the whacky contraptions from the era before building airplanes became an engineering discipline. I would likely have tried building a backyard ornithopter powered by mining explosives, if I had been alive during that period of experimentation. Prediction: the best interfaces for this will be the ones we use for everything else as humans. I am trying to approach it more like that, and less like APIs and “document vs relational vs vector storage”.
- orasis 3y agoOne caveat about about embedding based retrieval is that there is no guarantee that the embedded documents will look like the query. One trick is to have a LLM hallucinate a document based on the query, and then embed that hallucinated document. Unfortunately this increases the latency since it incurs another round trip to the LLM.
- williamcotton 3y ago“We’re gonna need a bigger boat.”
- rco8786 3y ago> One trick is to have a LLM hallucinate a document based on the query I'm not following why you would want to do this? At that point, just asking the LLM without any additional context would/should produce the same (inaccurate) results.
- BoorishBears 3y agoYou're not having the LLM answer from the hallucination, you're looking for the document that looks most similar to the hallucination and having it answer on that instead.
- taberiand 3y agoIs that something easily handed off to a faster/cheaper LLM? I'm imagining something like running the main process through GPT-4 and hand of the hallucinations to GPT 3 turbo. If you could spot the need for it while streaming a response you could possibly even have it ready ahead of time
- wasabi991011 3y ago>One caveat about about embedding based retrieval is that there is no guarantee that the embedded documents will look like the query. Aleph Alpha provides an asymmetric embedding model which I believe is an attempt to resolve this issue (haven't looked into it much, just saw the entry in langchain's documentation)
- 3y ago
- pbhjpbhj 3y ago>There is an important part of this prompt that is partially cut off from the image: >> “If you don't know the answer, just say that you don't know, don't try to make up an answer” // It seems silly to make this part of the prompt rather than a separate parameter, surely we could design the response to be close to factual. Then run a checker to ascertain a score for the factuality of the output?
- sudoapps 3y agoA lot of what prompting has turned into seems silly to me too, but it has shown to be effective (at least with GPT-4).
- TeMPOraL 3y agoOnly a month or two ago I found this ridiculous, but then my mental model of GPTs shifted and I don't think it's so stupid anymore. Technobabble explanation: such "silly" additions are a natural way to emphasize certain dimensions of the latent space more than others, focusing the proximity search GPTs are doing. Working model I've been getting some good mileage off: GPT-4 is like a 4 year old kid, that somehow managed to read half of the Internet. Sure, it kinda remembers and possibly understands a lot, but it still thinks like a 4 year old, has about as much attention span, and you need to treat it like a kid that age.
- sudoapps 3y agoCompletely agree
- selfhoster11 3y agoWhenever I see a comment posted by you here on HN, I am never disappointed by the insights and/or quality within. This is definitely one of those. My personal mental model of GPT-4's capabilities is closer to that of an Atari 2600 - very capable, even if it only has 128 bytes of RAM. Except this time round, we can easily scale up a huge network of them (series of GPT-4 'threads') that each do a small micro-portion of the overall task, if a high degree of precision is required. When we get to a Commodore 64-magnitude AI, things will become a lot more interesting.
- Der_Einzige 3y agoI get annoyed by articles like this. Yes, it's cool to educate readers who aren't aware of embeddings/embeddings stores/vectorDB technologies that this is possible. What these articles don't touch on is what to do once you've got the most relevant documents. Do you use the whole document as context directly? Do you summarize the documents first using the LLM (now the risk of hallucination in this step is added)? What about that trick where you shrink a whole document of context down to the embedding space of a single token (which is how ChatGPT is remembering the previous conversations). Doing that will be useful but still lossey What about simply asking the LLM to craft its own search prompt to the DB given the user input, rather than returning articles that semantically match the query the closest? This would also make hybird search (keyword or bm25 + embeddings) more viable in the context of combining it with an LLM Figuring out which of these choices to make, along with an awful lot more choices I'm likely not even thinking about right now, is what will seperate the useful from the useless LLM + Extractive knowledge systems
- gaogao 3y ago> What about simply asking the LLM to craft its own search prompt to the DB given the user input, rather than returning articles that semantically match the query the closest? I played with that approach in this post - https://friend.computer/jekyll/update/2023/04/30/wikidata-llms.html https://friend.computer/jekyll/update/2023/04/30/wikidata-ll.... "Craft a query" is nice as it gives you a very declarative intermediate state for debugging.
- EForEndeavour 3y ago> What about that trick where you shrink a whole document of context down to the embedding space of a single token (which is how ChatGPT is remembering the previous conversations) This is news to me. Where could I read about this trick?
- deleted 3y ago[deleted]
- deleted 3y ago[deleted]
- nico 3y agoCan we build a model based purely on search? The model searches until it finds an answer, including distance and resolution Search is performed by a DB, the query then sub-queries LLMs on a tree of embeddings Each coordinate of an embedding vector is a pair of coordinate and LLM Like a dynamic dictionary, in which the definition for the word is an LLM trained on the word Indexes become shortcuts to meanings that we can choose based on case and context Does this exist already?
- fzliu 3y agoNot sure what you mean by dynamic dictionary, but the embedding tree you mention is already freely available Milvus via the Annoy index.
- nico 3y agoAn entry in a dictionary is static text, ex: per·snick·et·y: placing too much emphasis on trivial or minor details; fussy. "she's very persnickety about her food" A dynamic entry could instead be an LLM what will answer things related to they word, ex: What is the definition of persnickety? How can I use it in a sentence? What are some notable documents that include it? Any famous quotes? … So each entry is an LLM trained mostly only on that keyword/concept definition There are some that believe in smaller models: https://twitter.com/chai_research/status/1655649081035980802?s=46 https://twitter.com/chai_research/status/1655649081035980802...
- nadermx 3y agoI think someone did this https://github.com/pashpashpash/vault-ai https://github.com/pashpashpash/vault-ai
- xtracto 3y agoThis looks pretty promising, will check out later. Thanks for sharing
- chartpath 3y agoSearch query expansion: https://en.wikipedia.org/wiki/Query_expansion https://en.wikipedia.org/wiki/Query_expansion We've done this in NLP and search forever. I guess even SQL query planners and other things that automatically rewrite queries might count. It's just that now the parameters seem squishier with a prompt interface. It's almost like we need some kind of symbolic structure again.
- ftxbro 3y ago> "Once these models achieve a high level of comprehension, training larger models with more data may not offer significant improvements (not to be mistaken with reinforcement learning through human feedback). Instead, providing LLMs with real-time, relevant data for interpretation and understanding can make them more valuable." To me this viewpoint looks totally alien. Imagine you have been training this model to predict the next token. At first it can barely interleave vowels and consonants. Then it can start making words, then whole sentences. Then it starts unlocking every cognitive ability one by one. It begins to pass nearly every human test and certification exam and psychological test of theory of mind. Now imagine thinking at this point "training larger models with more data may not offer significant improvements" and deciding that's why you stop scaling it. That makes absolutely no sense to me unless 1) you have no imagination or 2) you want to stop because you are scared to make superhuman intelligence or 3) you are lying to throw off competitors or regulators or other people.
- joshspankit 3y agoMy takeaway from his statements is that if you sum up all of human knowledge then add every unique bit of knowledge that humans could uncover in the next 20 years, there’s a plateau and that plateau is probably lower than our dreams of what LLMs can do.
- sudoapps 3y agoThis wasn't mean't to say that all training would stop. I think, to some extent, the model won't need additional recent data (that is already similar in structure to what it has) to better understand language and interpret the next set of characters. I could be completely wrong, but I still think techniques like transformers, RLHF and of course others will still exist and evolve to eventually get to some higher intelligence level.
- vidarh 3y agoI think it's more a question of diminishing return and the cost of scaling it up, which is getting to a point where looking for ways of maximizing the impact of what is there makes sense. I'm sure we'll see models trained on more data, but maybe after efficiency improvements makes it cheaper both to train and run large models.
- m3kw9 3y agoThis is like asking gpt to summarize what it found on Google, this is basically what bing does when you try to find stuff like hotels and other recent subjects. Not the revolution we are all expecting
- jeffchuber 3y agohi everyone, this is jeff from Chroma (mentioned in the article) - happy to answer any questions.
- hartator 3y agoIs Chroma already trained or only trained in the supplied documents? I can try to make a Ruby client.
- jeffchuber 3y agoChroma is not an LLM, it is "just" a database that you pass vectors into to search. A Ruby client would be great. Our FastAPI spec makes this pretty easy - it's at localhost:8000/openapi.json when the docker backend is running.
- flukeshott 3y agoI wonder how effectively compressed LLMs are going to become...
- iot_devs 3y agoA similar idea is been developed in: https://github.com/pieroit/cheshire-cat https://github.com/pieroit/cheshire-cat
- A_D_E_P_T 3y ago"Infinite" is a technical term with a highly specific meaning. In this case, it can't possibly be approached. It certainly can't be attained. Borges' Library of Babel, which represents all possible combinations of letters that can fit into a 400-page book, only contains some 25^1312000 books. And the overwhelming majority of its books are full of gibberish. The amount of "knowledge" that a LLM can learn or describe is VERY strictly bounded and strictly finite. (This is perhaps its defining characteristic.) I know this is pedantic, but I am a philosopher of mathematics and this is a matter that's rather important to me.
- hartator 3y ago> I know this is pedantic, but I am a philosopher of mathematics and this is a matter that's rather important to me. I don’t think this is pedantic. Words carry a specific meaning or what’s the point of words otherwise.