10 ms·
How did you come up with the long-term memory idea?
by crazypython 4y ago
How did you come up with the long-term memory idea?
- gamegoblin 4y agoSeemed vaguely like how the brain does it. You think a thought, and somehow your brain conjures memories that are semantically related to that thought. That sounds a lot like a nearest neighbors algorithm on a language model embedding vector to me.
- visarga 4y agoThere is a paper from DeepMind RETRO - Improving language models by retrieving from trillions of tokens (2021) > With a 2 trillion token database, our Retrieval-Enhanced Transformer (Retro) obtains comparable performance to GPT-3 and Jurassic-1 on the Pile, despite using 25× fewer parameters. https://www.deepmind.com/publications/improving-language-models-by-retrieving-from-trillions-of-tokens https://www.deepmind.com/publications/improving-language-mod...