4 ms·
What does this even mean. There is strong evidence of LLMs doing in context learning. Some of the linear RNN layers in recent models are provably doing SGD in
by angusturner 14d ago
What does this even mean. There is strong evidence of LLMs doing in context learning.
Some of the linear RNN layers in recent models are provably doing SGD in hidden space during inference
- oblio 14d ago> What does this even mean. There is strong evidence of LLMs doing in context learning. 1. Is this learning persistent? 2. Do they verify these new lessons against core principles? 3. Do they and protect themselves/ignore requests if these new lessons contradict those core principles? Humans do that from the time they're 3 years old (not that well, but they do do it).
- NuclearPM 14d agoYes. Yes. Yes.
- oblio 14d agoIn my experience all those claims are false. So the next step is to ask for evidence and ideally independent and peer reviewed research.
- CamperBob2 14d agoYou know they can take notes, right? And ICL dates all the way back to 2020, at least: https://arxiv.org/abs/2005.14165 https://arxiv.org/abs/2005.14165
- oblio 14d agoThat's training during training. They can't learn afterwards.
- deleted 14d ago[deleted]
- DoctorOetker 14d agodo you have a reference where that claim is demonstrated?
- gtirloni 13d agoBy in context learning do you mean latent space? https://transformer-circuits.pub/2021/framework/index.html https://transformer-circuits.pub/2021/framework/index.html