5 ms·

In thinking of directions where LLM's could develop from here, I cant help but think that a models ability to self introspect would immensely improve their util

by pyaamb 1mo ago

In thinking of directions where LLM's could develop from here, I cant help but think that a models ability to self introspect would immensely improve their utility. The R&D on how to achieve that is beyond me though. How do you train someone how to introspect? Also would it require a continuous learning architecture that doesn't separate training and inference?

rolandog 1mo ago
Well, the algorithms in social media that are rewarding "engagement" (but accidentally ripping society apart by fueling rage) have made the training data — online discussions — a very poor training medium for introspection. So, if companies want training data on introspection, they would need to first fix what they broke (trust in each other and society) and have those discussions and realizations made public (by bringing back blogs where people might jot down their "maybe I've been an asshole" realizations) and rewarding other things than "engagement".
- queenkjuul 1mo ago
  I get the impression this isn't really what is meant by introspection here. I think it means much more plainly that the model is aware of its own "state of mind" so to speak, not so much the sense of reflecting on one's actions. I don't think you'd necessarily need training data about humans reflecting on their behavior for the former to come about in a model, i think it'd have more to do with the architecture of the model (does the model allow for "awareness of the state of mind")
Enginerrrd 1mo ago
I think it’s possible this might just require reworking the transformer architecture a bit to give it an inherent awareness of the contents of its attention. Kind of a working memory workspace of sorts. I.e. Instead of limiting autoregression to the token level, you introduce a persistent compressed global workspace latent memory vector that is fed back into the self-attention mechanism at every layer or every token step, allowing the network to attend to its own prior attentional states before computing the next token. Obviously that’s going to involve some compression steps. Trouble is… I think the architecture there is much simpler a tweak than figuring out how to train it. …that’s likely to just destabilize training for not much if any gain at first. You’re probably gonna have to resort to some really clever (and currently missing) tricks to figure out how to train the network to actually use that feature.
- astrange 1mo ago
  https://www.anthropic.com/research/global-workspace https://www.anthropic.com/research/global-workspace

by pyaamb 1mo ago

rolandog 1mo ago

Well, the algorithms in social media that are rewarding "engagement" (but accidentally ripping society apart by fueling rage) have made the training data — online discussions — a very poor training medium for introspection. So, if companies want training data on introspection, they would need to first fix what they broke (trust in each other and society) and have those discussions and realizations made public (by bringing back blogs where people might jot down their "maybe I've been an asshole" realizations) and rewarding other things than "engagement".

queenkjuul 1mo ago
I get the impression this isn't really what is meant by introspection here. I think it means much more plainly that the model is aware of its own "state of mind" so to speak, not so much the sense of reflecting on one's actions. I don't think you'd necessarily need training data about humans reflecting on their behavior for the former to come about in a model, i think it'd have more to do with the architecture of the model (does the model allow for "awareness of the state of mind")

Enginerrrd 1mo ago

I think it’s possible this might just require reworking the transformer architecture a bit to give it an inherent awareness of the contents of its attention. Kind of a working memory workspace of sorts. I.e. Instead of limiting autoregression to the token level, you introduce a persistent compressed global workspace latent memory vector that is fed back into the self-attention mechanism at every layer or every token step, allowing the network to attend to its own prior attentional states before computing the next token. Obviously that’s going to involve some compression steps. Trouble is… I think the architecture there is much simpler a tweak than figuring out how to train it. …that’s likely to just destabilize training for not much if any gain at first. You’re probably gonna have to resort to some really clever (and currently missing) tricks to figure out how to train the network to actually use that feature.

astrange 1mo ago
https://www.anthropic.com/research/global-workspace https://www.anthropic.com/research/global-workspace