> I think that's... up for debate Been trying to inform myself on how these models work, and it’s pretty interesting, I have to say. Came across this paper fr

> I think that's... up for debate Been trying to inform myself on how these models work, and it’s pretty interesting, I have to say. Came across this paper from Anthropic, Scaling Monosemanticity [0], where they’re extracting features (via a trained sparse autoencoder) from the ‘middle’ layer of Claude 3 for the purpose of interpretability, and quite convincingly find features corresponding to abstract concepts that do seem to encode a model of computation of sorts. Most remarkable to me is their example of a feature that activates for functions implementing addition, which holds up under function composition. I guess there’s more going on under the hood than I’ve been giving credit. Of course, that one example is a tiny window into it, and I recognise that even being able to extract that kind of insight into the model’s workings is a feat. > What they don't have is an ability to infer constraints you didn't explicitly or implicitly spell out. And only limited capability to ask clarifying question. Interesting to think about how the concept of clarification can be formalized and whether it’s possible to work in to the next-token prediction paradigm. I have too many holes in my understanding at this point to go much further with the idea... > They also are horrible at correcting mistakes through ongoing conversation I guess this one is somewhat understandable with how the models work, though it’s unfortunate that the typical chat interface strongly encourages you to attempt to resolve your issue through conversation. > a lot of code is in between those two lane markers Yup. [0]: https://transformer-circuits.pub/2024/scaling-monosemanticity/index.html https://transformer-circuits.pub/2024/scaling-monosemanticit...