9 ms·
Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet
- byteknight 2y agoThis reminds me of how people often communicate to avoid offending others. We tend to soften our opinions or suggestions with phrases like "What if you looked at it this way?" or "You know what I'd do in those situations." By doing this, we subtly dilute the exact emotion or truth we're trying to convey. If we modify our words enough, we might end up with a statement that's completely untruthful. This is similar to how AI models might behave when manipulated to emphasize certain features, leading to responses that are not entirely genuine.
- nathan_compton 2y agoCounterpoint: "What if you looked at it this way?" communicates both your suggestion AND your sensitivity to the person's social status whatever. Given that humans are not robots, but social, psychological, animals, such communication is entirely justified and efficient.
- byteknight 2y agoYou can't always do both to the fullest truth. They often conflict. To do what you suggest, would imply my feelings perfectly align with the sympathetic view. That is not the case for a lot of humans or instances. If I am not saying exactly how I feel it is watered down. And telling me "just do both" is enforcing your world view and that is precisely what we're talking about _not_ doing.
- infogulch 2y agoThe "fullest truth" includes your desired outcome and knowledge that they are a human. If you just want to dump facts at them and get them to shut up, go ahead and speak unfiltered. Twitter may be an example of the outcome of that strategy. Consider a situation where you are teaching a child. She tries her best and makes a mistake on her math homework. Saying that her attempt was terrible because an adult could do better may be the "fullest truth" in the most eye-rolling banal way possible, and discourages her from trying in the future which is ultimately unproductive. This "fullest truth" argument fails to take into account desire and motivation, and thus is a bad model of the truth.
- nathan_compton 2y agoIts rarely the case that speaking without considering other people's feelings will be the optimal method of getting what you want, even if you are a sociopath and what you want doesn't have anything to do with other people's well beings. In fact, sociopaths are a great example: they are typically quite adept at communicating in such a way as to ingratiate themselves with others. If even a sociopath gets this, then you might want to consider the wisdom of following suit.
- harshaxnim 2y agoSadly "sensitivity" has been over done. It's a fine line and corporations would rather cross it for legal/social reasons. Similar to how too much political correctness will hamper the society, so does the overly done sensitivity in an agent, be it a human, or AI.
- nathan_compton 2y agoThat might be the case, but how and who determines how much is too much? I mean in the case of AI, let the market decide seems like the right answer.
- HarHarVeryFunny 2y agoA true AGI would learn to manipulate it's environment to achieve it's goals, but obviously we are not there yet. An LLM has no goals - it's just a machine optimized to minimize training errors, although I suppose you could view this as an innate hard-coded goal of minimizing next word error (relative to training set), in same way we might say a machine-like insect has some "goals". Of course RLHF provides a longer time span (entire response vs next word) error to minimize, but I doubt training volume is enough for the model to internally model a goal of manipulating the listener as opposed to just favoring surface forms of response.
- Nevermark 2y agoAn LLM has no explicit goals. But simply by approximating human communication which often models goal oriented behavior, an LLM can have implicit goals. Which likely vary widely according to conversation context. Implicit goals can be very effective. Nowhere in DNA is there any explicit goal to survive. However combinations of genes and markers selected for survivability create creatures with implicit goals to survive as tenacious as any explicit goals might be.
- HarHarVeryFunny 2y agoYes, the short term behavior/output of the LLM could reflect an implicit goal, but I doubt it'd maintain any such goal for an extended period of time (long-term coherence of behavior is a known shortcoming), since there is random sampling being done, and no internal memory from word to word - it seems that any implicit goal will likely rapidly drift.
- Nevermark 2y agoAgreed. They can’t accurately model our communication for very long. So any implicit motives are limited to that. But their capabilities are improving rapidly in both kind and measure.
- rpozarickij 2y agoThe next big breakthrough in the LLM space will be having a way to represent goals/intentions of the LLM and then execute them in the way that is the most appropriate/logical/efficient (I'm pretty sure some really smart people have been thinking about this for a while). Perhaps at some point LLMs will start to evolve from the prompt->response model into something more asynchronous and with some activity happening in the background too.
- optimalsolver 2y ago>what the model is "thinking" before writing its response An actual "thinking machine" would be constantly running computations on its accumulated experience in order to improve its future output and/or further compress its sensory history. An LLM is doing exactly nothing while waiting for the next prompt.
- fassssst 2y agoWhy does the timing of the “thinking” matter?
- verdverm 2y agothinking is generally considered an internal process, without input/output (of tokens), though some people decide to output some of that thinking into a more permanent form I see thinking as less about "timing" and more about a "process" What this post seems to be describing is more about where attention is paid and what neurons fire for various stimuli
- sabrina_ramonov 2y agowe know so little about thinking and consciousness, these claims seem premature
- verdverm 2y agoThat one can fix the RNG and get consistent output indicates a lack of dynamics They certainly do not self update the weights in an online process as needed information is experienced
- whimsicalism 2y agoIf we could perfectly simulate the brain and there were quantum hidden variables, we too could “fix RNG and get deterministic output”
- 2y ago
- feverzsj 2y agoSo they made a system by trying out thousands of combinations to find the one gives best result, but they don't understand what's actually going on inside.
- whimsicalism 2y agoI continue to be impressed by Anthropic’s work and their dual commitment to scaling and safety. HN is often characterized by a very negative tone related to any of these developments, but I really do feel that Anthropic is trying to do a “race to the top” in terms of alignment, though it doesn’t seem like all the other major companies are doing enough to race with them. Particularly frustrating on HN is the common syllogism of: 1. I believe anything that “thinks” must do X thing. 2. LLM doesn’t do X thing 3. LLM doesn’t think X thing is usually both poorly justified as constitutive of thinking (usually constitutive of human thinking but not writ large) nor is it explained why it matters whether the label of “thinking” applies to LLM or not if the capabilities remain the same.
- phyalow 2y agoAlot of this really isnt new, Andrej Karpathy covered the principles here 8 years ago for CS231n at Stanford https://youtu.be/yCC09vCHzF8&t=1640 https://youtu.be/yCC09vCHzF8&t=1640
- whimsicalism 2y agoneural probing has been around for a while, true - and this result is definitely building on past results. it’s basically just a scaled up version of their paper from a little while ago anywho but Karpathy was looking at very simple LSTMs of 1-3 layers, looking at individual nodes/cells, and these results have generally thus far been difficult to replicate among large scale transformers. Karpathy also doesn’t provide a recipe for doing this in his paper, which makes me think he was just guess and checking various cells. The representations discovered are very simple
- kalkin 2y agoThis is an illustrative comment for meta reasons, I think. Karpathy's lecture almost certainly doesn't cover the superposition hypothesis (which hadn't been invented for ANNs 8 years ago), or sparse dictionary learning (whose application to ANNs is motivated by the superposition hypothesis). It certainly doesn't talk about actual specific features found in post-ChatGPT language models. What's happening here seems like a thing LLMs are often accused of dismissively - you're pattern-matching to certain associated words without really reasoning about what is or isn't new in this paper. I worry this is going to come across as insulting, but that's not my intention. I do this too sometimes; I think everyone does. The point is we shouldn't define true reasoning so narrowly that we think no system capable of it would ever be caught doing what most of us are in fact doing most of the time.
- null_point 2y agoStrategic timing for the release of this paper. As of last week OpenAI looks weak in their commitment to _AI Safety_, losing key members of their Super Alignment team.
- tel 2y agoThis is exceptionally cool. Not only is it very interesting to see how this can be used to better understand and shape LLM behavior, I can’t help but also think it’s an interesting roadmap to human anthropology. If we see LLMs as substantial compressed representations of human knowledge/thought/speech/expression—and within that, a representation of the world around us—then dictionary concepts that meaningfully explain this compressed representation should also share structure with human experience. I don’t mean to take this canonically, it’s representations all the way down, but I can’t help but wonder what the geometry of this dictionary concept space says about us.
- davedx 2y agoThe vector space projection of the human experience. I like it.
- wwarner 2y agohuge. the activation scan, which looks for which nodes change the most when prompted with the words "Golden Gate Bridge" and later an image of the same bridge, is eerily reminiscent of a brain scan under similar prompts...
- verdverm 2y agoI find this outcome expected and not really surprising, more confirmation of previous results. Consider vision transformers and the papers that showed what each layer was focused on.
- wwarner 2y agowell that's exactly the point -- no such result is available for language models.
- verdverm 2y agoThere are multiple papers and efforts that have inspected the internal state of LLMs. One could even see the word2vec analysis along these lines, as evidence that the model is specializing neurons One such example: The Internal State of an LLM Knows When It's Lying (https://arxiv.org/abs/2304.13734 https://arxiv.org/abs/2304.13734) Searching phrases like "llm interpretability" and "llm activation analysis" uncover more https://github.com/JShollaj/awesome-llm-interpretability https://github.com/JShollaj/awesome-llm-interpretability
- wwarner 2y agoYes, lots of activity in the space. I thought you were saying it was a dumb problem, but I was wrong. I think this is a great paper.
- verdverm 2y agoyup, if you look at drop out, what it does and why, you can see additional interesting results along these lines (drop-out was found to increase resilience in models because they had to encode information in the weights differently, i.e. could not rely on single neuron (at the limit))
- pdevr 2y agoSo, to summarize: >Used "dictionary learning" >Found abstract features >Found similar/close features using distance >Tried amplifying and suppressing features Not trying to be snary, but sounds mundane in the ML/LLM world. Then again, significant advances have come from simple concepts. Would love to hear from someone who has been able to try this out.
- sjkoelle 2y agothe interesting advance in the anthropic/mats research program is the application of dictionary learning to the "superpositioned" latent representations of transformers to find more "interpretable" features. however, "interpretability" is generally scored by the explainer/interpreter paradigm which is a bit ad hoc, and true automated circuit discovery (rather than simple concept representation) is still a bit off afaik.
- youssefabdelm 2y agoFor anyone who has read the paper, have they provided code examples or enough detail to recreate this with, say, Llama 3? While they're concerned with safety, I'm much more interested in this as a tool for controllability. Maybe we can finally get rid of the woke customer service tone, and get AI to be more eclectic and informative, and less watered down in its responses.
- tantalor 2y agoI always assumed the way to map these models would be by ablation, the same way we map the animal brain. Damage part X of the network and see what happens. If the subject loses the ability to do Y, then X is responsible for Y. See https://en.wikipedia.org/wiki/Phineas_Gage https://en.wikipedia.org/wiki/Phineas_Gage
- rmorey 2y agoWe are so far ahead in the case of these models - we already have the complete wiring diagram! In biological systems we have only just begun to be able to create the complete neuronal wiring diagrams - currently worms, flies, perhaps soon mice
- bilsbie 2y agoHow are they handling attention in their approach? That’s going to completely change what features are looked at.
- tel 2y agoThey target the residual stream. Also they may have a definition of “feature” that’s more general than what you’re using. Consider reading their superposition work.
- bilsbie 2y agoIf anyone wants to team up and work on stuff like this (on toy models so we can run locally) please get in touch. (Email in profile) I’m so fascinated by this stuff but I’m having trouble staying motivated in this short attention span world.
- Mobil1 2y ago[dead]
- quotemstr 2y agoAt this risk of anthropomorphizing too much, I can't help but see parallels between the "my physical form is the Golden Gate Bridge" screenshot and the https://en.wikipedia.org/wiki/God_helmet https://en.wikipedia.org/wiki/God_helmet in humans --- both cognitive distortions caused by targeted exogenous neural activation.
- bjterry 2y agoIt would be interesting to allow users of models to customize inference by tweaking these features, sort of like a semantic equalizer for LLMs. My guess is that this wouldn't work as well as fine-tuning, since that would tweak all the features at once toward your use case, but the equalizer would require zero training data. The prompt itself can trigger the features, so if you say "Try to weave in mentions of San Francisco" the San Francisco feature will be more activated in the response. But having a global equalizer could reduce drift as the conversation continued, perhaps?
- ericflo 2y agoRelated: https://vgel.me/posts/representation-engineering/ https://vgel.me/posts/representation-engineering/
- bjterry 2y agoThanks!
- kromem 2y agoAt least for right now this approach would in most cases still be like using a shotgun instead of a scalpel. Over the next year or so I'm sure it will refine enough to be able to be more like a vector multiplier on activation, but simply flipping it on in general is going to create a very 'obsessed' model as stated.
- pagekicker 2y agoThe article doesn't explain how users can exploit these features in UI or prompt. Does anyone have any insight on how to do so?
- CephalopodMD 2y agoThey explicitly aren't releasing any tools to do this with their models for safety reasons. But you could probably do it from scratch with one of the open models by following their methodology.
- parentheses 2y agoIt's interesting that they used this to manipulate models. I wonder if "intentions" can be found and tuned. That would have massive potential for use and misuse. I could imagine a villain taking a model and amplifying "the evil" using a similar technique.
- parentheses 2y agoI wonder how interpretability and training can interplay. Some examples: Imagine taking Claude, tweaking weights relevant to X and then fine tuning it on knowledge related to X. It could result in more neurons being recruited to learn about X. Imagine performing this during training to amplify or reduce the importance of certain topics. Train it on a vast corpus, but tune at various checkpoints to ensure the neural network's knowledge distribution skews. This could be a way to get more performance from MoE models. I am not an expert. Just putting on my generalist hat here. Tell me I'm wrong because I'd be fascinated to hear the reasons.
- sanxiyn 2y agoSomeone should do this for Llama 3.
- e63f67dd-065b 2y agoI find Anthorpic's work on mech interp fascinating in general. Their initial towards monosemanticity paper was highly surprising, and so is this with the ability to scale to a real production-scale LLM. My observation is, and this may be more philosophical than technical: this process of "decomposing" middle-layer activations with a sparse autoencoder -- is it capturing accurately underlying features in the latent space of the network, or are we drawing order from chaos, imposing monosemanticity where there aren't any? Or to put it another way, were the features always there, learnt by training, or are we doing post-hoc rationalisations -- where the features exist because that's how we defined the autoencoders' dictionaries, and we learn only what we wanted to learn? Are the alien minds of LLMs truly also operating on a similar semantic space as ours, or are we reading tea leaves and seeing what we want to see? Maybe this distinction doesn't even make sense to begin with; concepts are made by man, if clamping one of these features modifies outputs in a way that is understandable to humans, it doesn't matter if it's capturing some kind of underlying cluster in the latent space of the model. But I do think it's an interesting idea to ponder.
- refulgentis 2y agoI'm allergic to latent space because I've yet to find any meaning to it beyond poetics, I develop an acute allergy when it's explicitly related to visually dimensional ideas like clustering. I'll make a probably bad analogy: does your mindmap place things near each other like my mindmap? To which I'd say, probably not, mindmaps are very personal, and the more complex we put on ours, the more personal and arbitrary they would be, and the less import the visuals would have ex. if we have 3 million things on both our mindmaps, it's peering too closely to wonder why you put mcdonalds closer to kids food than restaurants, and you have restaurants in the top left, whereas I put it closer to kids foods, in the top mid left.
- anentropic 2y agowhat if you averaged over millions of peoples' mindmaps?
- TeMPOraL 2y agoWhy would that matter? The absolute orientation of the mind map doesn't matter - maybe my map is actually very close to yours, subject to some rotation and mirroring? More than that, I'd think a better 2D analogy for the latent space is a force-directed graph that you keep shaking as you add things to it. It doesn't seem unlikely for two such graphs, constructed in different order, to still end up identical in the end. Thirdly: > if we have 3 million things on both our mindmaps, it's peering too closely to wonder why you put mcdonalds closer to kids food than restaurants, and you have restaurants in the top left, whereas I put it closer to kids foods, in the top mid left. In 2D analogy, maybe, but that's because of limited space. In 20 000 D analogy, there's no reason for our mind maps to meaningfully differ here; there's enough dimensions that terms can be close to other terms for any relationship you could think of.
- gautomdas 2y agoI've really been enjoying their series on mech interp, does anyone have any other good recs?
- kromem 2y agoThe Othello-GPT and Chess-GPT lines of work. Was the first research work that clued me into what Anthropic's work today ended up demonstrating.
- PoignardAzur 2y ago"Transformers Represent Belief State Geometry in their Residual Stream": https://www.lesswrong.com/posts/gTZ2SxesbHckJ3CkF/transformers-represent-belief-state-geometry-in-their https://www.lesswrong.com/posts/gTZ2SxesbHckJ3CkF/transforme... Basically finding that transformers don't just store a world-model as in "what does the world that produce the observed inputs look like?", they store a "Mixed-State Presentation", basically a weighted set of possible worlds that produce the observed inputs.
- kromem 2y agoGreat work as usual. I was pretty upset seeing the superalignment team dissolve at OpenAI, but as is typical for the AI space, the news of one day was quickly eclipsed by the next day. Anthropic are really killing it right now, and it's very refreshing seeing their commitment to publishing novel findings. I hope this finally serves as the nail in the coffin on the "it's just fancy autocomplete" and "it doesn't understand what it's saying, bro" rhetoric.
- jwilber 2y agoLove Anthropic research. Great visuals between Olah, Carter, and Pearce, as well. I don’t think this paper does much in the way of your final point, “it doesn’t understand what it’s saying”, though our understanding certainly has improved.
- kromem 2y agoThey were able to demonstrate conceptual vectors that were consistent across different languages and different mediums (text vs images) and that when manipulated were able to represent the abstract concept in the output regardless of prompt. What kind of evidentiary threshold would you want if that's not sufficient?
- jwilber 2y agoMy point is that you claimed this is a rebuff against those claiming models don’t understand themselves. Your interpretation seems to assign intelligence to the algorithms. While this research allows us to interpret larger models in an amazing way, it doesn’t mean the models themselves ‘understand’ anything. You can use this on much smaller scale models as well, as they showed 8 months ago. Does that research tell us about how models understand themselves? Or does it help us understand how the models work?
- kromem 2y ago"Understand themselves" is a very different thing than "understand what they are saying." Which exactly are we talking about here? Because no, the research doesn't say much about the former, but yes, it says a lot about the latter, especially on top of the many, many earlier papers working in smaller toy models demonstrating world modeling.
- gdiamos 2y agoIt looks like Anthropic is now leading the charge on safety
- maherbeg 2y agoThey always were given that is a part of their mission.
- maciejgryka 2y agoI recorded myself trying to read through and understand the high-level of this if anyone's interested in following along: https://maciej.gryka.net/papers-in-public/#scaling-monosemanticity https://maciej.gryka.net/papers-in-public/#scaling-monoseman...
- HanClinto 2y agoReminds me of this paper from a couple of weeks ago that isolated the "refusal vector" for prompts that caused the model to decline to answer certain prompts: https://news.ycombinator.com/item?id=40242939 https://news.ycombinator.com/item?id=40242939 I love seeing the work here -- especially the way that they identified a vector specifically for bad code. I've been trying to explore the way that we can use adversarial training to increase the quality of code generated by our LLMs, and so using this technique to get countering examples of secure vs. insecure code (to bootstrap the training process) is really exciting. Overall, fascinating stuff!!
- watersb 2y agoAm I the only one to read 'monosemanticity' as 'moose-mantically'? Like, its talking about moose magick...
- astrange 2y ago> Many features are multilingual (responding to the same concept across languages) and multimodal (responding to the same concept in both text and images), as well as encompassing both abstract and concrete instantiations of the same idea (such as code with security vulnerabilities, and abstract discussion of security vulnerabilities). This seems like it's trivially true; if you find two different features for a concept in two different languages, just combine them and now you have a "multilingual feature". Or are all of these features the same "size"? They might be and I might've missed it.
- wrycoder 2y agoThey are trying to figure out what they actually built. I suspect the time is coming when there will always be an aligned search AI between you and the internet.
- deleted 2y ago[deleted]
- justanotherjoe 2y agoMy thoughts - LLM Just got a whole set of buttons you can push. Potential for the LLM to push its own buttons? - Read the paper and ctrl+f 'deplorable'. This shows once again how we are underestimating LLM's ability to appear conscious. It can be really effective. Reminiscence of Dr.Ford in Westworld :'you (robots) never look more human than when you are suffering.' Or something like that, anyway. I might be hallucinating dialogue but pretty sure something like that was said and I think it's quite true. - Intensely realistic roleplaying potential unlocked. - Efficiency by reducing context length by directly amplifying certain features instead. Very powerful stuff. I am waiting eagerly when I can play with it myself. (Someone please make it a local feature)