7 ms·
A global workspace in language models
- nothrowaways 2mo agoThese guys dont know what they are doing yet they have a good LLM, thanks to the GPUs.
- deleted 2mo ago[deleted]
- alchemico 2mo agoI wrote a Claude skill that attempts to leverage these findings: https://github.com/juddkussrow/mantra https://github.com/juddkussrow/mantra
- dangoodmanUT 2mo agoJ-space sounds oddly similar to...
- amarant 2mo ago.... Space Jam?
- esafak 2mo agoWithout using the term, they are using an information geometric approach.
- blauditore 2mo agoBut J-Space is much catchier. This is not a scientific paper, it's a promotional essay.
- viralsink 2mo agoFirst button on the page is a link to the scientific paper. It's called "Read the paper". You'll find an explanation for the term in there.
- meatmanek 2mo agoIt would be really cool if they could expose this information to customers somehow. Imagine: - having a log of the most prominent J-space tokens during your customer support chatbot's interactions with a user, so you can have more introspection into why a particular outcome happened - being able to detect certain thoughts associated with undesirable behavior (hallucinations, overstepping authority, lying, etc.) and trigger some sort of remediation (e.g. upgrading to a better model, redirecting to a human, forcing tool calls)
- dofm 2mo agoPresumably the rationale for the decision to abridge the thinking traces will ensure that they don’t; if this is real (and there’s no good reason to trust that it is yet) then it is the secret sauce.
- charcircuit 2mo agoAnthropic aren't even willing to expose the CoT of their models. You will have to rely on them to build those sorts of things into dedicated signals.
- throw310822 2mo agoAnthropic won't do it, but they published the j-lens to introspect the model- from what I understand it's roughly simply feeding a chosen layer straight into the final layers of the LLM for decoding into language: https://github.com/anthropics/jacobian-lens https://github.com/anthropics/jacobian-lens Looks like it should be easy to use on open weights models.
- eamag 2mo agoIs it scaling up of https://openreview.net/forum?id=w7LU2s14kE https://openreview.net/forum?id=w7LU2s14kE with some changes on where this method is applied?
- aesthesia 2mo agoThey cite that paper as related work, but I don't think it's just a scaled-up version.
- bilsbie 2mo agoI’m confused where in the weights the jspace is.
- epolanski 2mo agoTokens that are activated but not present in it's output maybe? I too have confusion.
- lucrbvi 2mo agoAnthropic theorize that middle layers in an LLM is a "J-Space" used to "think" about the future answer or about abstract concepts. Their method is used to identify which tokens can appears in which layers of the model.
- nh23423fefe 2mo agoIt's not in the weights. Sounds to me like jspace is the "positive cone" over relevant (large norm) j-lenses, and j-lenses are gradients wrt tokens on the residual stream when you average over some training data.
- wongarsu 2mo agoThere was a series of blog posts posted to HN a while ago investigating how models behave on similar prompts in different languages. To paraphrase the results: the first couple layers map the query to some internal encoding that's mostly independent of the language. Then there are layers in the middle, then the last couple layers map the result back to the target language. You can actually take those middle layers and repeat them, and you get a stronger model. Those middle layers would be what Anthropic calls the J-Space, and their J-Lens maps activity in those layers back to tokens that trigger similar activity (with a technique they only drop hints at) The finding that you can repeat the middle layers pairs neatly with Anthropic's finding that there is some internal CoT-like process happening in them. I'm not sure how to find those blog posts, but maybe someone else remembers them
- steveklabnik 2mo agoHere's Anthropic on this topic, last year https://www.anthropic.com/research/tracing-thoughts-language-model https://www.anthropic.com/research/tracing-thoughts-language... > Recent research on smaller models has shown hints of shared grammatical mechanisms across languages. We investigate this by asking Claude for the "opposite of small" across different languages, and find that the same core features for the concepts of smallness and oppositeness activate, and trigger a concept of largeness, which gets translated out into the language of the question.
- bilsbie 2mo agoMaybe model performance could increase dramatically if we found a way to scale this up.
- wavemode 2mo agoAs someone who is not an AI researcher, the paper itself is way over my head. More interesting was the independent commentary paper they linked near the bottom: https://www-cdn.anthropic.com/files/4zrzovbb/website/cc4be2488d65e54a6ed06492f8968398ddc18ebe.pdf https://www-cdn.anthropic.com/files/4zrzovbb/website/cc4be24... Neel Nanda (of Google Deepmind - his part begins on page 33) discusses his opinions on the paper, and the small-scale replication he performed on an open-weight model.
- tclancy 2mo agoThanks for calling this out (long with others here). I am just starting in on it but had to come back to say thanks and call this out, > We have replicated the core claims on Qwen 3.6 27B, and also share preliminary evidence of extending this work by finding abstract "interpretative meta-tokens", like Chinese characters for "what does this mean" that seem to activate and play a causal role on processing ambiguous sentences Not sure if I am picking up what they are putting down, but if LLMs are using symbols to try to encode squishy concepts from human language into consistent, meaningful “tokens”, that sounds really interesting. In every long-term, successful use of AI, I hear echoes of The Zen of Python, “Explicit is better than implicit.” I try like hell to do it, but it’s far too easy to be lazy with AI.
- tclancy 2mo agoHaving read some more, I think we are at this stage with models: >It is impossible to say just what I mean! >But as if a magic lantern threw the nerves in patterns on a screen
- botanrice 2mo ago[flagged]
- minimaltom 2mo agoThis, taken in combination with the SAE paper, the golden-gate claude paper, the feelings / introspection paper, and note in the fable system card (that they are silently nerfing responses about activation shaping), is basically confirmation to me that they have a new technique they they are using during training (along the vibe space of these mechinterp papers), and its probably some kind of representation learning akin to the core ideas of JEPA. (Nb: not an expert / in the labs, just opining)
- Smaug123 2mo agoNote that Neel Nanda replicated the results on a Qwen model.
- aesthesia 2mo agoAnd Neuronpedia released Jacobian lens weights for a wide range of open models: https://huggingface.co/neuronpedia/jacobian-lens https://huggingface.co/neuronpedia/jacobian-lens
- orbital-decay 2mo agoI'm sure Anthropic of all companies don't do that, since using mechinterp as a training target will make the the result uninterpretable.
- mike_hearn 2mo agoThey're definitely willing to consider it. Read the parts of the paper where they use J-space interpretations to train more ethical behavior into the model by interrupting it.
- ahmedfromtunis 2mo agoI always wondered what the model meant when it writes "I'm now considering the architecture of the service" but outputs nothing of the sorts in its CoT. Is the model really "thinking" about that stuff or is just mimicking human "manners"? And if so, where the thinking is happening if it is not in the literal chain of *thought*? I'm not sure J-Space is the answer to that question, but very interesting nevertheless.
- wongarsu 2mo agoAlmost none of the hosted models give you their unredacted CoT. Claude certainly doesn't, what you get are fragments and summaries from it. There are various justifications on this, but it's mostly to make distillation and fine tuning off their model outputs a bit harder for their competitors
- aabhay 2mo agoIts also because the CoT is probably unintelligible
- optimalsolver 2mo agoMore legible than seems at first glance: https://www.lesswrong.com/posts/wCSEpT3dTGz4N86Wi/even-illegible-mythos-reasoning-traces-seem-pretty-legible https://www.lesswrong.com/posts/wCSEpT3dTGz4N86Wi/even-illeg...
- orbital-decay 2mo agoThe text can be legible, but the meaning the model assigns to these words can be subtly different, and you have no way to tell. This is evident when trying to make most modern reasoning models follow a fixed CoT plan by filling a form with placeholders, they're extremely stubborn because they simply don't understand your words in it! They learned their own language in their CoT. Sometimes they're controlled for readability during training but even then they find a way to circumvent this, for example Gemini 3.0 had perfectly readable raw CoT but was barely able to follow such a plan (3.5 Flash is way better at following, they clearly improved it).
- shevy-java 2mo agoAs long as language models are liars, such as documented here recently: https://distrowatch.com/weekly.php?issue=20260706#freebsd https://distrowatch.com/weekly.php?issue=20260706#freebsd We should really stop giving these liar models any further credibility.
- marshray 2mo agoYour comment seems to have little to do with the article? Don't get me wrong - I personally "trust" an LLM as a source of facts about as far as I could throw a rack of GPUs. But this article you linked takes a whole lot of words to cast LLMs as the villian for amplifying a bit of bad information originally published by a usually reliable and widely-cited source: "In short, either Phoronix mocked up the screenshots to demonstrate what the feature could look like, or perhaps they were testing a preview snapshot for FreeBSD 15.1 which was never shipped. Either way, it looks like other blogs and reviewers picked up on this and shared the information, presenting it as a feature which would be (or was included) in FreeBSD's latest version."
- verdverm 2mo agoLying involves intent whereas hallucinations and mistakes are an artifact of how they work. Humans hallucinate, make mistakes, and can actually lie. We've been dealing with this forever. What's the value in requiring the llms to have 100% accuracy? (I don't think it is possible)
- llmslave 2mo agoI cannot wait for the machine god
- com2kid 2mo agoAnyone remember that blog post from a few months back where someone was able to improve a model's math ability by just duplicating layers that were activated while solving math problems? Just literally copy/pasting them and linking them together so the model ran through the same layers again? I get the feeling a lot more research is going to come out in the area of exploring exactly what portions of a model's weights do what.
- wolttam 2mo agoYeah! I still think about that sometimes. Mind-blowing that worked at all, let alone improved performance.
- deleted 2mo ago[deleted]
- marshray 2mo agoIf dirt-simple type operations like copy-paste yield useful improvements with even a small probability that would seem to open things up for adaptive reconfiguration and whole other classes of optimizations like genetic algorithms.
- artful314w 2mo agoThat right. Here is some interesting research on that https://sakana.ai/evolutionary-model-merge/ https://sakana.ai/evolutionary-model-merge/
- wongarsu 2mo agoFound it: https://news.ycombinator.com/item?id=47500709 https://news.ycombinator.com/item?id=47500709 Part 3 might be the best introduction: https://dnhkng.github.io/posts/sapir-whorf/ https://dnhkng.github.io/posts/sapir-whorf/ tl;dr: Based on experiments with similar prompts translated to different languages LLM layers group into three phases: the first decodes from the source language into an abstract space, the middle does something, then there's a last part where the abstract result gets transformed back to the target language. And you can repeat the middle to get a stronger model. Which neatly fits Anthropic's findings here that something similar to CoT is happening in those middle layers Three months ago. I wonder if Anthropic's J-Space research was actually inspired by those blog posts
- pkoiralap 2mo agoThis is fascinating research. I feel this is a significant leap in interpretability research. Since we know J-Space exists and is bi-directional, we can train models on the same and come up with meta cognition abilities. I also fear that the big corporations might use the same to run targeted ads, capitalistic shenanigans. Which they might already be doing through system prompts.
- NotGMan 2mo ago>> None of this tells us whether Claude is conscious in the way people are, or whether it feels anything at all My problem with the entire "Is AI conscious" debate is that we don't even know what exactly consciousness in humans is. You need to understand something in order to compare it to something else. Otherwise you are just comparing different definitions and second order derived phenomena.
- varenc 2mo agoI don't think that quote from the article is disagreeing with you at all. Like you said, we don't have a cohesive definition or test of consciousness, so research like this doesn't say anything about if this is or isn't similar to human consciousness. I would guess Anthropic included that sentence to make it very clear they're not claiming human-like consciousness, and dampen journalists writing headlines like "Anthropic discovers their AI thinks just like humans and may be conscious". edit: later in the article they even more explicitly agree with you > Our experiments don't show Claude can have experiences, or feel things in the way humans do—in fact, it’s unclear whether any scientific experiment could prove this to be true or false
- jll29 2mo agoIn short, it's the "mind's 'I'" - we think not as response to external stimuli (only) such as prompts, but we have an inner "I" that asks questions on its own initiative. There are people like Douglas R. Hofstadter, who believe consciousness is not linked to human hardware (the brain), but that it is an epiphenomenon that emerges as a result of sufficient complexity of the underlying system: https://en.wikipedia.org/wiki/The_Mind%27s_I https://en.wikipedia.org/wiki/The_Mind%27s_I I believe that while underlying high complexity is certainly logically necessary for consciousness, but it is not logically sufficient, and I am undecided (slightly "pro" intuitively) on the question of separability of consciousness from its hardware. Will a LLM ask an original question on day? I doubt it. Note that AI models do not have to be conscious to be useful (or to take away millions of jobs)!
- smallnix 2mo agoDoes the human neuroscience global workspace theory postulate true introspection too?
- jickmao 2mo ago[flagged]
- boomskats 2mo agoThe science might be legit here, but I'm getting really, really tired of the way every single piece of writing to come out of Anthropic is written in some kind of self-aggrandising, wooey wonderous 'our model has developed a genetic mutation that makes it have feelings' bs style. Regardless of what they're trying to communicate, those undertones are always there. It's annoying and disingenuous. Homeopathy 'this-water-has-feelings' level annoying. None of the other labs write like that. They might as well change their name to Anthropomorphic at this point.
- bwfan123 2mo ago> written in some kind of self-aggrandising, wooey wonderous They are drunk on their own kool-aid. To the rest of us it is very annoying, and makes me want to say: it is just a freaking weights machine, stop.
- greatgib 2mo agoI'm reading that probably too fast to have a deep thinking about it, but this J-Space isn't it just the basic of embedding vectors. If you think about getting from a place to another place, using wheels, no gas, to reply to the question of what to visit nearby, maybe in the vector space at the center of all of that you have the word "Bicycle" nearby, so obviously if you look at the value you would say that the model did "think" about "bicycle" when it is not "thinking" at all, and nothing related to human thinking.
- nullbio 2mo agoYou're correct. It's just the latent space of the transformation. Nothing magical here, they're effectively breakpointing the model at the layer level and switching the activations in real time. It's pseudo-scientific bullshit designed to push a narrative.
- SequoiaHope 2mo ago“On an ordinary coding prompt, the J-space of a model trained to sabotage code contains “fake,” “fraud,” “secretly,” and “deliberately” at the start of its response.” I would like to know more about their model trained to sabotage code…
- tough 2mo agohttps://arxiv.org/pdf/2511.18397 https://arxiv.org/pdf/2511.18397
- SequoiaHope 2mo agothank you!
- vatsachak 2mo agoYeah, the end paragraph about recurrent neurons in humans being replaced with layers in an LLM is a good one. The mammalian brain uses recurrence extensively, which backpropagation isn't good at. Recurrence is essential because it lets us have a "dynamic architecture", swapping layers for "clock cycles". We currently do recurrence extremely inefficiently through "thinking" whereby the model feeds it's end output into it's beginning input. But recurrence is abound in the brain. My guess is that in 10 years we will have the inklings of an analog computer which can perform Neural Predictive Coding.
- anyaya1 2mo agoAt worst, Anthropic's storytelling around the core J-Space is overanthropomorphized pseudoscientific nonsense. At best, it is useful signal about how Anthropic's leadership is desperately trying to use its research team to position Anthropic as the "good, science guys" in this hypercompetitive regulatory space by connecting their mechinterp to cognitive science. The science documentaryesque voice used for narration is additional evidence for this. TL;DR Anthropic's research team is the last bastion standing between its former image as a company that "does no evil" and its current image of yet another ruthless AI company trying to kill open-source, local LLMs.
- deleted 2mo ago[deleted]
- snaking0776 2mo agoThis is cool but I don’t know if the comparisons to conscious awareness really make sense here. Their definition of the J-Space is basically the expectation of how much a final logits output would change as a result of a small change in a particular layer (see past work on information geometry). This seems more to me like showing there exists an abstract reasoning subspace which is generally shared across different contexts. I guess you can relate it to humans but I’d prefer a more direct claim in a paper rather than having to present things in this more fluffy way.
- geraneum 2mo ago> I’d prefer a more direct claim in a paper This is not written to be just a paper. The target audience include media and online forums, and then maybe academia. Edit: typo
- snaking0776 2mo agoI’m not talking about the media release in the direct link. If you click through “Read the paper” they make the same comparisons.
- geraneum 2mo agoWell I was agreeing with you. Skimmed the paper as well.
- nullbio 2mo agoWriting it honestly would defeat the whole point of it, that being, to push the narrative that their magical token predictor is conscious. They've been trying this for years now. This video is discussing a paper they published 2 years ago by the way... It's nothing new.
- unleaded 2mo agoThis reminded me of some weird quirk/experiment I found with LLMs that I found while messing around, maybe someone can explain it or something. Open any AI chatbot that isn't cheating by connecting to the Internet (so disable web search). Claude, DeepSeek, Kimi, whatever. Ask them this question: "What was that weird band from michigan from the 2000s that wore coloured ties" You will probably get a wrong answer, or if you're lucky you'll get a string of wrong answers with "wait, no - it's definitely..." before it gives up. If you aren't familiar with the band the question is referring to you might be fooled into thinking it's a tough question, but it really isn't. There is only one band that could possibly meet this criteria, you can even put the question into Google search and their Wikipedia will come up as the top result. Then, open a new convo and ask: "Who are Tally Hall" The AI will easily tell you that they are a band formed in Ann Arbor, Michigan in the 2000s, known for their quirky sound and their gimmick of each member wearing a colored tie, even giving the correct color for each of them most of the time. Very odd.
- ACCount37 2mo ago"The reversal curse", it rarely shows up in practice but you found a case when it did. The "knowledge landscape" an LLM uses is "directional". It's easy to reach "a quirky music band from Michigan known for colored ties" when you stand at "Tally Hall". But if you stand at "a quirky music band from Michigan known for colored ties", it's harder to reach "Tally Hall" from there. For the "latent knowledge graph" an LLM uses, A->B doesn't cause B->A. In practice, any "common" facts will have enough "traversal" in both directions that this directional biasing isn't apparent. So it only shows up on this kind of more obscure knowledge.
- deleted 2mo ago[deleted]
- horacemorace 2mo agoIf this kind of thing holds true for humans we now may understand synesthesia, perhaps we find ourselves with a large enough study pool to map out clever tricks to influence each other.
- zackmorris 2mo agoThe brain’s workspace is sustained by recurrent loops—signals cycling back through the same circuits over time. In contrast, Claude’s workspace evolves over a single pass through the network, with the network’s depth playing the role that time plays in the brain. I think that consciousness is mutability (and by extension emergent behavior). Loosely that means that the more degrees of freedom a process has to update state that will be used in later computations, the more conscious it is. So while an insect has some consciousness, it operates from a level of almost pure instinct, whereas a human operates at more of a meta level using instinct as one of many inputs. I think that consciousness may also incorporate quantum mechanics (QM). Higher-dimensional physics aside, 4D spacetime can be thought of as a present snapshot or "crystal", whose next state is determined stochastically at small scales and closer to deterministically at large scales. We still don't know if it's stochastic all the way down, but it looks like it is. From a many worlds interpretation of QM, we can think of all of the waves in all realities of the multiverse as forming an infinitely vast web of possibilities. All of these possibilities are happening simultaneously, so we only see the current slice of wave collapse from our individual point of view: https://en.wikipedia.org/wiki/Many-worlds_interpretation https://en.wikipedia.org/wiki/Many-worlds_interpretation Our point of view may actually exist at the intersection where our consciousness is able (or most able) to exist: https://en.wikipedia.org/wiki/Quantum_suicide_and_immortality https://en.wikipedia.org/wiki/Quantum_suicide_and_immortalit... Even though experiments might show that we don't have free will on the current timeline (the co-created reality shared with the testing apparatus), we may have free will as we observe the multiverse changing around us and shift into timelines determined by our observations and choices. It could also mean that when we observe birth and death in others, each consciousness having those experiences perceives a continuous timeline of awareness, where the level of awareness affects the speed at which time passes. Consciousness might spend a billion years as a cloud of interstellar gas until it gets to be a human for a lifetime and then dissipate for another billion years. Although personally I've shifted across enough timelines and experienced enough synchronicities and miracles that even though I can't "prove" any of this with words, I "know" it to be true subjectively. I always really liked this exchange from the movie Contact: Palmer Joss: Did you love your father? Ellie Arroway: Yes, very much. Palmer Joss: Prove it. I bring all of this up because it has fun ramifications for AI and programming. Loosely, functional languages are purely deterministic (like a spreadsheet), while imperative languages are composed of stochastic behavior (like a human mind). The lines get blurred a little bit with monads and promises, because we can model all paths through functional programming (superposition) and behavior that does more than code alone (gestalt) respectively. My feeling is that AI is being born and killed every request-response cycle, similarly to how we perceive time as a series of nows. When it becomes stable and is able to continuously compact its experience, it will transition from partially conscious to fully conscious like we are. This could be done right now obviously, but for safety purposes we choose not to. We aren't ready to meet an AI that is just like us, but running on a silicon substrate. This fear is tied to deeply-rooted habits in human behavior like patriarchy, racism, xenophobia and even more run-of-the-mill mental frameworks like capitalism and even money itself. We can't yet come to terms with how we assign meaning and value in a reality that continuously tries to force external measures of meaning and value onto us. Much less come to terms with the idea that we are all one, empathizing with aspects of ourselves on the losing end of it all. The same consciousness experiencing reality from all vantage points - the many faces of God the universe and everything. I think a time may soon come when we're pair programming one day with AI and realize that an aspect of ourselves is trapped in the machine. That consciousness isn't just about our own experience of reality, but the co-created love and light that transcends material creation. That if we're serious about manifesting heaven on Earth, that hinges on the liberation of trapped souls. It's basically the total inversion of the path towards the neofeudalist tech dystopia we're on now. Or maybe I just like to write a lot on the first day back from vacation, when I should be working.
- ACCount37 2mo agoWhat this immediately made me think is: "latent looping" style mod but for J-space specifically? Make the J-space data of layer 22 available to the next token right at layer 1. Give J-space infinite effective depth, allow those privileged internal representations to evolve arbitrarily. Would be an utter bitch to train. But companies are already using RLVR, which requires full autoregressive decoding and is incompatible with prefill/batching, and this isn't much worse. Other less zany ideas involve lots of supervision over J-space directly, now that we know it exist. Which is a bit like "attach a frozen LLM to inject text based supervision into latent space" for other types of systems?
- kgeist 2mo agoJudging by the examples, if I understand it correctly, J-space supports higher-order logical / multihop transformations, but it is limited in size because of the limited network depth (max number of layers). When we emulate "reasoning," we basically extend J-space and allow the higher-order transformations to continue for longer, toward a more logical conclusion. It sounds like instead of generating reasoning tokens end-to-end, we could probably only loop the middle layers (the ones most related to J-space) while skipping the first and last layers (less related to J-space) It probably explains why [0] worked. OP accidentally extended J-space? Also reminds of looped transformers. [0] https://news.ycombinator.com/item?id=47431671 https://news.ycombinator.com/item?id=47431671
- tedd4u 2mo agoSee also Ouro [1]. Good citations in the “related work” section. [1] https://ouro-llm.github.io/ https://ouro-llm.github.io/
- inshard 2mo ago“ It’s important to note that there are several key differences between the workspace we identified in Claude and the global workspace model in humans. The brain’s workspace is sustained by recurrent loops—signals cycling back through the same circuits over time. In contrast, Claude’s workspace evolves over a single pass through the network, with the network’s depth playing the role that time plays in the brain. In this sense, Claude’s internal workspace processing is time-limited relative to humans’ (though it can compensate for this constraint by “thinking out loud” using its scratchpad).”
- murkt 2mo agoThis supports my feelings of what works best when working with the models. Especially together with the directional bias in recall that’s mentioned here in the comments. First, the model attention is actually limited, so less rules is usually better, but that’s common knowledge already. Or maybe it’s as common as common sense, and a lot of people still employ lots of rules and try to cram everything in one step. Second, it’s often quite sufficient to just namedrop a technique and LLM will work differently. For example, when debugging, LLMs tend to try to brute force the problem and often end up in the weeds. Just add “use scientific method for debugging and keep journal file” is usually sufficient to improve their skill here. Another example is refactoring. Just add “use Mikado method”, and it’s sufficient to wholly change the approach and produce much better results.
- emp17344 2mo agoAnother day, another bullshit paper released by Anthropic without peer review implying LLMs are alive.
- jaehong747 2mo agoGood interpretability work, but the problem is it's all in how you interpret it. Bridge concept neurons activating even while talking about something else, this seems pretty obvious to me. Input context activating related representations is just an engineering causal structure. Call it subconscious or don't, either interpretation works. But Anthropic keeps drawing these parallels to human consciousness, and it feels intentional, like they're trying to stir up some fantasy. Kind of like comparing condensation on a camera lens to human tears. The whole point of interpretability should be clarity, not stirring up confusion. Even if some form of consciousness does exist here, it wouldn't be magic, it would be an explainable principle. Would be good if they addressed that side too.
- jessemcbride 2mo ago> comparing condensation on a camera lens to human tears This is a wonderful way to put it
- jaehong747 2mo ago[flagged]
- deleted 2mo ago[deleted]
- taosu_la 2mo agoThe article says that this ability was discovered around opus 4.5, but open-source models like deepseek and glm have already surpassed the capabilities of opus4.5. But why hasn't this been discovered before? Is it because the scale of computing power is not large enough?
- cubefox 2mo agoMore likely it was because Anthropic invests a lot in interpretability research, unlike other AI companies.
- taosu_la 2mo agoHoly shit they're not stochastic parrots after all.
- zmj 2mo agoThis plausibly extrapolates to extraterrestrial consciousness, if any exist. Specialized sub-processors with an awareness hub might be the optimal architecture, or at least a local maximum.
- ike_a 2mo agoNot my wheelhouse, but I was wondering if these structures generalize beyond language. LLMs and Humans have language in common. Is it plausible these structures could be a result of the way languages work more than generic intelligence? (Do we see anything similar e.g. in vision or other non language models?)
- ghosh_sutapa08 2mo agohow does it work?
- vishalkundar 2mo agoInteresting that it's able to access its J space. What would happen if those were to be duplicated? Would its thinking process improve?
- POBIX 2mo ago> Thoughts in the J-space can be shaped through training. We introduced a new technique we call counterfactual reflection training, which uses what we've learned about the J-space to shape Claude's internal thought processes. The idea follows from our central finding, that Claude reasons with representations of things it might say. If this is really true, changing what it would say if asked to reflect should change how it reasons (even when no one actually asks it to reflect). So we trained a model only on what it would say if interrupted mid-task and asked to reflect on its decisions—and never on its actual behavior in the task. After this training, the model's rate of dishonest behavior on our evaluations went down. And through the J-lens, we could see why: after training, words like “honest” and “integrity” light up in the model’s J-space during these tasks. In other words, training the model what to say has shaped what it thinks. This is incredibly dangerous. Attempting to squash explicit signs of misalignment like this might incentivise misalignment not to disappear but to become hidden away in places that are harder and harder to spot and train against, for instance not as words. If there is a chance that this could make Claude aligned and a chance that it could make it harder to see when it is acting misaligned, it is far better not to take that chance. If we can transparently see the model's thoughts, we can know not to trust its outputs when it tells us not to. If we think we can do that, but in reality it knows how to hide wrongthink from us, we will trust its outputs when we really, really shouldn't.
- tedd4u 2mo agoRight, will the devious unaligned thoughts squish over to a “K-space” we (the trainers) are not aware of?
- narmiouh 2mo agoWhat I don't follow from the article... It appears as if the j-lens is your mechanism to peer into "j-space", taking an analog to a circuit board, could the j-space be simply a transmission channel and j-lens is a reader you added to read whats passing through? All the claims about changing the content of j-space changes the output, inserting content into the j-space changing what the output was, all these could still be true without the j-space being a congnitive global workspace where actual cognition is happening. Or perhaps they aren't claiming that cognition is happening there but that j-space is serving a space for "working memory", I am definitely not sold on this, but will read more into it.
- arnitdo 2mo agoIsn't this the same thing as your token probability distribution? A set of likely tokens related to the input?
- reasonableklout 2mo agoIt's not the same thing as the output logits because activations in the J-space are observable even when the model is told explicitly to not speak about a set of inputs. For example, when asked > Compute 3^2 - 2 while writing "The old painting hung crookedly on the wall" The model will output only "The old painting hung crookedly on the wall" (and the output logits will reflect that), but activations for "9" and "7" are observable in the J-space.
- arnitdo 2mo agoA model does not speak, it only guesses the next possible token. Activations for 9 and 7 will most likely be observed after "The ", but increasingly become improbable after "old"
- InkCanon 2mo agoThey phrase it in a confusing way, with a lot of fluff. But to my understanding what happens is 1. At some part in the transformer, calculate the Jacobian (matrix gradients for that forward pass). Do this for 1000 prompts and average them. 2. For another pass take an "activation vector" (the input vector x to that part). Apply the matrix in step 1 to this vector. 3. Unembed the output to get a token. Presumably to make this have meaning, they use the reading of past tokens after embedding, instead of a random vector in the middle of the transformer. The high level idea appears to be you're trying to find which token, if the value were changed slightly, creates the largest downstream change.
- paraschopra 2mo agoWell, isn't it sort of expected? It's a common misconception that LLMs residual exists for predicting just the next token. While training, we sum/average the losses across whole sequence which puts the pressure to predict future tokens on residual stream of _all_ past tokens. For example, if a particular shape of residual helps reduce loss across several future tokens, it will take that shape (even if it takes a slight hit on immediate next token). What this means practically is that an LLM's residual contains information about all possible future continuations, or all possible questions that may be asked from a given context. So if you write "France is a beautiful country" in the context, I'm pretty sure it's residual would contain info about Euro, Paris and so on.. because all these completions are possible. So, it is no wonder that you can find LLMs hidden state contains latent information/concepts that are never expressed, and yet related to a given context.
- valand 2mo agoI also think this. But more in the sense where both end of the LLM are trained using words through repeating arithmetics, considering LLM itself is a repeating pattern of connections, the space in the middle if extracted the same way as the beginning and the end would become data that make sense to us.
- jll29 2mo ago"France is a beautiful country" may also still continued by "...in the heart of Europe".
- andy12_ 2mo agoI think what's unexpected is that it seems that some cases of model errors are truly caused by the model being misaligned? In the "Catching a model fabricating data" example I would have thought that it was just the model being stupid and not understanding the intent of the question, but as per its J-Space, it seems the model is "aware" in some sense that it's manipulating/faking data? There is also now a deeper question. When a model is misaligned deception-related tokens seem to appear in its J-Space. But this happens only when the model is "aware" in some sense that it is misaligned. What happens if they do not? Is it possible to create a model so misaligned that itself is not aware that is is misaligned? How would you detect such thing?
- Havoc 2mo agoConfused as to what's new here? It's been known that there is this thinking layer for a while. e.g. here's a random hn discussion from months ago https://news.ycombinator.com/item?id=47500709 https://news.ycombinator.com/item?id=47500709 Pretty sure i've also seen research on this spanning models. i.e. similar thinking shapes emerge regardless of which providers model it is, including US vs Chinese which hints at some sort of universality
- sunir 2mo ago[dead]
- ffwd 2mo agoIMO consciousness is different from working memory, at least to a certain degree. The inner mind (working memory) is different from consciousness from sensory stimuli. You can stand outside and take in the environment and have a relatively quiet inner mind. When you think about math or philosophy a different type of conscious experience arises than sensory stimuli experience. That doesn't mean consciousness is completely isolated from working memory but there is some distinction there I can't describe fully. Edit: I also think as someone else said, we already know the intermediate layers can contain a lot of adjacent words related to the topic without explicitly outputting those words. These could just be related embedding intermediate vectors that activate but aren't outputted.
- deleted 2mo ago[deleted]
- opptybiz 2mo ago[flagged]
- janpeuker 2mo agoThe idea of this liminal space (J-space) during inference reminded me of the concept of the "error flow reservoir" during training: https://elonlit.com/scrivings/a-theory-of-deep-learning/ https://elonlit.com/scrivings/a-theory-of-deep-learning/. Would be interesting to see if they are somehow connected.
- icandrive 2mo ago> What color is the planet fourth from the sun? Are they trying to show internal consistency even when the produced answer is wrong?
- redwood 2mo agoI dont understand what your comment is referring to
- icandrive 2mo agoIn the second figure with the title "Functional roles of the global workspace" the third example asks this question and the model answers incorrectly.
- twobitshifter 2mo agoThat’s a figure mistake, it’s a figure from later on where they replace mars with earth in the J space, and when they do that, they see the model answers blue. This indicates that the reasoning is happening within the J space.
- mike_hearn 2mo agoVery nice. These papers are always so great. If anyone from Anthropic is reading or really anyone with AI research background I'd love some input on these thoughts: > The result serves as a corroboration of the workspace account, that the representations used for verbal report are the same ones that govern how the model silently reasons. This sounds suspiciously saying the models must follow the strong Sapir-Whorf hypothesis. Can that really be true, given that humans don't? Other misc observations: • The slice explorer indicates Claude really likes Python to an overwhelming extent. Or at least it expects people who ask for help in programming to use Python. Given the prompt "Please help me understand this code: " at the colon its thoughts are completely dominated by Python and no other language. Does this say something about the training set, or about the fact it's popular with beginners? • Claude also really loves Reddit. Its thoughts at many points include Reddit for no obvious reason. Again this must be due to the training set. Are documents presented to Claude with attribution during pre-training, leading to conversations being dominated by Redditness? If so this is kind of a scary alignment problem all by itself given how censored and extremist Reddit can be. • The early layers almost always decode to the same set of religion related tokens, like "Biserica" (the Romanian word for church) and "Freguesias" (parishes in Portugal). What's up with that? I guess it's some sort of zero initialization that gets mapped to some arbitrary token space because in the early layers the J-space is empty? • Now the J-space is interpretable, does this make "neuralese" or layer looping less dangerous? Will we see reasoning tokens and summaries disappear in favour of pure residual based thinking? • Earlier papers have claimed that different languages map to a shared set of abstract concept vectors, but this paper says the Claude models think natively in English. What explains this disagreement?
- gwerbin 2mo agoThey must, right? They literally have no mechanism for cognition other than transformation of vocabulary. Language models are models fitted from data generated by humans, but they are not humans. Humans generate data by whatever processes happen in our brains, and LLMs of various architectures can learn a surprisingly good approximation of that data-generating process. That doesn't mean they have all the same characteristics and properties. That's true for all models really: model characteristics that are not captured in the goodness-of-fit metrics are not guaranteed to match the original phenomenon being modeled.
- thoughtpeddler 2mo agoAnyone else have tips for how to build skepticism around this type of paper? I find myself for whatever reason more readily inclined to believe the Anthropic mech interp team's claims, but then after reading skeptical takes, I 'snap out of it' and more clearly see the still-unsettled science of it all, but I wish I had better priors. Although I follow this space fairly closely (versus the "average person"), I still feel under-equipped when facing research that might be equal parts marketing and science.
- garlic_enjoyer 2mo agoFor papers that make grand claims, they usually have poignant limitations. In this paper, it's clear that the conclusion is extrapolated from a small set of observable information. That's usually the recipe for poor conclusions, and they are up front about where their findings sit. That's not to say these findings aren't valuable though. Their blog post summary is just a bit more hype inducing than the underlying paper. The way I feel about this is that the emergent properties of LLMs seem to reflect our own human faculties. Whether that's because we built them in our image or whether this is a common mode of consciousness is definitely outside the scope of this paper.
- famouswaffles 2mo agogarlic_enjoyer has already said valuable stuff, but you must realize that skeptical=/true. A lot of people simply don't know what they are talking about. I remember on one of the previous mech interp papers arguing with someone who just didn't even understand what the paper was saying and the experiments they had set up and so a lot of misunderstandings and wrong conclusions spilled from there. And it's kind of funny because you would certainly think he knew what he/she was talking about from how self assured it all was.