6 ms·

One line of evolution seems to be toward some form of latent-space reasoning, as in [1]. Natural language seems like a relatively low-bandwidth channel for inte

• niek_pas 1mo ago
  What does ‘latent’ mean in this context?
  • entrope 1mo ago
    "Latent" means it (the space) is there all the time but usually not exposed or apparent. Latent space is a high-dimensional space that is not legible to humans, so normally reasoning gets constrained to progress in tokens instead so that it can be evaluated. I think https://arxiv.org/abs/2412.06769 https://arxiv.org/abs/2412.06769 is a good description of the premise of reasoning in latent space, although https://arxiv.org/abs/2604.15726 https://arxiv.org/abs/2604.15726 argues it's already what really happens.
    • vardhanw 1mo ago
      So, does "reasoning in latent space" mean that it does/will not use tokens (e.g. using CoT etc.)?
      • Sharlin 1mo ago
        The computation still has to be done, even if it's more efficient in a higher-entropy space.
      • zarzavat 1mo ago
        Models already reason in latent space, but they have to keep encoding-decoding their "thoughts" from/to human interpretable tokens, and notably those tokens are sampled from a distribution. The model cannot output a vector and have that same vector fed back in at the next step, it only sees what token the sampler collapsed its vector into. It's as if the only way you could think was by writing down a word, erasing all the thoughts from your head, then reading the word you just wrote down and deciding on the next word, etc. Reasoning purely in latent space means that the model would still produce an output equivalent to tokens but unconstrained e.g. the output could be raw and opaque vectors. A significant downside is that you lose the ability to inspect the reasoning trace. It would also make the reasoning trace potentially larger which has operational issues.
        • jacquesm 1mo ago
          More like erasing all other branches just prior to that word from your head, the past thoughts would still remain.
        • 0-_-0 1mo ago
          The kv cache is not discretised, only the output tokens.
        • wren6991 1mo ago
          > The model cannot output a vector and have that same vector fed back in at the next step, it only sees what token the sampler collapsed its vector into. Not completely true: KV is a projection of the activation at each layer's input, so attention heads see (a representation of) all previous tokens' activations at that layer. The hard decision at the LM head doesn't change that.
• 0-_-0 1mo ago
  Reasoning is already taking place in latent space, the CoT is just a sideeffect
• lta 1mo ago
  While I was reading the top level message I was thinking on something wrong those lines. That being said I think it's somewhat useful to be able to see the reasoning in a meaningful format. There's already enough hidden layers of unfathomable meaning. What if instead of a latent they used at a shorter form of note-taking-like reasoning, using more symbol to achieve a denser CoT. We'd get the best of both worlds. WDYT ?