6 ms·

So-called "caveman" thinking attempts to address this. The important part of "actually wait, I really need to XYZ" is just "XYZ". The model can attend to just

So-called "caveman" thinking attempts to address this. The important part of "actually wait, I really need to XYZ" is just "XYZ". The model can attend to just "do XYZ" and produce almost the same vector modifications as full verbose "reasoning".

vanviegen 1mo ago
I don't think that's true. If a context contains a statement followed by something opposing that statement, that will confuse the model. So "actually wait, I really need to" is there to signal that the previous thinking may be flawed and that what follows is a new attempt. It's good to remember that LLMs have no more state then what they can derive from the context up til any point. So if that context is hard to interpret, that will reduce effectiveness.
- lxgr 1mo ago
  As long as the model itself understands that when it says "foo bar" in its own reasoning trace it means "pursue theory foo; no, this didn't lead anywhere, let's backtrack and pursue theory bar", where's the loss in clarity?
  - vanviegen 1mo ago
    How could it understand, if there's no distinction between that and "from foo logically follows bar"? If the notes it's taking for itself don't make sense, it might just as well not take them.
    - lxgr 1mo ago
      Obviously it's going to have to come up with unambiguous shorthand for itself during gradient descent. My point is that that shorthand doesn't necessarily have to be unambiguous to human observers.
    - suprjami 26d ago
      I feel you have both missed the point of caveman reasoning. LLMs are trained on human natural language, not a specialised internal-only monologue to make syntactic shortcuts. Their response should make grammatical sense to a human reader because they are mimicking human speech. "foo bar" is ambiguous. "pursue theory foo; no, this didn't lead anywhere, let's backtrack and pursue theory bar" is accurate, should be meaningful to LLM attention, but is too verbose. The minimal caveman way to say this is "not foo. instead do bar". This really should provide the LLM attention with everything it needs to grasp the intention, but with far fewer tokens used. Instead of all this I could have just said: "nothing ambiguous. clear but less words"

vanviegen 1mo ago

I don't think that's true. If a context contains a statement followed by something opposing that statement, that will confuse the model. So "actually wait, I really need to" is there to signal that the previous thinking may be flawed and that what follows is a new attempt. It's good to remember that LLMs have no more state then what they can derive from the context up til any point. So if that context is hard to interpret, that will reduce effectiveness.

lxgr 1mo ago
As long as the model itself understands that when it says "foo bar" in its own reasoning trace it means "pursue theory foo; no, this didn't lead anywhere, let's backtrack and pursue theory bar", where's the loss in clarity?
- vanviegen 1mo ago
  How could it understand, if there's no distinction between that and "from foo logically follows bar"? If the notes it's taking for itself don't make sense, it might just as well not take them.
  - lxgr 1mo ago
    Obviously it's going to have to come up with unambiguous shorthand for itself during gradient descent. My point is that that shorthand doesn't necessarily have to be unambiguous to human observers.
  - suprjami 26d ago
    I feel you have both missed the point of caveman reasoning. LLMs are trained on human natural language, not a specialised internal-only monologue to make syntactic shortcuts. Their response should make grammatical sense to a human reader because they are mimicking human speech. "foo bar" is ambiguous. "pursue theory foo; no, this didn't lead anywhere, let's backtrack and pursue theory bar" is accurate, should be meaningful to LLM attention, but is too verbose. The minimal caveman way to say this is "not foo. instead do bar". This really should provide the LLM attention with everything it needs to grasp the intention, but with far fewer tokens used. Instead of all this I could have just said: "nothing ambiguous. clear but less words"