4 ms·

this seems like more anthropomorphizing. even if it has similar results often enough to be useful, next token prediction is not searching a solution space. you

• perrygeo 27d ago
  > next token prediction is not searching a solution space. Interesting take. Next token prediction (via the attention mechanism) is a "walk" through the token embedding space. Searching the solution space is what it does, mathematically. It's how we take tokens x context length possible combinations and prune them to converge on viable answers so quickly. Does it look like search at inference time? No. With given weights, a given prompt, and a given random seed, you get the exact same answer. There's not much searching happening at inference... The key is that most of that space is searched at training time. The weights implicitly prune the search space, blocking off or make certain token combinations effectively impossible. It's easy to think "we're just applying weights at inference time" without considering all the pre-work that's done to prune that search space. Which is exactly why "thinking" traces (and randomization) are useful! They bust out of any local optima created by too-tightly-constrained models or system prompts. It's both useful and technically correct to speak of the process as a high-dimension combinatorial search.