6 ms·
> I’m struggling to understand how that was causing such a large difference in the output. It is incremental, the more a pattern appears in the context, the m
by tarruda 24d ago
> I’m struggling to understand how that was causing such a large difference in the output.
It is incremental, the more a pattern appears in the context, the more likely it was to continue appearing in future turns.
So the model was likely trained to end a reasoning trace with a single linefeed and a `</think>`. It was also likely trained that two consecutive linefeeds sometimes produce an "Actually..." sequence.
So you can think of it as:
- the first time the reasoning trace was parsed, the two trailing linefeeds followed by </think> were added to the context by the template.
- next time the model was finishing a reasoning block, it added an extra linefeed instead of just closing directly with </think>. This slightly increases the chance that the next token will begin an "actually" sequence instead of closing.
- If it caused an actually, that was added to the context, further increasing the chance of a self correction at the end of the thinking block. The more self correction paragraphs are added, the higher the chance that following turns will have more.
- Eventually it can result in a state where it enters that loop forever (or at least for a very long time).
> Is the “autoparser” vulnerable to injections somehow?
The autoparser was (and still is) incorrectly parsing a trailing linefeed as part of a reasoning block. The were two ways to fix this, both of which must be implemented for the fix to be complete IMO:
- fix the autoparser definition to ensure remove surrounding whitespace is not returned as part of the text blocks
- trim leading/trailing whitespace in the encoding phase, so it fixes bugs or even "injections" where the client deliberately adds the whitespace to trigger problems.
For this specific issue, the maintainer later fixed by trimming the extra linefeed before passing to the template.
> How do you distinguish between user text, model text, and metadata, or is there ambiguity in the parsing?
That is model specific. Ultimately, a token stream is being produced and parsed by the inference engine, and each model uses different tokens/formats. The goal of the autoparser engine was to simplify the creation of parsers for new models by inferring the delimiter tokens from the chat template.
The llama.cpp API server returns pre-parsed data, so clients don't need to do any parsing to know what is a thinking block, a text block or a tool call.