5 ms·

Tried it on Android and got "!!!!!!!!!!!!!" for answers.

• verdverm 2mo ago
  That's what happens when you quant too hard. I'm working on quant strats and evals for the same underlying qwen 27b models. When I saw 27b on a phone, I thought not fitting, big phone, or aggressive quant. NVFP4 still takes 27G before KV cache.
• gunalx 2mo ago
  The qwen models really seem to have this as a failure mode, its so annoying having a proper trace ending up in !!!!!! Garbage.
  • amelius 2mo ago
    Wait in a regular sentence, what is the probability of "!!!" being followed by "!"? Sounds like the model is not following a proper probabilistic choice here, so maybe more a programming error than a model training error.
    • jldugger 2mo ago
      After the third !, the probability of a fourth probably skyrockets =)
  • aesthesia 2mo ago
    Oh, interesting. "!" is token id 0 in the Qwen tokenizer; I wonder if there's some tokenizer shenanigans either in inference or training that end up causing this specific behavior.