6 ms·
> Both the NVFP4 and AWQ W4A16 failed to properly close their tool calls ... If I understand correctly, this failure mode is just not possible with llama.cpp /
by throwdbaaway 25d ago
> Both the NVFP4 and AWQ W4A16 failed to properly close their tool calls ...
If I understand correctly, this failure mode is just not possible with llama.cpp / ik_llama.cpp, which enforces token generation to follow the grammar once a tool call is detected.
> ... and botched Cisco command line syntax (the correct command was ‘show arp’, while they executed ‘show run’)
But this failure mode can still happen.
Anyway, NVFP4 and AWQ W4A16 are generally regarded as low quality quants. IQK/Trellis quants from ik_llama.cpp and EXL3 quants from exllama should work better.
So, perhaps the lesson here is "don't use vllm at home"?
- throwdbaaway 25d agoAs for KV cache quantization, Q8_0 from llama.cpp / ik_llama.cpp should also work better than FP8 from vllm (see https://github.com/vllm-project/vllm/issues/33480#issuecomment-4201272360 https://github.com/vllm-project/vllm/issues/33480#issuecomme...).