5 ms·
Is this any different from using --cache-type-k and --cache-type-v?
by ondra 1y ago
Is this any different from using --cache-type-k and --cache-type-v?
- azinman2 1y agoThat’s what I want to know!
- landl0rd 1y agoI'm guessing it's a bit different since MLX/MPS doesn't have native 4-bit support (or even 8 if I remember correctly?) It didn't launch with bf16 support even. So I think the lowest you could go on the old type_k/v solution and apple GPUs was 16-bit f16/bf16 but not a llama.cpp internals expert so maybe wrong?
- Aurornis 1y agoNo, it appears to be an LLM-generated attempt to gain GitHub stars. See my other comment for a sampling of the other oddities in the repo.