6 ms·
llama.cpp has router mode these days, swapping out models for you, you no longer need llama-swap.
by NorwegianDude 1mo ago
llama.cpp has router mode these days, swapping out models for you, you no longer need llama-swap.
- dangoljames 1mo agoI use it on the daily. Haven't exactly missed ollama.
- cptskippy 1mo agoThe router mode and matrix routing in llama.cpp is still early days and it can't easily juggle multiple models as easily as llama-swap so there's still benefits if you're using 24-32GB cards that can run multiple models simultaneously. The gulf is quickly shrinking though and it seems like the need for llama-swap will disappear soon.