48 ms·
Project is de facto dead, used it for many years and had to rip it out 6 months ago, don't waste your time.
by refulgentis 14d ago
Project is de facto dead, used it for many years and had to rip it out 6 months ago, don't waste your time.
- skybrian 14d agoWhat did you switch to?
- seamossfet 14d agoWe use the ONNX runtime for small models in the browser https://github.com/microsoft/onnxruntime https://github.com/microsoft/onnxruntime
- theanonymousone 14d agoTransformers.js supports a lot of models since v4
- refulgentis 14d agoThere's, quietly, a llama.cpp WebGPU backend that works *great*. Some hacking required, it's unsupported, a side project for one of the lead maintainers and someone in school. Note of caution, llama.cpp isn't what it was, the grunt-level maintainers are left to their own devices. There's one key subsystem where things break regularly and the engineering is poor, and the "lead maintainer" is aggro and isn't really involved after delivering their big refactor that was DOA, other than telling people on issues he's never seen their repro himself. Carefully pick models you can support down to "I can patch around the Jinja template engine". Go through the same llama web server APIs. c.f. github / telosnex / fllama if you need a reference (n.b. seeing peer comments, its much better than ONNX, ONNX never got within spitting distance of llama.cpp, my understanding from watching the LLM runtime is its used for Windows AI features so their models probably work great on Windows x Qualcomm. ONNX is great for smol models though, like VAD, a god send even.)
- roflcopter69 13d agoI seem to be out of the loop when it comes to llama.cpp? What happened? Can you please provide some more pointers to sources so one can verify your claims? Do you see alternatives to llama.cpp in whose future you have more trust?
- woadwarrior01 13d agoYeah, both WebLLM and mlc-llm have been mostly abandoned since NVIDIA acquired OctoAI in late 2024.