6 ms·
here's a quick recording from the 20b model on my 128GB M4 Max MBP: https://asciinema.org/a/AiLDq7qPvgdAR1JuQhvZScMNr https://asciinema.org/a/AiLDq7qPvgdAR1JuQh
by n42 1y ago
here's a quick recording from the 20b model on my 128GB M4 Max MBP: https://asciinema.org/a/AiLDq7qPvgdAR1JuQhvZScMNr https://asciinema.org/a/AiLDq7qPvgdAR1JuQhvZScMNr
and the 120b: https://asciinema.org/a/B0q8tBl7IcgUorZsphQbbZsMM https://asciinema.org/a/B0q8tBl7IcgUorZsphQbbZsMM
I am, um, floored
- Davidzheng 1y agothe active param count is low so it should be fast.
- Rhubarrbb 1y agoGeneration is usually fast, but prompt processing is the main limitation with local agents. I also have a 128 GB M4 Max. How is the prompt processing on long prompts? processing the system prompt for Goose always takes quite a while for me. I haven't been able to download the 120B yet, but I'm looking to switch to either that or the GLM-4.5-Air for my main driver.
- anonymoushn 1y agoit's odd that the result of this processing cannot be cached.
- lostmsu 1y agoIt can be and it is by most good processing frameworks.
- ghc 1y agoHere's a sample of running the 120b model on Ollama with my MBP: ``` total duration: 1m14.16469975s load duration: 56.678959ms prompt eval count: 3921 token(s) prompt eval duration: 10.791402416s prompt eval rate: 363.34 tokens/s eval count: 2479 token(s) eval duration: 1m3.284597459s eval rate: 39.17 tokens/s ```
- andai 1y agoYou mentioned "on local agents". I've noticed this too. How do ChatGPT and the others get around this, and provide instant responses on long conversations?
- bluecoconut 1y agoNot getting around it, just benefiting from parallel compute / huge flops of GPUs. Fundamentally, it's just that prefill compute is itself highly parallel and HBM is just that much faster than LPDDR. Effectively H100s and B100s can chew through the prefill in under a second at ~50k token lengths, so the TTFT (Time to First Token) can feel amazingly fast.
- mike_hearn 1y agoThey cache the intermediate data (KV cache).