7 ms·
> You can get 100 tok/sec on a 3090, and it will feel a lot like the cloud models I have a beefy Linux box with a 4090 but never took the time to set it up pro
by bambax 14d ago
> You can get 100 tok/sec on a 3090, and it will feel a lot like the cloud models
I have a beefy Linux box with a 4090 but never took the time to set it up properly beyond simple testing; any tutorial you would recommend?
- ActorNightly 13d agoJust ollama if you are feeling lazy. With ollama, pull the model (start with https://ollama.com/library/gemma3:27b https://ollama.com/library/gemma3:27b), and `ollama run gemma3:27b` If you want to build agentic frameworks, use llama.cpp with its built in http server, and build the framework with python