7 ms·

Call me back when you can run these models on 16GB of RAM and any recent i5/i7. Until then, there’s no point on using these toy models.

giancarlostoro 3mo ago
You need it to run in about 8 GB so you have extra space for the context window.
Catloafdev 3mo ago
Hello, it's the internet calling, today is that day. https://github.com/ikawrakow/ik_llama.cpp https://github.com/ikawrakow/ik_llama.cpp Edit: it's gonna be slow if you're not using any VRAM. But it's possible. Software isn't going to speed that up anytime soon, it's just a hardware bandwidth limit.
guax 3mo ago
Its so funny, these "toy models" would be the wet dreams of researchers not 5 years ago. Progress marches without mercy.
- kgeist 3mo ago
  Yeah people don't realize these "toy models" now completely destroy gpt-4o on most tasks, and no one called gpt-4o a toy model back in the day... It was OpenAI's flagship model from 2024 to 2025.
  - Gigachad 3mo ago
    Tbh in 2024 most were calling these models useless for programming and a scam. It wasn't until this year things really changed. My experience with Qwen 3.6 is it can do things, and it's super impressive it can do things, but it's not any more productive than doing it myself.
jboss10 3mo ago
They can be ran on 32GB with 8GB VRAM. I don't think these will be on 16GB for a while. (35B MoE)
- TheCycoONE 3mo ago
  I have 32GB of RAM with 16GB VRAM and I haven't had a lot of luck running larger models like this. Are you able to expand on that?
  - slim 3mo ago
    use llama.cpp with cuda
    - TheCycoONE 3mo ago
      The problem may be that it's a 7800XT which handles memory contention by freezing.
  - jboss10 3mo ago
    I'm running llama-swap in a docker container with nvidia container utis to pass through the GPU. This then runs the correct llama-server command to provide the model I want. I have a folder full of guff s I mount in the container. But this could be done with just llama-server normally. I don't use any special command, just ensure that it's using the GPU. I've found the default fitting to be good. From memory: llama-server -m models/Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf -fa on -c 128000

giancarlostoro 3mo ago

You need it to run in about 8 GB so you have extra space for the context window.

Catloafdev 3mo ago

Hello, it's the internet calling, today is that day. https://github.com/ikawrakow/ik_llama.cpp https://github.com/ikawrakow/ik_llama.cpp Edit: it's gonna be slow if you're not using any VRAM. But it's possible. Software isn't going to speed that up anytime soon, it's just a hardware bandwidth limit.

guax 3mo ago

Its so funny, these "toy models" would be the wet dreams of researchers not 5 years ago. Progress marches without mercy.

kgeist 3mo ago
Yeah people don't realize these "toy models" now completely destroy gpt-4o on most tasks, and no one called gpt-4o a toy model back in the day... It was OpenAI's flagship model from 2024 to 2025.
- Gigachad 3mo ago
  Tbh in 2024 most were calling these models useless for programming and a scam. It wasn't until this year things really changed. My experience with Qwen 3.6 is it can do things, and it's super impressive it can do things, but it's not any more productive than doing it myself.

jboss10 3mo ago

They can be ran on 32GB with 8GB VRAM. I don't think these will be on 16GB for a while. (35B MoE)

TheCycoONE 3mo ago
I have 32GB of RAM with 16GB VRAM and I haven't had a lot of luck running larger models like this. Are you able to expand on that?
- slim 3mo ago
  use llama.cpp with cuda
  - TheCycoONE 3mo ago
    The problem may be that it's a 7800XT which handles memory contention by freezing.
- jboss10 3mo ago
  I'm running llama-swap in a docker container with nvidia container utis to pass through the GPU. This then runs the correct llama-server command to provide the model I want. I have a folder full of guff s I mount in the container. But this could be done with just llama-server normally. I don't use any special command, just ensure that it's using the GPU. I've found the default fitting to be good. From memory: llama-server -m models/Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf -fa on -c 128000