7 ms·

What did you set the context window to? That's been my main issue with models on my macbook, you have to set the context window so short that they are way less

What did you set the context window to? That's been my main issue with models on my macbook, you have to set the context window so short that they are way less useful than the hosted models. Is there something I'm misisng there?

hrpnk 1y ago
With LM Studio you can configure context window freely. Max is 131072 for gpt-oss-20b.
- coltonv 1y ago
  Yes but if I set it above ~16K on my 32gb laptop it just OOMs. Am I doing something wrong?
  - mekpro 1y ago
    try enable flash attention and offload all layer to GPU
simonw 1y ago
I punted it up to the maximum in LM Studio - seems to use about 16GB of RAM then, but I've not tried a long prompt yet.