7 ms·

The reason to run local models is not for coding mostly it's for learning how to deploy models and tinker with self hosting. It's also for massively crunching d

The reason to run local models is not for coding mostly it's for learning how to deploy models and tinker with self hosting. It's also for massively crunching data 24/7. Imaging having an agent analyzing constinous log streams etc .. that could be a usescse where even deepseek could add up cost.

epolanski 1mo ago
Also privacy and offline capabilities.
- DanielHB 1mo ago
  One thing I realized is just how much offline local models can hurt mass data collection. For example, I needed to write an invitation letter for immigration control for a relative visiting me. Previously I would have used a search engine for a template. Today I fire up my local qwen 3.5-9b for this kind of stuff and feed it all the private data I need. Unfortunately it is unlikely the average user will known how to avoid this data collection. Even if the LLM is local you are likely feeding the prompts to remote servers if you harness/chat-interface is not properly vetted.
  - epihelix 1mo ago
    I hope for local model chat inexperienced users are just using llama.cpp's built-in web server interface, which gives you everything you need. No need for a harness or any other chat client.
    - DanielHB 1mo ago
      I have tried running llamma.cpp on my PC and I found it hard getting it to run at decent speed. On Qwen 3.5-9b I get at most 10tk/s. I eventually switched to LM studio and the same model runs much better, like 70tk/s. Not sure if it was because I was running llama.cpp inside podman or badly tuned LLM arguments. But LM studio is unfortunately much more practical. Although I agree with you. I do not really know what kind of telemetry LM studio is running and I would rather not be using it.
testerius 1mo ago
Privacy, no need to pay for tokens, offline usage, endless possibilities - you do not have to pay for tokens (subscribtion payment), so you can do more. The big issue is that local models are not "ready yet" compared to frontier and paid services. It is hard to run decent model without decent hardware. And to be honest even if you can buy hq hardware and spend a lot of money then it is not the same quality.
cyanydeez 1mo ago
For me its stability and workflow. Models are all non deterministic and local is the only gauntee that your investment can continue to pay. Cloud models will continually change nondeterminism ontop of the model.

epolanski 1mo ago

Also privacy and offline capabilities.

DanielHB 1mo ago
One thing I realized is just how much offline local models can hurt mass data collection. For example, I needed to write an invitation letter for immigration control for a relative visiting me. Previously I would have used a search engine for a template. Today I fire up my local qwen 3.5-9b for this kind of stuff and feed it all the private data I need. Unfortunately it is unlikely the average user will known how to avoid this data collection. Even if the LLM is local you are likely feeding the prompts to remote servers if you harness/chat-interface is not properly vetted.
- epihelix 1mo ago
  I hope for local model chat inexperienced users are just using llama.cpp's built-in web server interface, which gives you everything you need. No need for a harness or any other chat client.
  - DanielHB 1mo ago
    I have tried running llamma.cpp on my PC and I found it hard getting it to run at decent speed. On Qwen 3.5-9b I get at most 10tk/s. I eventually switched to LM studio and the same model runs much better, like 70tk/s. Not sure if it was because I was running llama.cpp inside podman or badly tuned LLM arguments. But LM studio is unfortunately much more practical. Although I agree with you. I do not really know what kind of telemetry LM studio is running and I would rather not be using it.

testerius 1mo ago

Privacy, no need to pay for tokens, offline usage, endless possibilities - you do not have to pay for tokens (subscribtion payment), so you can do more. The big issue is that local models are not "ready yet" compared to frontier and paid services. It is hard to run decent model without decent hardware. And to be honest even if you can buy hq hardware and spend a lot of money then it is not the same quality.

cyanydeez 1mo ago

For me its stability and workflow. Models are all non deterministic and local is the only gauntee that your investment can continue to pay. Cloud models will continually change nondeterminism ontop of the model.