6 ms·
And you’ve opened wireshark and verified the model is sending absolutely nothing? Not caching and sending later, etc?
by rubicon33 3mo ago
And you’ve opened wireshark and verified the model is sending absolutely nothing? Not caching and sending later, etc?
- razemio 3mo agoNot sure if you mean something else, but the model itself is not able to send anything.
- adrian_b 3mo agoIf you self host then you can audit the open-source llama.cpp or whichever other program you are using for inference, to see exactly what it does, and also whichever open-source harness you use for implementing a coding assistant or other agentic workflow. The model consists of a bunch of data files, it does absolutely nothing by itself. If you run inference on your own hardware, you have absolute control on how the LLM is used, not like when you use an external service provider.