5 ms·
Unsloth quantizations are available on release as well. [0] The IQ4_XS is a massive 361 GB with the 754B parameters. This is definitely a model your average loc
by Yukonv 5mo ago
Unsloth quantizations are available on release as well. [0] The IQ4_XS is a massive 361 GB with the 754B parameters. This is definitely a model your average local LLM enthusiast is not going to be able to run even with high end hardware.
[0] https://huggingface.co/unsloth/GLM-5.1-GGUF https://huggingface.co/unsloth/GLM-5.1-GGUF
- zozbot234 5mo agoSSD offload is always a possibility with good software support. Of course you might easily object that the model would not be "running" then, more like crawling. Still you'd be able to execute it locally and get it to respond after some time. Meanwhile we're even seeing emerging 'engram' and 'inner-layer embedding parameters' techniques where the possibility of SSD offload is planned for in advance when developing the architecture.
- adrian_b 5mo agoFor conversational purposes that may be too slow, but as a coding assistant this should work, especially if many tasks are batched, so that they may progress simultaneously through a single pass over the SSD data.
- QuantumNomad_ 5mo agoThree hour coffee break while the LLM prepares scaffolding for the project.
- tempoponet 5mo agoRather, Imagine you have 2-3 of these working 24/7 on top of what you're doing today. What does your backlog look like a month from now?
- cyanydeez 5mo ago[flagged]
- dcreater 5mo ago@dang
- pbhjpbhj 5mo agoLike computing used to be. When I first compiled a Linux kernel it ran overnight on a Pentium-S. I had little idea what I was doing, probably compiled all the modules by mistake.
- drowsspa 5mo agoAt least the compiler was free
- adrian_b 5mo agoThe point of doing local inference with huge models stored on an SSD is to do it free, even if slow.
- fireant 5mo agoYou are just trading opex for capex. Local GPUs aren't free.
- adrian_b 5mo agoTrue, but this is not only a trade-off between opex and capex. Local inference using open weight models provides guaranteed performance which will remain stable over time, and be available at any moment. As many current HN threads show, depending on external AI inference providers is extremely risky, as their performance can be degraded unpredictably at any time or their prices can be raised at any time, equally unpredictably. Being dependent on a subscription for your programming workflow is a huge bet, that you will gain more from a slightly higher quality of the proprietary models than you will lose if the service will be degraded in the future. As the recent history has shown, many have already lost this bet. I am not a gambler, so I have made my choice, which is local AI inference, using a variety of models depending on the task, i.e. both small models completely executable on relatively cheap GPUs (like the new Intel GPUs), medium models that need e.g. 128 GB on a CPU, and huge models that must be stored on fast SSDs (e.g. interleaved on multiple PCIe 5.0 SSDs). Such a strategy is achievable with a modest capex, in the lower half of the 4-digit range.
- zozbot234 5mo agoBatching many disparate tasks together is good for compute efficiency, but makes it harder to keep the full KV-cache for each in RAM. You could handle this in an emergency by dumping some of that KV-cache to storage (this is how prompt caching works too, AIUI) and offloading loads for that too, but that adds a lot more overhead compared to just offloading sparsely-used experts, since KV-cache is far more heavily accessed.