5 ms·

Yeah, local LLMs are probably an order of magnitude less efficient at a fixed level of "intelligence" if not more.

• tjwebbnorfolk 1mo ago
  Inefficient per watt, yes, but local inference capacity is greatly underutilized in aggregate. If a model can run on a machine that already exists, that's a bunch of additional chips that don't need to be built.
  • rickydroll 1mo ago
    You picked up on what I didn't write existing machines don't have to be built, you just use them. It occurred to me much later that local machines could become a shared resource of a small co-op. Or it only runs off solar or energy harvested at times of excess generation (too much daytime solar, lots of wind, or overnight when demand naturally drops). Exploring how to partition inference across many machines and shed load for idle sections of hardware would be a problem that could be solved by a community of smart people like us here
    • tjwebbnorfolk 1mo ago
      yea right now local models are like the swing producer. When claude throws 503s or 429s at me, or I run out of quota, I fire up a local model. It's also resilience. I can still do AI stuff if the internet goes down.