5 ms·
I've been watching and waiting for this, interested to see how smart it is, as it fits with my interest of getting the smartest possible model running in 10GB o
by thomasjb 2mo ago
I've been watching and waiting for this, interested to see how smart it is, as it fits with my interest of getting the smartest possible model running in 10GB of VRAM (RTX3060 that has to drive 2 monitors and run an llm)
- dakolli 2mo agostart saving your money.
- thomasjb 2mo agoOr watch and wait as models get denser
- kennywinker 2mo agoToss the rtx into a cheapo optiplex or thinkcenter, and run it headless - the load on your machine is gonna make doing other stuff while it’s running painful. Plus that frees up the rest of your vram.
- xyzsparetimexyz 2mo agoWhy would they do that when they already have a perfectly good PC?
- kennywinker 2mo ago…? I said why. > the load on your machine is gonna make doing other stuff while it’s running painful Is your question about something else?
- thomasjb 2mo agoI aspire to someday move up to an AM5 system, but for now, the Dell T3610 has to do everything. Might get a second GPU for it though.
- HalfCrimp 2mo agoRunning an llm on a PC running a desktop environment really isn't that bad. You lose a bit of ram & vram but that only matters if you _reaaaally_ want to push to the max model size your hardware can handle. The biggest issue I've found is absent mindedly opening YouTube or the like that spike ram requirements and freezing the system up. But that's a me problem