5 ms·
What will be the requirement, like 128G of RAM and 12G of VRAM ?
by _ache_ 23d ago
What will be the requirement, like 128G of RAM and 12G of VRAM ?
- embedding-shape 23d agoHow long is a rope? Technically you could probably run it off a SSD, but it'll be slow as molasses. If you want it "fast", you want it all within GPU and VRAM, who knows what that'd be. If the engram parameters are separate, I guess it'd be like BF16 ~400 GB, FP8 ~200 GB, NVFP4 ~100GB. Otherwise maybe like ~300GB, ~150GB and ~70GB or alike, don't quote me that, only some guesses. The one who waits will see :)
- _ache_ 23d agoI think a reasonable expectation of MAX requirement to claim "runable on consumer hardware" is to 32G VRAM and 128GB RAM and it run at +10tps.
- latentsea 23d agoI've got an R9700 32GB and an RTX 5060 Ti 16GB plus 64GB of system ram. Hoping to be able to run this at around 30 t/s on a Q4 quant. Hoping. Really hoping. Anything below that isn't usable as a daily driver since at deep context it drops quite significantly, so if you start out at say 20 t/s then you'll wind up at like 10 t/s and 20 t/s is already too slow.
- embedding-shape 23d agoOk, so you already know what your expectations of the requirements are, and you aren't interested in more conservative perspectives, why do you ask to begin with?
- _ache_ 19d agoI will rephrase it. Will it run on any consumer hardware?