5 ms·
I think a reasonable expectation of MAX requirement to claim "runable on consumer hardware" is to 32G VRAM and 128GB RAM and it run at +10tps.
by _ache_ 25d ago
I think a reasonable expectation of MAX requirement to claim "runable on consumer hardware" is to 32G VRAM and 128GB RAM and it run at +10tps.
- latentsea 24d agoI've got an R9700 32GB and an RTX 5060 Ti 16GB plus 64GB of system ram. Hoping to be able to run this at around 30 t/s on a Q4 quant. Hoping. Really hoping. Anything below that isn't usable as a daily driver since at deep context it drops quite significantly, so if you start out at say 20 t/s then you'll wind up at like 10 t/s and 20 t/s is already too slow.
- embedding-shape 24d agoOk, so you already know what your expectations of the requirements are, and you aren't interested in more conservative perspectives, why do you ask to begin with?
- _ache_ 21d agoI will rephrase it. Will it run on any consumer hardware?