if you take a model that requires 200GB of VRAM and you run it on the CPU, it requires 200GB of RAM instead. Still unfeasible on consumer hardware. With this ap

if you take a model that requires 200GB of VRAM and you run it on the CPU, it requires 200GB of RAM instead. Still unfeasible on consumer hardware. With this approach you can easily do it on 12GB or less of either RAM of VRAM, at several seconds per token instead of tokens per seconds. Very unusable, but certainly interesting!