5 ms·
IQ4 qwen 122b-a10b would mean 61GB total size and 5GB active, so about 5GB of the model loaded into GPURAM plus any generated context, and 61GB of weights loade
by TonyStr 2mo ago
IQ4 qwen 122b-a10b would mean 61GB total size and 5GB active, so about 5GB of the model loaded into GPURAM plus any generated context, and 61GB of weights loaded into system RAM? I don't know if that math is correct, but does that run well? Wouldn't that only leave 3GB of system RAM?