Or, theres the option of a 32gb v100 (about $600USD on taobao etc), where you can get 1200 of prefill at 80 of decode: https://github.com/geoffwatts/ninfer-v100

Or, theres the option of a 32gb v100 (about $600USD on taobao etc), where you can get 1200 of prefill at 80 of decode: https://github.com/geoffwatts/ninfer-v100 https://github.com/geoffwatts/ninfer-v100 - that's _really_ cheap inference, and it's not a modified card - you just need to add a blower or water block.