6 ms·
I'd like to understand this please. Why would the 1M context be kept in VRAM if you're using DSV4 Pro through the API? Or did you refer to different sessions?
by scope2093 2mo ago
I'd like to understand this please. Why would the 1M context be kept in VRAM if you're using DSV4 Pro through the API? Or did you refer to different sessions?
- throwdbaaway 2mo agoDifferent sessions. With https://github.com/fairydreaming/llama.cpp/tree/dsv4 https://github.com/fairydreaming/llama.cpp/tree/dsv4, 1M context with DSV4 Flash takes less than 6GB of VRAM. I can't run DSV4 Pro, but it should take less than 9GB of VRAM for 1M context, based on the numbers shared in https://arxiv.org/html/2606.19348v1 https://arxiv.org/html/2606.19348v1.
- scope2093 2mo agoThank you for the links/docs. I'm quite excited to try it myself.