7 ms·
I must be missing something important here. How do the Chinese train these models if they don't have access to the GPUs to train them?
by 0xjmp 11mo ago
I must be missing something important here. How do the Chinese train these models if they don't have access to the GPUs to train them?
- barrell 11mo agoI believe they mean distribution (inference). The Chinese model is currently B.Y.O.GPU. The American model is GPUaaS
- 0xjmp 11mo agoWhy is inference less attainable when it technically requires less GPU processing to run? Kimi has a chat app on their page using K2 so they must have figured out inference to some extent.
- throwaway314155 11mo agotl;dr the person you originally responded too is wrong.
- root_axis 11mo ago> it technically requires less GPU processing to run Not when you have to scale. There's a reason why every LLM SaaS aggressively rate limits and even then still experiences regular outages.
- jychang 11mo agoThat entirely depends on the number of users. Inference is usually less gpu-compute heavy, but much more gpu-vram heavy pound-for-pound compared to training. General rule of thumb is that you need 20x more vram for training a model with X params, than for inference for that same size model. So assuming batch size b, then serving more than 20*b users would tilt vram use on the side of inference. This isn't really accurate; it's an extremely rough rule of thumb and ignores a lot of stuff. But it's important to point out that inference is quickly adding to costs for all AI companies. Deepseek claims that they used $5.6mil to train Deepseek R1; that's about 10-20 trillion tokens at their current pricing- or 1 million users sending just 100 requests at full context size.
- Der_Einzige 11mo agoThat's super wrong. A lot of why people flipped out about Deepseek V3 is because of how cheap and how fast their GPUaaS model is. There is so much misinformation both on HN, and in this very thread about LLMs and GPUs and cloud and it's exhausting trying to call it out all the time - especially when it's happening from folks who are considered "respected" in the field.
- riku_iki 11mo ago> How do the Chinese train these models if they don't have access to the GPUs to train them? they may be taking some western models: llama, chatgpt-oss, gemma, mistral, etc, and do postraining, which required way less resources.
- zackangelo 11mo agoWhat 1T parameter base model have you seen from any of those labs?
- simonw 11mo agoIf they were doing that I expect someone would have found evidence of it. Everything I've seen so far has lead me to believe that these Chinese AI labs are training their own models from scratch.
- riku_iki 11mo agonot sure what kind of evidence it could be..
- simonw 11mo agoJust one example: if you know the training data used for a model you can prompt it in a way that can expose whether or not that training data was used. The NYT used tricks like this as part of their lawsuit against OpenAI: page 30 onwards of https://nytco-assets.nytimes.com/2023/12/NYT_Complaint_Dec2023.pdf https://nytco-assets.nytimes.com/2023/12/NYT_Complaint_Dec20...