5 ms·

how is this open vs whatdeepseek did?

simonw 2y ago
From that article: > The release of DeepSeek-R1 is an amazing boon for the community, but they didn’t release everything—although the model weights are open, the datasets and code used to train the model are not. > The goal of Open-R1 is to build these last missing pieces so that the whole research and industry community can build similar or better models using these recipes and datasets.
- boznz 2y ago
  Genuine question, but how do you replicate the effort exactly without $5M in compute? and can you test that the published weights etc are actually those in the model? Am I missing something?
  - simonw 2y ago
    The $5.5m in compute wasn't for R1, it was for DeepSeek v3. The R1 trick looks like it may be a whole lot cheaper than that. R1 apparently used just 800,000 samples - I don't fully understand the processing needed on top of those samples but I get the impression it's a whole lot less compute than the $5.5m used to train v3.
choilive 2y ago
deepseek claims they are "open source" but they are not. They are open weight. IMO a truly "open" AI model should have 3 components publicly available: the weights, the code, and the dataset. Without all 3 the model is not reproducible. Could make the argument that the code and data are sufficient though.
- brianstrimp 2y ago
  Only meaningful if code+data deterministically reproduce the weights. At that point, the weights are just the cached output. Which has value since it's costly to produce from code+data.
  - frabcus 2y ago
    I don't think it needs to be deterministic - and if it isn't, having the data and code becomes even more important! Compilers generally aren't deterministic (see the reproduceable build movement), yet we still use their output binaries.
    - brianstrimp 2y ago
      [dead]
- blackeyeblitzar 2y ago
  The only one I’ve see people talk about that shares all the components is OLMo (https://allenai.org/blog/olmo2 https://allenai.org/blog/olmo2)
  - reedciccio 2y ago
    There are more, like the work by Eleuther AI and LLM360.
- mirsadm 2y ago
  No one will release the dataset because we all know it is gathered through dodgy means.
  - notyourwork 2y ago
    And likely massaged by the CCP.
    - chgs 2y ago
      Deep mind has one set of censorship, OpenAI another, anything musk does a third It’s all “massaged”
      - redcobra762 2y ago
        I don’t think “Taiwan is China” is the same kind of massaging as not telling people how to make napalm… What a weird thing to equate, though!
        foxglacier 2y ago
        It is though. Western AI tries to hide information like that with the justification of safety as well as things that might be offensive to current popular beliefs. Chinese AI presumably says Taiwan is China to help get more people on side for a possible future invasion. Propaganda does work - look at how many people think Donbas is still Ukraine and Israel is still Palestine.
        redcobra762 2y ago
        The difference is that in China the info isn’t available without use of Western content, due to the totalitarian control over media, whereas in the West, information is pretty trivially available, even if the big companies keep it off of their platforms. And sure ignorance is prevalent, but even GPT4 will tell me Donbas is still Ukraine, for instance. What a strange example to use, though!
- reedciccio 2y ago
  That's what the Open Source AI Definition states https://opensource.org/ai https://opensource.org/ai In any case, Deepseek like Llama fail much before hitting that new definition. Both have licenses containing restrictions on field of use and discrimination of users. Their license will never be approved as Open Source.
- mythz 2y ago
  This nitpicking is pointless. DeepSeek's gifts to the world of its open weights, public research and OSS code of its SOTA models are all any reasonable person should expect given no organization is going to release their dataset and open themselves up to criticism and legal exposure. You shouldn't expect to any to see datasets behind any SOTA models until they're able to be synthetically generated from larger models. Models only trained on sanctioned "public" datasets are not going to perform as well which makes them a lot less interesting and practically useful. Yes it would be great for their to be open models containing original datasets and a working pipeline to recreate models from scratch. But when few people would even have the resources to train the models and the huge training costs just result in worse performing models, it's only academically interesting to a few research labs. Open model releases should be celebrated, not criticized with unreasonable nitpicking and expectations that serves no useful purpose other than discouraging future open releases. When the norm is for Open Models to include their datasets, we can start criticizing those that don't, but until then be gracious that they're contributing anything at all.
  - redcobra762 2y ago
    “Gifts to the world”? What a strange thing to say…
  - jimjimwii 2y ago
    Terminology exists for a reason. Doubly so for well-established terms of art that pertain to licensing and contract law. They could have used "open wights" which would have conveyed the company's desired intent just as well as "open source", but without the ambiguity. They deliberately chose to misuse a well established term instead. I applaud and thank deepseek for opening their weights, but i absolutely condemn them and others (e.g Facebook) for their deliberate and continued misuse of the term. I and others like me will continue to raise this point as long as we are active in this field, so expect to see this criticism for decades. Hopefully one of these companies losses a lawsuit due to these shenanigans. Perhaps then they wouldn't misuse these terms so brazenly.
    - mythz 2y ago

From that article: > The release of DeepSeek-R1 is an amazing boon for the community, but they didn’t release everything—although the model weights are open, the datasets and code used to train the model are not. > The goal of Open-R1 is to build these last missing pieces so that the whole research and industry community can build similar or better models using these recipes and datasets.

boznz 2y ago
Genuine question, but how do you replicate the effort exactly without $5M in compute? and can you test that the published weights etc are actually those in the model? Am I missing something?
- simonw 2y ago
  The $5.5m in compute wasn't for R1, it was for DeepSeek v3. The R1 trick looks like it may be a whole lot cheaper than that. R1 apparently used just 800,000 samples - I don't fully understand the processing needed on top of those samples but I get the impression it's a whole lot less compute than the $5.5m used to train v3.

choilive 2y ago

deepseek claims they are "open source" but they are not. They are open weight. IMO a truly "open" AI model should have 3 components publicly available: the weights, the code, and the dataset. Without all 3 the model is not reproducible. Could make the argument that the code and data are sufficient though.

brianstrimp 2y ago
Only meaningful if code+data deterministically reproduce the weights. At that point, the weights are just the cached output. Which has value since it's costly to produce from code+data.
- frabcus 2y ago
  I don't think it needs to be deterministic - and if it isn't, having the data and code becomes even more important! Compilers generally aren't deterministic (see the reproduceable build movement), yet we still use their output binaries.
  - brianstrimp 2y ago
    [dead]
blackeyeblitzar 2y ago
The only one I’ve see people talk about that shares all the components is OLMo (https://allenai.org/blog/olmo2 https://allenai.org/blog/olmo2)
- reedciccio 2y ago
  There are more, like the work by Eleuther AI and LLM360.
mirsadm 2y ago
No one will release the dataset because we all know it is gathered through dodgy means.
- notyourwork 2y ago
  And likely massaged by the CCP.
  - chgs 2y ago
    Deep mind has one set of censorship, OpenAI another, anything musk does a third It’s all “massaged”
    - redcobra762 2y ago
      I don’t think “Taiwan is China” is the same kind of massaging as not telling people how to make napalm… What a weird thing to equate, though!
      - foxglacier 2y ago
        It is though. Western AI tries to hide information like that with the justification of safety as well as things that might be offensive to current popular beliefs. Chinese AI presumably says Taiwan is China to help get more people on side for a possible future invasion. Propaganda does work - look at how many people think Donbas is still Ukraine and Israel is still Palestine.
        redcobra762 2y ago
        The difference is that in China the info isn’t available without use of Western content, due to the totalitarian control over media, whereas in the West, information is pretty trivially available, even if the big companies keep it off of their platforms. And sure ignorance is prevalent, but even GPT4 will tell me Donbas is still Ukraine, for instance. What a strange example to use, though!
reedciccio 2y ago
That's what the Open Source AI Definition states https://opensource.org/ai https://opensource.org/ai In any case, Deepseek like Llama fail much before hitting that new definition. Both have licenses containing restrictions on field of use and discrimination of users. Their license will never be approved as Open Source.
mythz 2y ago
This nitpicking is pointless. DeepSeek's gifts to the world of its open weights, public research and OSS code of its SOTA models are all any reasonable person should expect given no organization is going to release their dataset and open themselves up to criticism and legal exposure. You shouldn't expect to any to see datasets behind any SOTA models until they're able to be synthetically generated from larger models. Models only trained on sanctioned "public" datasets are not going to perform as well which makes them a lot less interesting and practically useful. Yes it would be great for their to be open models containing original datasets and a working pipeline to recreate models from scratch. But when few people would even have the resources to train the models and the huge training costs just result in worse performing models, it's only academically interesting to a few research labs. Open model releases should be celebrated, not criticized with unreasonable nitpicking and expectations that serves no useful purpose other than discouraging future open releases. When the norm is for Open Models to include their datasets, we can start criticizing those that don't, but until then be gracious that they're contributing anything at all.
- redcobra762 2y ago
  “Gifts to the world”? What a strange thing to say…
- jimjimwii 2y ago
  Terminology exists for a reason. Doubly so for well-established terms of art that pertain to licensing and contract law. They could have used "open wights" which would have conveyed the company's desired intent just as well as "open source", but without the ambiguity. They deliberately chose to misuse a well established term instead. I applaud and thank deepseek for opening their weights, but i absolutely condemn them and others (e.g Facebook) for their deliberate and continued misuse of the term. I and others like me will continue to raise this point as long as we are active in this field, so expect to see this criticism for decades. Hopefully one of these companies losses a lawsuit due to these shenanigans. Perhaps then they wouldn't misuse these terms so brazenly.
  - mythz 2y ago