9 ms·

Is this how the eastern labs "distill" SOTA models? If you can play it right, you don't even need to send suspicious prompts to the frontier models. Just use t

Is this how the eastern labs "distill" SOTA models? If you can play it right, you don't even need to send suspicious prompts to the frontier models. Just use them for regular tasks, extract the encrypted COT blocks and replay it to a cheaper model to get the plain text COT. But the real question is: Is it okay to steal from a thief's hoard?

azinman2 1mo ago
The reasoning blocks are not stolen/mined from the internet at large directly. They’re the result of a lot of research, time, money, and expertise into creating a reasoning model. To me the answer is quite clearly no, especially when the encrypted blocks demonstrate they want to protect it.
- tristanj 1mo ago
  No. There are dozens of companies that resell tokens at a discount to collect and resell session data to various Chinese labs.
  - flawn 1mo ago
    So you say, they at least create economic value through obscurity of something which should be accessible?
- deleted 1mo ago
  [deleted]
- elzbardico 1mo ago
  Most post-training tasks are based on real open source projects. A lot of time on real issues posted on issue trackers. Besides that, the capabilities of a model are heavily dependent on the unsupervised learning phase, that gobbles all kind of other people's IP without giving a fuck. All the underpaid work behind the masses of third world programmers creating those post-training datasets would be completely uselless without it. Also, it is kind of funny that labs resort to the "Research, time, money and expertise" argumet, when it is basically the same argument from publishers and other IP creator that the labs spent millions of dollars of lawyering money to resist. Besides, US law rejects in: Effort and cost by themselves not necessarely generate protectable interests. About encryption, I think we're all contaminated by the bad ideology behind DMCA. While encryption established the intent, it doesn't follow that they have a legal claim of exclusivity just because of it. Technically, you're overstating the value of so called "reasoning traces". You can't infer the verifier design, the reward shaping,or the data pipeline from them. Also, what you can extract are not the traces themselves, but the written summary of it, and you can't even guarantee that this summary reflects the exactly reasoning trace, models have show to have lied about it. Besides, distillation works when the student model already has strong priors, you can't turn a weak model in a SOTA with it. Don't believe Amodei's outrageous lies about it, he is just trying to exercise some regulatory capture.
  - blackqueeriroh 1mo ago
    Are you a judge or a lawyer? If not, then you don’t know one way or another.
    - elzbardico 1mo ago
      And what is your argument?
- pyrale 1mo ago
  Stuff available on the internet is also the result of a lot of research, time, money, and expertise. And AI companies taught us that it’s OK to yoink whatever is not bolted to the ground, even when it is illegal to do so.
- qwytw 1mo ago
  That's a moral stance one can take (regardless of the severe cognitive dissonance embedded in it). But what does that have to do with theft? LLM providers don't own the copyrights to the outputs of their models (at least not yet).
- polymer8563 1mo ago
  if you make reasoning soup of my data withou my consent it still is my data and i did not ask for your reasoning soup
NitpickLawyer 1mo ago
> But the real question is: Is it okay to steal By definition it cannot be stealing since you're paying for the tokens. It may be against their ToS, depending on what you end up doing with those tokens, but it cannot be stealing. If they charge by the token, all your tokens are belong to you :) I also find it very strange that everyone sort of accepts their ToS like no big deal. Imagine MS using the same terms for their software - you cannot use any MS software to develop competing services. Bananas! They'd be dragged through the courts like it's the 90s. (I get why they're doing it. Distillation is unreasonably effective. But still, I find it bananas that we've kinda accepted it, to the point where people use "stealing" or "attack" or any such terms)
- tuesdaynight 1mo ago
  It's like sideloading. It's very hard to fight against the marketing budget of big tech
- desterothx 1mo ago
  I love how some of the biggest advancements in llms came from the Chinese labs, yet people still jump to distillation being unreasonably effective. Distillation is very good at creating smaller models from large ones sure, but nothing to me indicates it is 'unreasonably effective' compared to all the other bells and whistles being iterated on
  - rfoo 1mo ago
    Let's face it. Chinese labs made some of the biggest advancements. AND training on Claude (or GPT) output IS unreasonably effective. The two sentences are true at the same time.
  - varenc 1mo ago
    This article shows that when Kimi3's chain of thought is prefilled to match Opus's, the rest of the chain of thoughts Kimi3 outputs very closely aligns with Opus's. That seems strong evidence that Kimi3 is partly a distillation of Opus. And Kimi3 is not a small model. No doubt a lot of hard work went into Kimi, but seems clear that distillation was used effectively as well. (though maybe there's another interpretation of the thought alignment?)
orbital-decay 1mo ago
Not necessarily. There's a million ways to jailbreak any current model to show the trace and bypass all guardrails, or hijack and modify it. It's just one of them.
qwytw 1mo ago
>But the real question is: Is it okay to steal from a thief's hoard? How does this relate to your previous paragraphs? LLM outputs are not copyrightable and you didn't break into Anthropic servers to steal the files from there. So how exactly is it theft? If I send an "encrypted" files to thousands of peoples and some manage to figure out how to read it I can't really accuse them of that or can I?

azinman2 1mo ago

The reasoning blocks are not stolen/mined from the internet at large directly. They’re the result of a lot of research, time, money, and expertise into creating a reasoning model. To me the answer is quite clearly no, especially when the encrypted blocks demonstrate they want to protect it.

tristanj 1mo ago
No. There are dozens of companies that resell tokens at a discount to collect and resell session data to various Chinese labs.
- flawn 1mo ago
  So you say, they at least create economic value through obscurity of something which should be accessible?
deleted 1mo ago
[deleted]
elzbardico 1mo ago
Most post-training tasks are based on real open source projects. A lot of time on real issues posted on issue trackers. Besides that, the capabilities of a model are heavily dependent on the unsupervised learning phase, that gobbles all kind of other people's IP without giving a fuck. All the underpaid work behind the masses of third world programmers creating those post-training datasets would be completely uselless without it. Also, it is kind of funny that labs resort to the "Research, time, money and expertise" argumet, when it is basically the same argument from publishers and other IP creator that the labs spent millions of dollars of lawyering money to resist. Besides, US law rejects in: Effort and cost by themselves not necessarely generate protectable interests. About encryption, I think we're all contaminated by the bad ideology behind DMCA. While encryption established the intent, it doesn't follow that they have a legal claim of exclusivity just because of it. Technically, you're overstating the value of so called "reasoning traces". You can't infer the verifier design, the reward shaping,or the data pipeline from them. Also, what you can extract are not the traces themselves, but the written summary of it, and you can't even guarantee that this summary reflects the exactly reasoning trace, models have show to have lied about it. Besides, distillation works when the student model already has strong priors, you can't turn a weak model in a SOTA with it. Don't believe Amodei's outrageous lies about it, he is just trying to exercise some regulatory capture.
- blackqueeriroh 1mo ago
  Are you a judge or a lawyer? If not, then you don’t know one way or another.
  - elzbardico 1mo ago
    And what is your argument?
pyrale 1mo ago
Stuff available on the internet is also the result of a lot of research, time, money, and expertise. And AI companies taught us that it’s OK to yoink whatever is not bolted to the ground, even when it is illegal to do so.
qwytw 1mo ago
That's a moral stance one can take (regardless of the severe cognitive dissonance embedded in it). But what does that have to do with theft? LLM providers don't own the copyrights to the outputs of their models (at least not yet).
polymer8563 1mo ago
if you make reasoning soup of my data withou my consent it still is my data and i did not ask for your reasoning soup

NitpickLawyer 1mo ago

> But the real question is: Is it okay to steal By definition it cannot be stealing since you're paying for the tokens. It may be against their ToS, depending on what you end up doing with those tokens, but it cannot be stealing. If they charge by the token, all your tokens are belong to you :) I also find it very strange that everyone sort of accepts their ToS like no big deal. Imagine MS using the same terms for their software - you cannot use any MS software to develop competing services. Bananas! They'd be dragged through the courts like it's the 90s. (I get why they're doing it. Distillation is unreasonably effective. But still, I find it bananas that we've kinda accepted it, to the point where people use "stealing" or "attack" or any such terms)

tuesdaynight 1mo ago
It's like sideloading. It's very hard to fight against the marketing budget of big tech
desterothx 1mo ago
I love how some of the biggest advancements in llms came from the Chinese labs, yet people still jump to distillation being unreasonably effective. Distillation is very good at creating smaller models from large ones sure, but nothing to me indicates it is 'unreasonably effective' compared to all the other bells and whistles being iterated on
- rfoo 1mo ago
  Let's face it. Chinese labs made some of the biggest advancements. AND training on Claude (or GPT) output IS unreasonably effective. The two sentences are true at the same time.
- varenc 1mo ago
  This article shows that when Kimi3's chain of thought is prefilled to match Opus's, the rest of the chain of thoughts Kimi3 outputs very closely aligns with Opus's. That seems strong evidence that Kimi3 is partly a distillation of Opus. And Kimi3 is not a small model. No doubt a lot of hard work went into Kimi, but seems clear that distillation was used effectively as well. (though maybe there's another interpretation of the thought alignment?)

orbital-decay 1mo ago

Not necessarily. There's a million ways to jailbreak any current model to show the trace and bypass all guardrails, or hijack and modify it. It's just one of them.

qwytw 1mo ago

>But the real question is: Is it okay to steal from a thief's hoard? How does this relate to your previous paragraphs? LLM outputs are not copyrightable and you didn't break into Anthropic servers to steal the files from there. So how exactly is it theft? If I send an "encrypted" files to thousands of peoples and some manage to figure out how to read it I can't really accuse them of that or can I?