5 ms·

I’m surprised by the comments here being so favorable to anthropic. The comments are right about there being no “best” token, and yeah Gruber may have an agenda

I’m surprised by the comments here being so favorable to anthropic. The comments are right about there being no “best” token, and yeah Gruber may have an agenda here. But I think the fundamental principle is that this approach messes with the distribution in ways that deviate from the trained model. Take the “gray” and “overcast” choices. And lets say before applying synthid the percentages were 48% and 52%. Those percentages were learned from the training data and RL. To change those percentages to 45% and 55% in a non learned way makes it seem like training wasn’t important? Or more likely they dont have the data that shows the failure modes? also, don’t these choices compound the changes to the distribution in later sampling choices? It is a bit of a mystery to say that “its okay to choose different tokens that we would have for watermarking bc people don’t notice” as though word choice doesn’t matter. If it doesn’t matter, doesn’t that mean that intelligence is more of a commodity than they would want it to be?

matheusmoreira 1mo ago
> But I think the fundamental principle is that this approach messes with the distribution in ways that deviate from the trained model. This. I want the model I'm paying for to be "pure". I don't Anthropic or anyone else messing around with it, especially not for idiotic reasons like facillitating AI stigmatization. The "safety" nonsense is obnoxious enough. They should train the best possible model and let the weights speak for themselves, not degrade it into some perverted form to appease people who hate AI anyway.
- applicative 1mo ago
  You can’t be serious. The weights for composition are massively degraded by RLVR training for coding
  - mangoman 1mo ago
    I agree, but thats also learned, and it is done to encourage specific responses for a task, different than applying a mask to the distribution based on a key.
- josephg 1mo ago
  > especially not for idiotic reasons like facillitating AI stigmatization. From the people I’ve talked to at universities, LLM based cheating in education is an unstoppable nightmare. I don’t have a problem with LLMs. But I do want the cheating to - somehow - stop. The people who cheat miss out on learning. And the people who don’t cheat have their degrees devalued by those who do cheat.
  - matheusmoreira 1mo ago
    > The people who cheat miss out on learning. They aren't there to learn. They are there to jump through hoops to get a degree that will let them get a job so they can make money and prosper. The learning is entirely secondary. The cheating will stop when there is no longer any economic incentive to be there in the first place. People with "pure" motives will refuse to cheat on their own, precisely because they want to learn and cheating prevents them from learning. Watermarking AI output is a treatment for symptoms. The cause is the higher education meme. Somehow, getting a degree just became the default. Can't get a good job without a degree. That meme needs to die, and higher education will never recover its integrity until it does.
    - sfink 1mo ago
      Easy to say, hard to come up with a believable alternative. In the meantime, you're flunking out a lot of people for having the integrity to not cheat and as a result not being able to keep up with an artificially inflated workload. You can't just destroy some signal and handwave that you'll make it up in some other way.
      - matheusmoreira 1mo ago
        > the integrity There is none. It's just a jobs program fueled by student loans. Higher education in the west has been corrupt for quite a while now. AI is just the final nail in its coffin.
        josephg 1mo ago
- remus 1mo ago
  > I want the model I'm paying for to be "pure". I don't think the models are pure in any meaningful sense. The labs have some idea of what kind of output they want from the models and then they put a huge amount of effort into training the models on the right sorts of data and massaging the models afterwards to push them towards the desired output. Then at a more practical level there's the layers of filters before your prompt even hits the model (e.g. anthropic's auto-mode classifier), system prompts, response level filtering etc.
josephg 1mo ago
I’m sure it matters. But how much does it matter? How much (perceived) intelligence would you be willing to sacrifice for an accurate AI predictor? I’d sacrifice a few %, easily. Maybe 10%. The models are getting smarter at such a fast rate that I’d be willing to lose a month or two of progress to help slow down the AI cheating epidemic. It sounds like you expect this fingerprinting approach would dramatically reduce the intelligence of their models. But I’m sure anthropic has measured it. I doubt they would have rolled this out if the intelligence cost were that significant. I suspect the cost is less than 5%. I personally can’t tell any difference from before they added fingerprinting. I bet you can’t either.
- mangoman 1mo ago
  10% is a lot! And why would this slow down the cheating epidemic? There’s tons of ways like declaude etc to get around the check. Also Claude’s style is very distinctive (e.g. “load bearing”) AND has changed since 4.5 quite dramatically. I might not be able to tell on a specific response, but I can tell you that I went from canceling my ChatGPT plan in November, to now reaching for it first and considering cancelling Claude because its style is getting really groan-inducing. It’s weird because I thought ChatGPT was really annoying about a year ago, and now codex is my first choice. Is the cheating epidemic so bad? I’m a little out of the loop there truthfully, what are the consequences of not being able to detect AI generated text in non academic settings? And in academic settings… maybe I am underestimating the challenge, but it does feel like the assignment and ways education happens needs to change? As an aside, I’m not totally sure why this solution feels so icky to me. There’s something Orwellian about how the phrasing of a passage embeds hidden information that only Anthropic can see i guess
  - josephg 1mo ago
    > There’s tons of ways like declaude etc to get around the check How effective is this on the new fingerprinting mechanisms? > Is the cheating epidemic so bad? From what I’ve heard, yeah it’s out of control. And all the existing llm detectors that academics use have a high false positive rate, which catches a bunch of innocent students in the cross fire. > it does feel like the assignment and ways education happens needs to change? Why? Was there something fundamentally wrong with how universities teach and assess? The sector is responding. For example by moving back to more in person exams and reducing the load of any take home exams. Is that good, for some reason? > There’s something Orwellian about how the phrasing of a passage embeds hidden information Interesting. I don’t have the same response. LLMs give me an acute sense of existential dread each time their capabilities improve. But fingerprinting doesn’t move me at all. Do some soul searching on why this bothers you. I’d love to hear why, and I bet you aren’t alone.
jorisw 1mo ago
> Gruber may have an agenda here Such as?
wewxjfq 1mo ago
You're assuming the training data is a true representation of language usage, which it isn't. In the "gray" or "overcast" case, a single weather website might already skew the count of these words.
andy12_ 1mo ago
> I think the fundamental principle is that this approach messes with the distribution in ways that deviate from the trained model. But it doesn't! The distribution doesn't change at all. The only thing that changes is that sampling of that distribution becomes deterministic as per a precomputed seed.
- sfink 1mo ago
  You could describe that as taking an input distribution and a sampling procedure and producing an output distribution. This is a difference in sampling procedure that produces a deviation in the output distribution. (If you don't like calling it a distribution when it's at 100% for the chosen token and 0% for all others, then look at it as an output distribution across all possible prompt inputs, or perhaps just the cluster of prompts that achieve whatever you're trying to accomplish.)
elim_garak 1mo ago
> To change those percentages to 45% and 55% in a non learned way makes it seem like training wasn’t important? I think (or I hope, anyway) that this overestimates how much impact the tweaks actually have on text. The model has some things it "wants" to say. If it wants to tell a story about how someone reacted to dreary weather, it's going to tell approximately the same story regardless of whether the dice-roll caused it to describe the weather as "gray" or "overcast". And because "gray" and "overcast" were _already_ possibilities, the tweak from 52% -> 55% is completely lost in the noise. But it's true that this is all based on hope. I'm confident that you could make the tweak against arbitrary prose and even a true artiste like Gruber would never be able to tell the difference. I'm less confident that there isn't some edge case somewhere that causes a tweak to be worse than 3%, especially in some narrow application where word choice _does_ matter (like law). Even then, though, laws are already written by people who are as noisy if not noisier than LLMs.
oconnor663 1mo ago
> To change those percentages to 45% and 55% This seems like a fundamental misunderstanding of how this sort of watermarking works. (Either that, or I have a fundamental misunderstanding of how it works lol.) It doesn't change the probability distribution of the next token at all. If you were getting XYZ 48% of the time before, you're still getting XYZ 48% of the time. What's changed is where the random numbers come from. But as far as you're concerned, there's just as random as they were before, just like an encrypted message is indistinguishable from random bytes if you don't know the key.
- mangoman 1mo ago
  huh, thanks for commenting this! I trusted the linked explanation post https://declaude.org/watermarking/ https://declaude.org/watermarking/ but actually reading the the synthid paper showed me that my understanding was wrong: https://www.nature.com/articles/s41586-024-08025-4 https://www.nature.com/articles/s41586-024-08025-4 It is definitely blurrier whether you can say this approach changes the distribution then. By definition, it _has_ to change the probabilities of output tokens, but it's not totally clear that the pseudorandomly generated scoring functions does affect the learned distribution. put another way, I think it's safer to do: compute distribution -> sample -> watermark from sampled options than it would be to do: compute distribution -> watermark distribution -> sample
  - Nimitz14 1mo ago
    You’re not getting it. The probabilities do not change at all. The only change is given some probabilities there is a deterministic method for determining which symbol was sampled from that distribution. The distribution or sampling process itself is not modified.
    - mangoman 1mo ago
      Based on the SynthID-Text paper https://www.nature.com/articles/s41586-024-08025-4 https://www.nature.com/articles/s41586-024-08025-4 I agree that the LLM's learned distribution isn't modified, but I don't think it's correct to say that the sampling process is not modified. Also I just read the paper today so I could be misinterpreting things. As described in the paper, you're right that it doesn't affect the main sampling technique, but what they do is they sample the distribution for 2^m samples, and then use Tournament sampling to choose the tokens among those 2^m samples, and the watermark key changes the scoring of the tournament options, using the watermark key as an input to the random generator that generates the scoring functions. Then, to calculate the watermark, they take the text, and compute the mean g-values of the text, and a higher score means that it's more likely that it was sampled using the provided selection of tournament watermarking functions. let's say you had some top P words: mango, banana, pineapple, guava, and you sampled 8 times, and got each one twice in the following order: 1. mango 2. banana 3. pineapple 4. guava 5. mango 6. banana 7. pineapple 8. guava without tournament sampling, you'd truly see any of those come through. But in tournament sampling, you take those 8 options, create m scoring functions based on the pseudorandom generator, and score the 'tournament' by sampling the biased distribution you create from the g values. That does change the sampling from based purely on the LLM and entropy, but i mean, if the watermark key is also generated from some entropy, it's probably representative of the original sampling options as expected? this is a very fascinating topic! I do still stand by my point that anthropic is the only one who can tell if something is watermarked or not and feeling icky, but the paper has mostly quelled my concern on impacting the intelligence part.

matheusmoreira 1mo ago

> But I think the fundamental principle is that this approach messes with the distribution in ways that deviate from the trained model. This. I want the model I'm paying for to be "pure". I don't Anthropic or anyone else messing around with it, especially not for idiotic reasons like facillitating AI stigmatization. The "safety" nonsense is obnoxious enough. They should train the best possible model and let the weights speak for themselves, not degrade it into some perverted form to appease people who hate AI anyway.

applicative 1mo ago
You can’t be serious. The weights for composition are massively degraded by RLVR training for coding
- mangoman 1mo ago
  I agree, but thats also learned, and it is done to encourage specific responses for a task, different than applying a mask to the distribution based on a key.
josephg 1mo ago
> especially not for idiotic reasons like facillitating AI stigmatization. From the people I’ve talked to at universities, LLM based cheating in education is an unstoppable nightmare. I don’t have a problem with LLMs. But I do want the cheating to - somehow - stop. The people who cheat miss out on learning. And the people who don’t cheat have their degrees devalued by those who do cheat.
- matheusmoreira 1mo ago
  > The people who cheat miss out on learning. They aren't there to learn. They are there to jump through hoops to get a degree that will let them get a job so they can make money and prosper. The learning is entirely secondary. The cheating will stop when there is no longer any economic incentive to be there in the first place. People with "pure" motives will refuse to cheat on their own, precisely because they want to learn and cheating prevents them from learning. Watermarking AI output is a treatment for symptoms. The cause is the higher education meme. Somehow, getting a degree just became the default. Can't get a good job without a degree. That meme needs to die, and higher education will never recover its integrity until it does.
  - sfink 1mo ago
    Easy to say, hard to come up with a believable alternative. In the meantime, you're flunking out a lot of people for having the integrity to not cheat and as a result not being able to keep up with an artificially inflated workload. You can't just destroy some signal and handwave that you'll make it up in some other way.
    - matheusmoreira 1mo ago
      > the integrity There is none. It's just a jobs program fueled by student loans. Higher education in the west has been corrupt for quite a while now. AI is just the final nail in its coffin.
      - josephg 1mo ago
remus 1mo ago
> I want the model I'm paying for to be "pure". I don't think the models are pure in any meaningful sense. The labs have some idea of what kind of output they want from the models and then they put a huge amount of effort into training the models on the right sorts of data and massaging the models afterwards to push them towards the desired output. Then at a more practical level there's the layers of filters before your prompt even hits the model (e.g. anthropic's auto-mode classifier), system prompts, response level filtering etc.

josephg 1mo ago

I’m sure it matters. But how much does it matter? How much (perceived) intelligence would you be willing to sacrifice for an accurate AI predictor? I’d sacrifice a few %, easily. Maybe 10%. The models are getting smarter at such a fast rate that I’d be willing to lose a month or two of progress to help slow down the AI cheating epidemic. It sounds like you expect this fingerprinting approach would dramatically reduce the intelligence of their models. But I’m sure anthropic has measured it. I doubt they would have rolled this out if the intelligence cost were that significant. I suspect the cost is less than 5%. I personally can’t tell any difference from before they added fingerprinting. I bet you can’t either.

mangoman 1mo ago
10% is a lot! And why would this slow down the cheating epidemic? There’s tons of ways like declaude etc to get around the check. Also Claude’s style is very distinctive (e.g. “load bearing”) AND has changed since 4.5 quite dramatically. I might not be able to tell on a specific response, but I can tell you that I went from canceling my ChatGPT plan in November, to now reaching for it first and considering cancelling Claude because its style is getting really groan-inducing. It’s weird because I thought ChatGPT was really annoying about a year ago, and now codex is my first choice. Is the cheating epidemic so bad? I’m a little out of the loop there truthfully, what are the consequences of not being able to detect AI generated text in non academic settings? And in academic settings… maybe I am underestimating the challenge, but it does feel like the assignment and ways education happens needs to change? As an aside, I’m not totally sure why this solution feels so icky to me. There’s something Orwellian about how the phrasing of a passage embeds hidden information that only Anthropic can see i guess
- josephg 1mo ago
  > There’s tons of ways like declaude etc to get around the check How effective is this on the new fingerprinting mechanisms? > Is the cheating epidemic so bad? From what I’ve heard, yeah it’s out of control. And all the existing llm detectors that academics use have a high false positive rate, which catches a bunch of innocent students in the cross fire. > it does feel like the assignment and ways education happens needs to change? Why? Was there something fundamentally wrong with how universities teach and assess? The sector is responding. For example by moving back to more in person exams and reducing the load of any take home exams. Is that good, for some reason? > There’s something Orwellian about how the phrasing of a passage embeds hidden information Interesting. I don’t have the same response. LLMs give me an acute sense of existential dread each time their capabilities improve. But fingerprinting doesn’t move me at all. Do some soul searching on why this bothers you. I’d love to hear why, and I bet you aren’t alone.

jorisw 1mo ago

> Gruber may have an agenda here Such as?

wewxjfq 1mo ago

You're assuming the training data is a true representation of language usage, which it isn't. In the "gray" or "overcast" case, a single weather website might already skew the count of these words.

andy12_ 1mo ago

> I think the fundamental principle is that this approach messes with the distribution in ways that deviate from the trained model. But it doesn't! The distribution doesn't change at all. The only thing that changes is that sampling of that distribution becomes deterministic as per a precomputed seed.

sfink 1mo ago
You could describe that as taking an input distribution and a sampling procedure and producing an output distribution. This is a difference in sampling procedure that produces a deviation in the output distribution. (If you don't like calling it a distribution when it's at 100% for the chosen token and 0% for all others, then look at it as an output distribution across all possible prompt inputs, or perhaps just the cluster of prompts that achieve whatever you're trying to accomplish.)

elim_garak 1mo ago

> To change those percentages to 45% and 55% in a non learned way makes it seem like training wasn’t important? I think (or I hope, anyway) that this overestimates how much impact the tweaks actually have on text. The model has some things it "wants" to say. If it wants to tell a story about how someone reacted to dreary weather, it's going to tell approximately the same story regardless of whether the dice-roll caused it to describe the weather as "gray" or "overcast". And because "gray" and "overcast" were _already_ possibilities, the tweak from 52% -> 55% is completely lost in the noise. But it's true that this is all based on hope. I'm confident that you could make the tweak against arbitrary prose and even a true artiste like Gruber would never be able to tell the difference. I'm less confident that there isn't some edge case somewhere that causes a tweak to be worse than 3%, especially in some narrow application where word choice _does_ matter (like law). Even then, though, laws are already written by people who are as noisy if not noisier than LLMs.

oconnor663 1mo ago

> To change those percentages to 45% and 55% This seems like a fundamental misunderstanding of how this sort of watermarking works. (Either that, or I have a fundamental misunderstanding of how it works lol.) It doesn't change the probability distribution of the next token at all. If you were getting XYZ 48% of the time before, you're still getting XYZ 48% of the time. What's changed is where the random numbers come from. But as far as you're concerned, there's just as random as they were before, just like an encrypted message is indistinguishable from random bytes if you don't know the key.

mangoman 1mo ago
huh, thanks for commenting this! I trusted the linked explanation post https://declaude.org/watermarking/ https://declaude.org/watermarking/ but actually reading the the synthid paper showed me that my understanding was wrong: https://www.nature.com/articles/s41586-024-08025-4 https://www.nature.com/articles/s41586-024-08025-4 It is definitely blurrier whether you can say this approach changes the distribution then. By definition, it _has_ to change the probabilities of output tokens, but it's not totally clear that the pseudorandomly generated scoring functions does affect the learned distribution. put another way, I think it's safer to do: compute distribution -> sample -> watermark from sampled options than it would be to do: compute distribution -> watermark distribution -> sample
- Nimitz14 1mo ago
  You’re not getting it. The probabilities do not change at all. The only change is given some probabilities there is a deterministic method for determining which symbol was sampled from that distribution. The distribution or sampling process itself is not modified.
  - mangoman 1mo ago
    Based on the SynthID-Text paper https://www.nature.com/articles/s41586-024-08025-4 https://www.nature.com/articles/s41586-024-08025-4 I agree that the LLM's learned distribution isn't modified, but I don't think it's correct to say that the sampling process is not modified. Also I just read the paper today so I could be misinterpreting things. As described in the paper, you're right that it doesn't affect the main sampling technique, but what they do is they sample the distribution for 2^m samples, and then use Tournament sampling to choose the tokens among those 2^m samples, and the watermark key changes the scoring of the tournament options, using the watermark key as an input to the random generator that generates the scoring functions. Then, to calculate the watermark, they take the text, and compute the mean g-values of the text, and a higher score means that it's more likely that it was sampled using the provided selection of tournament watermarking functions. let's say you had some top P words: mango, banana, pineapple, guava, and you sampled 8 times, and got each one twice in the following order: 1. mango 2. banana 3. pineapple 4. guava 5. mango 6. banana 7. pineapple 8. guava without tournament sampling, you'd truly see any of those come through. But in tournament sampling, you take those 8 options, create m scoring functions based on the pseudorandom generator, and score the 'tournament' by sampling the biased distribution you create from the g values. That does change the sampling from based purely on the LLM and entropy, but i mean, if the watermark key is also generated from some entropy, it's probably representative of the original sampling options as expected? this is a very fascinating topic! I do still stand by my point that anthropic is the only one who can tell if something is watermarked or not and feeling icky, but the paper has mostly quelled my concern on impacting the intelligence part.