6 ms·

Text watermarking has no effect on output quality, it just works by changing the explicit source of randomness that is in practice always present in LLM output

Text watermarking has no effect on output quality, it just works by changing the explicit source of randomness that is in practice always present in LLM output sampling. See for example https://www.seangoedecke.com/ai-text-watermarking-is-not-a-big-deal/ https://www.seangoedecke.com/ai-text-watermarking-is-not-a-b....

pkulak 15d ago
> Text watermarking has no effect on output quality It has an effect, and it's negative. It's hoped that the effect is negligible, and it probably is, but the whole point is that it has an effect.
- arrrg 15d ago
  Why do you claim that? There is no reason why there has to be a negative effect of text watermarking.
  - pkulak 15d ago
    It literally re-weights the output tokens from what the LLM would otherwise have chosen. It _has_ to. It can't be positive, because then that's not watermarking, it's a better LLM.
    - frabcus 15d ago
      It's a very unintuitive algorithm, and is pretty clever. I recommend reading up on it: https://www.nature.com/articles/s41586-024-08025-4 https://www.nature.com/articles/s41586-024-08025-4 But no, it only ever picks tokens that are in the probability distribution of the last layer, and it might have picked anyway.
      - throwuxiytayq 15d ago
        What if the next token represents a wrong or low-quality answer, but would have only been picked 10% of the time, but now it's picked 20% of the time? Doesn't that obviously decrease the model quality, even though "it might have picked that token anyway"?
        dan-robertson 15d ago
        It would be picked 10% of the time with watermarking. The randomness properties of the PRNG will be very similar to other random number generators, it is just chosen to be vulnerable to a particular cryptanalytic attack (that requires a private key known only to anthropic). I think of it like the Dual_EC_DRGB generator rather than a biased coin.
        throwuxiytayq 14d ago
        It is absolutely possible that it would not continue to be picked 10% of the time with a given fixed watermark key. The implementation literally labels tokens using a keyed hash and then modifies their scores. The entire point of the watermarking system is to bias certain tokens against others, and - as you would expect - this reportedly results in a reduced response diversity.
        deleted 14d ago
        [deleted]
        wasabi991011 14d ago
        What if the token represents a high-quality answer, but would have only been picked 80% of the time, but now it's picked 90% of the time? I'm not entirely sure (haven't read the original synthID proposal), but I believe that the re-weighing is set to make both your scenarios and mine equally likely, averaging out to net Zero effect on quality.
      - dbdr 15d ago
        To add nuance, that article does say one of the two versions does reduce text quality (and the other is worse at detectability): > SynthID-Text can be configured to be non-distortionary (preserving text quality) or distortionary (improving watermark detectability at the cost of text quality).
    - qgin 15d ago
      Unless you’re running at temperature 0, there’s not one single token that the model definitely would have chosen each time.
    - brokencode 15d ago
      Unless you’re at 0 temperature, there is no single token it would have chosen. It’s always picking one of multiple randomly according to a probability distribution.
      - northzen 15d ago
        Give me an example how would you watermark a single short sentence like "I like turtles"?
        qbit42 15d ago
        Watermarking just alters the pseudorandom number generator. If "I like turtles" was previously the response to your prompt with probability 100%, it will still be so. This is why watermarking is only effective for long strings of text
        northzen 15d ago
        It's like the sudden change of a language style and its verbosity didn't happen recently. To random words you pick and provide a sufficient amount of text to vary with random number without losing its meaning you need a text with high entropy.
        qlte 15d ago
        Nothing about watermarking would require padding the response length with pseudo-intelligible Claudese. Regular filler would work fine. Also, it would probably provide higher entropy to write normal human-sounding English instead of reusing a repetitive grab bag of load-bearing phrases. This theory doesn't really make any sense.
    - WithinReason 15d ago
      No, it basically uses a fixed seed for the random number generator to generate tokens. The generated random sequence is just as random as it would be with a random seed.
- NewsaHackO 15d ago
  I am pretty sure they did A/B testing to show it didn't. I could gave sworn they even released a quiz were the user has to try and guess which answer is watermarked or not and it was impossible to tell.
  - sejje 15d ago
    That's not the metric, though. You either output the best version, or you output something else. You can't do both.
    - frabcus 15d ago
      That's not the case, because LLMs are non-deterministic. It only alters outputs when the last layer of the neural network give significant weights to multiple tokens, and it would anyway have picked a random answer. Instead it picks a non-random one, but non-random in such a way that you can't tell without the private key of the watermarking. This mostly adds randomness these days for branches in syntax that make no difference, and the model has no reason to believe make a difference. Anything that matters, it is much more confident in the last layer of weights on the token to use.
      - SkyBelow 15d ago
        >That's not the case, because LLMs are non-deterministic. That feels a bit like a lie. At the core, they are deterministic. We found that adding some ability to randomly pick the second or third best tokens made for better output, so we added temperature. And then we started running them in optimized ways where your answer is deterministic only if the batch of tokens are the same (not your input tokens, but other tokens in another batch being processed), and in practice those are never the same. Lastly, we use harnesses that do things like adding IDs and timestamps to the context, which means the same exact text from the user does not lead to the same text hitting the AI. The final result is that, in practice, you are right (unless you run a model fully locally, where you can seed temperature and turn off all these other features). But strictly calling it non-deterministic makes it sound like the underlying algorithm is itself non-deterministic (and I've seen many people with that misunderstanding) rather than it being a result of how we purposefully changed the algorithm for better results. A bit like saying path finding is non-deterministic, because having the best pathfinding makes for poor gameplay, so we added some randomness to NPC path finding to make it more realistic. The given implementation is non-deterministic, but the underlying algorithm isn't.
- markasoftware 15d ago
  Its essentially swapping out the psuedo random number generated with a differently seeded one iirc. It has an effect on the output, but not the output quality
  - Razengan 15d ago
    umm so like wait, if I ask it to paint with a shade of red, but it paints with a slightly different shade of red, that is a fucking effect on quality, pardon my watermarking If you type like Joey using a thesaurus for the first time, it has an effect on quality
    - markasoftware 15d ago
      You do not get it. The llm never deterministically picks a shade of red. It's a probability distribution over shades of colors, with certain shades of red being more likely than others. Without fingerprinting, it randomly samples from the distribution using a certain pseudorandom RNG. With fingerprinting, it also selects from the distribution using a pseudorandom RNG. My understanding is that the fingerprinted prng is still a strong RNG. Neither output is more correct than the other. If a certain token is far more likely than any other, it's usually chosen even in the fingerprinted output.
      - Razengan 15d ago
        But language isn't like that, if you change any word, or even punctuation, it changes the tone perceived by humans
        markasoftware 15d ago
        Yes, language is like that, at least the kind of language produced by LLMs. All LLMs produce a probability distribution at each token. If you run the LLM multiple times with the same prompt you will observe it generate different responses. Using the watermarked prng does not change the distribution. When generating tokens that might be critical to the tone or grammar or correctness, the probability distribution might be 99% on a certain token. In these cases, with or without watermarking, the output will almost always be that same token. E.g., if you ask "please output the exact word watermelon", the LLM will output watermelon with 99%+ probability even with watermark (i.e., the output won't actually be detectable as watermarked).
- geysersam 15d ago
  It has an effect, 50% of the time it's positive, increasing the quality of the answer, and 50% of the time it's negative.
  - qgin 15d ago
    These models are already running with a temperature that is >0. This just replaces a random number with a slightly less random number.
    - geysersam 15d ago
      That was my point, replacing one random sampling by another has exactly that effect.
- k1m 15d ago
  It reduces diversity, which they don't talk about much. Wrote about it here. https://blog.keyvan.net/p/ai-text-watermarking-and-quality https://blog.keyvan.net/p/ai-text-watermarking-and-quality
- reasonableklout 15d ago
  It seems fine to me. The model is still solving my problems and writing code that works as well as any other. Google has been watermarking text with SynthID for a while now and nobody complained about it. Why all the fuss about Claude? It feels like the real reason behind most complaints is that people want to use AI for writing and not have others find out?
exabrial 15d ago
This is hilarious this keeps being repeated by the true believers ad nauseam. Also, don't apply EU law to the world. It's a knee jerk reactionary regulation by a bunch of aging ding dongs that can't print their emails.
- frabcus 15d ago
  You're on Hacker News - I suggest you have technical curiosity and actually understand this very unusual and innovative algorithm, before you claim things about it that aren't true.

pkulak 15d ago

> Text watermarking has no effect on output quality It has an effect, and it's negative. It's hoped that the effect is negligible, and it probably is, but the whole point is that it has an effect.

arrrg 15d ago
Why do you claim that? There is no reason why there has to be a negative effect of text watermarking.
- pkulak 15d ago
  It literally re-weights the output tokens from what the LLM would otherwise have chosen. It _has_ to. It can't be positive, because then that's not watermarking, it's a better LLM.
  - frabcus 15d ago
    It's a very unintuitive algorithm, and is pretty clever. I recommend reading up on it: https://www.nature.com/articles/s41586-024-08025-4 https://www.nature.com/articles/s41586-024-08025-4 But no, it only ever picks tokens that are in the probability distribution of the last layer, and it might have picked anyway.
    - throwuxiytayq 15d ago
      What if the next token represents a wrong or low-quality answer, but would have only been picked 10% of the time, but now it's picked 20% of the time? Doesn't that obviously decrease the model quality, even though "it might have picked that token anyway"?
      - dan-robertson 15d ago
        It would be picked 10% of the time with watermarking. The randomness properties of the PRNG will be very similar to other random number generators, it is just chosen to be vulnerable to a particular cryptanalytic attack (that requires a private key known only to anthropic). I think of it like the Dual_EC_DRGB generator rather than a biased coin.
        throwuxiytayq 14d ago
        It is absolutely possible that it would not continue to be picked 10% of the time with a given fixed watermark key. The implementation literally labels tokens using a keyed hash and then modifies their scores. The entire point of the watermarking system is to bias certain tokens against others, and - as you would expect - this reportedly results in a reduced response diversity.
        deleted 14d ago
        [deleted]
      - wasabi991011 14d ago
        What if the token represents a high-quality answer, but would have only been picked 80% of the time, but now it's picked 90% of the time? I'm not entirely sure (haven't read the original synthID proposal), but I believe that the re-weighing is set to make both your scenarios and mine equally likely, averaging out to net Zero effect on quality.
    - dbdr 15d ago
      To add nuance, that article does say one of the two versions does reduce text quality (and the other is worse at detectability): > SynthID-Text can be configured to be non-distortionary (preserving text quality) or distortionary (improving watermark detectability at the cost of text quality).
  - qgin 15d ago
    Unless you’re running at temperature 0, there’s not one single token that the model definitely would have chosen each time.
  - brokencode 15d ago
    Unless you’re at 0 temperature, there is no single token it would have chosen. It’s always picking one of multiple randomly according to a probability distribution.
    - northzen 15d ago
      Give me an example how would you watermark a single short sentence like "I like turtles"?
      - qbit42 15d ago
        Watermarking just alters the pseudorandom number generator. If "I like turtles" was previously the response to your prompt with probability 100%, it will still be so. This is why watermarking is only effective for long strings of text
        northzen 15d ago
        It's like the sudden change of a language style and its verbosity didn't happen recently. To random words you pick and provide a sufficient amount of text to vary with random number without losing its meaning you need a text with high entropy.
        qlte 15d ago
        Nothing about watermarking would require padding the response length with pseudo-intelligible Claudese. Regular filler would work fine. Also, it would probably provide higher entropy to write normal human-sounding English instead of reusing a repetitive grab bag of load-bearing phrases. This theory doesn't really make any sense.
  - WithinReason 15d ago
    No, it basically uses a fixed seed for the random number generator to generate tokens. The generated random sequence is just as random as it would be with a random seed.
NewsaHackO 15d ago
I am pretty sure they did A/B testing to show it didn't. I could gave sworn they even released a quiz were the user has to try and guess which answer is watermarked or not and it was impossible to tell.
- sejje 15d ago
  That's not the metric, though. You either output the best version, or you output something else. You can't do both.
  - frabcus 15d ago
    That's not the case, because LLMs are non-deterministic. It only alters outputs when the last layer of the neural network give significant weights to multiple tokens, and it would anyway have picked a random answer. Instead it picks a non-random one, but non-random in such a way that you can't tell without the private key of the watermarking. This mostly adds randomness these days for branches in syntax that make no difference, and the model has no reason to believe make a difference. Anything that matters, it is much more confident in the last layer of weights on the token to use.
    - SkyBelow 15d ago
      >That's not the case, because LLMs are non-deterministic. That feels a bit like a lie. At the core, they are deterministic. We found that adding some ability to randomly pick the second or third best tokens made for better output, so we added temperature. And then we started running them in optimized ways where your answer is deterministic only if the batch of tokens are the same (not your input tokens, but other tokens in another batch being processed), and in practice those are never the same. Lastly, we use harnesses that do things like adding IDs and timestamps to the context, which means the same exact text from the user does not lead to the same text hitting the AI. The final result is that, in practice, you are right (unless you run a model fully locally, where you can seed temperature and turn off all these other features). But strictly calling it non-deterministic makes it sound like the underlying algorithm is itself non-deterministic (and I've seen many people with that misunderstanding) rather than it being a result of how we purposefully changed the algorithm for better results. A bit like saying path finding is non-deterministic, because having the best pathfinding makes for poor gameplay, so we added some randomness to NPC path finding to make it more realistic. The given implementation is non-deterministic, but the underlying algorithm isn't.
markasoftware 15d ago
Its essentially swapping out the psuedo random number generated with a differently seeded one iirc. It has an effect on the output, but not the output quality
- Razengan 15d ago
  umm so like wait, if I ask it to paint with a shade of red, but it paints with a slightly different shade of red, that is a fucking effect on quality, pardon my watermarking If you type like Joey using a thesaurus for the first time, it has an effect on quality
  - markasoftware 15d ago
    You do not get it. The llm never deterministically picks a shade of red. It's a probability distribution over shades of colors, with certain shades of red being more likely than others. Without fingerprinting, it randomly samples from the distribution using a certain pseudorandom RNG. With fingerprinting, it also selects from the distribution using a pseudorandom RNG. My understanding is that the fingerprinted prng is still a strong RNG. Neither output is more correct than the other. If a certain token is far more likely than any other, it's usually chosen even in the fingerprinted output.
    - Razengan 15d ago
      But language isn't like that, if you change any word, or even punctuation, it changes the tone perceived by humans
      - markasoftware 15d ago
        Yes, language is like that, at least the kind of language produced by LLMs. All LLMs produce a probability distribution at each token. If you run the LLM multiple times with the same prompt you will observe it generate different responses. Using the watermarked prng does not change the distribution. When generating tokens that might be critical to the tone or grammar or correctness, the probability distribution might be 99% on a certain token. In these cases, with or without watermarking, the output will almost always be that same token. E.g., if you ask "please output the exact word watermelon", the LLM will output watermelon with 99%+ probability even with watermark (i.e., the output won't actually be detectable as watermarked).
geysersam 15d ago
It has an effect, 50% of the time it's positive, increasing the quality of the answer, and 50% of the time it's negative.
- qgin 15d ago
  These models are already running with a temperature that is >0. This just replaces a random number with a slightly less random number.
  - geysersam 15d ago
    That was my point, replacing one random sampling by another has exactly that effect.
k1m 15d ago
It reduces diversity, which they don't talk about much. Wrote about it here. https://blog.keyvan.net/p/ai-text-watermarking-and-quality https://blog.keyvan.net/p/ai-text-watermarking-and-quality
reasonableklout 15d ago
It seems fine to me. The model is still solving my problems and writing code that works as well as any other. Google has been watermarking text with SynthID for a while now and nobody complained about it. Why all the fuss about Claude? It feels like the real reason behind most complaints is that people want to use AI for writing and not have others find out?

exabrial 15d ago

This is hilarious this keeps being repeated by the true believers ad nauseam. Also, don't apply EU law to the world. It's a knee jerk reactionary regulation by a bunch of aging ding dongs that can't print their emails.

frabcus 15d ago
You're on Hacker News - I suggest you have technical curiosity and actually understand this very unusual and innovative algorithm, before you claim things about it that aren't true.