5 ms·

It literally re-weights the output tokens from what the LLM would otherwise have chosen. It _has_ to. It can't be positive, because then that's not watermarking

by pkulak 15d ago

It literally re-weights the output tokens from what the LLM would otherwise have chosen. It _has_ to. It can't be positive, because then that's not watermarking, it's a better LLM.

frabcus 15d ago
It's a very unintuitive algorithm, and is pretty clever. I recommend reading up on it: https://www.nature.com/articles/s41586-024-08025-4 https://www.nature.com/articles/s41586-024-08025-4 But no, it only ever picks tokens that are in the probability distribution of the last layer, and it might have picked anyway.
- throwuxiytayq 15d ago
  What if the next token represents a wrong or low-quality answer, but would have only been picked 10% of the time, but now it's picked 20% of the time? Doesn't that obviously decrease the model quality, even though "it might have picked that token anyway"?
  - dan-robertson 15d ago
    It would be picked 10% of the time with watermarking. The randomness properties of the PRNG will be very similar to other random number generators, it is just chosen to be vulnerable to a particular cryptanalytic attack (that requires a private key known only to anthropic). I think of it like the Dual_EC_DRGB generator rather than a biased coin.
    - throwuxiytayq 14d ago
      It is absolutely possible that it would not continue to be picked 10% of the time with a given fixed watermark key. The implementation literally labels tokens using a keyed hash and then modifies their scores. The entire point of the watermarking system is to bias certain tokens against others, and - as you would expect - this reportedly results in a reduced response diversity.
      - deleted 14d ago
        [deleted]
  - wasabi991011 14d ago
    What if the token represents a high-quality answer, but would have only been picked 80% of the time, but now it's picked 90% of the time? I'm not entirely sure (haven't read the original synthID proposal), but I believe that the re-weighing is set to make both your scenarios and mine equally likely, averaging out to net Zero effect on quality.
- dbdr 15d ago
  To add nuance, that article does say one of the two versions does reduce text quality (and the other is worse at detectability): > SynthID-Text can be configured to be non-distortionary (preserving text quality) or distortionary (improving watermark detectability at the cost of text quality).
qgin 15d ago
Unless you’re running at temperature 0, there’s not one single token that the model definitely would have chosen each time.
brokencode 15d ago
Unless you’re at 0 temperature, there is no single token it would have chosen. It’s always picking one of multiple randomly according to a probability distribution.
- northzen 15d ago
  Give me an example how would you watermark a single short sentence like "I like turtles"?
  - qbit42 15d ago
    Watermarking just alters the pseudorandom number generator. If "I like turtles" was previously the response to your prompt with probability 100%, it will still be so. This is why watermarking is only effective for long strings of text
    - northzen 15d ago
      It's like the sudden change of a language style and its verbosity didn't happen recently. To random words you pick and provide a sufficient amount of text to vary with random number without losing its meaning you need a text with high entropy.
      - qlte 14d ago
        Nothing about watermarking would require padding the response length with pseudo-intelligible Claudese. Regular filler would work fine. Also, it would probably provide higher entropy to write normal human-sounding English instead of reusing a repetitive grab bag of load-bearing phrases. This theory doesn't really make any sense.
WithinReason 15d ago
No, it basically uses a fixed seed for the random number generator to generate tokens. The generated random sequence is just as random as it would be with a random seed.

by pkulak 15d ago

It literally re-weights the output tokens from what the LLM would otherwise have chosen. It _has_ to. It can't be positive, because then that's not watermarking, it's a better LLM.

frabcus 15d ago

It's a very unintuitive algorithm, and is pretty clever. I recommend reading up on it: https://www.nature.com/articles/s41586-024-08025-4 https://www.nature.com/articles/s41586-024-08025-4 But no, it only ever picks tokens that are in the probability distribution of the last layer, and it might have picked anyway.

throwuxiytayq 15d ago
What if the next token represents a wrong or low-quality answer, but would have only been picked 10% of the time, but now it's picked 20% of the time? Doesn't that obviously decrease the model quality, even though "it might have picked that token anyway"?
- dan-robertson 15d ago
  It would be picked 10% of the time with watermarking. The randomness properties of the PRNG will be very similar to other random number generators, it is just chosen to be vulnerable to a particular cryptanalytic attack (that requires a private key known only to anthropic). I think of it like the Dual_EC_DRGB generator rather than a biased coin.
  - throwuxiytayq 14d ago
    It is absolutely possible that it would not continue to be picked 10% of the time with a given fixed watermark key. The implementation literally labels tokens using a keyed hash and then modifies their scores. The entire point of the watermarking system is to bias certain tokens against others, and - as you would expect - this reportedly results in a reduced response diversity.
    - deleted 14d ago
      [deleted]
- wasabi991011 14d ago
  What if the token represents a high-quality answer, but would have only been picked 80% of the time, but now it's picked 90% of the time? I'm not entirely sure (haven't read the original synthID proposal), but I believe that the re-weighing is set to make both your scenarios and mine equally likely, averaging out to net Zero effect on quality.
dbdr 15d ago
To add nuance, that article does say one of the two versions does reduce text quality (and the other is worse at detectability): > SynthID-Text can be configured to be non-distortionary (preserving text quality) or distortionary (improving watermark detectability at the cost of text quality).

qgin 15d ago

Unless you’re running at temperature 0, there’s not one single token that the model definitely would have chosen each time.

brokencode 15d ago

Unless you’re at 0 temperature, there is no single token it would have chosen. It’s always picking one of multiple randomly according to a probability distribution.

northzen 15d ago
Give me an example how would you watermark a single short sentence like "I like turtles"?
- qbit42 15d ago
  Watermarking just alters the pseudorandom number generator. If "I like turtles" was previously the response to your prompt with probability 100%, it will still be so. This is why watermarking is only effective for long strings of text
  - northzen 15d ago
    It's like the sudden change of a language style and its verbosity didn't happen recently. To random words you pick and provide a sufficient amount of text to vary with random number without losing its meaning you need a text with high entropy.
    - qlte 14d ago
      Nothing about watermarking would require padding the response length with pseudo-intelligible Claudese. Regular filler would work fine. Also, it would probably provide higher entropy to write normal human-sounding English instead of reusing a repetitive grab bag of load-bearing phrases. This theory doesn't really make any sense.

WithinReason 15d ago

No, it basically uses a fixed seed for the random number generator to generate tokens. The generated random sequence is just as random as it would be with a random seed.