7 ms·

It's a very unintuitive algorithm, and is pretty clever. I recommend reading up on it: https://www.nature.com/articles/s41586-024-08025-4 https://www.nature.co

by frabcus 15d ago

It's a very unintuitive algorithm, and is pretty clever. I recommend reading up on it: https://www.nature.com/articles/s41586-024-08025-4 https://www.nature.com/articles/s41586-024-08025-4 But no, it only ever picks tokens that are in the probability distribution of the last layer, and it might have picked anyway.

throwuxiytayq 15d ago
What if the next token represents a wrong or low-quality answer, but would have only been picked 10% of the time, but now it's picked 20% of the time? Doesn't that obviously decrease the model quality, even though "it might have picked that token anyway"?
- dan-robertson 15d ago
  It would be picked 10% of the time with watermarking. The randomness properties of the PRNG will be very similar to other random number generators, it is just chosen to be vulnerable to a particular cryptanalytic attack (that requires a private key known only to anthropic). I think of it like the Dual_EC_DRGB generator rather than a biased coin.
  - throwuxiytayq 14d ago
    It is absolutely possible that it would not continue to be picked 10% of the time with a given fixed watermark key. The implementation literally labels tokens using a keyed hash and then modifies their scores. The entire point of the watermarking system is to bias certain tokens against others, and - as you would expect - this reportedly results in a reduced response diversity.
    - deleted 13d ago
      [deleted]
- wasabi991011 14d ago
  What if the token represents a high-quality answer, but would have only been picked 80% of the time, but now it's picked 90% of the time? I'm not entirely sure (haven't read the original synthID proposal), but I believe that the re-weighing is set to make both your scenarios and mine equally likely, averaging out to net Zero effect on quality.
dbdr 14d ago
To add nuance, that article does say one of the two versions does reduce text quality (and the other is worse at detectability): > SynthID-Text can be configured to be non-distortionary (preserving text quality) or distortionary (improving watermark detectability at the cost of text quality).

by frabcus 15d ago

throwuxiytayq 15d ago

What if the next token represents a wrong or low-quality answer, but would have only been picked 10% of the time, but now it's picked 20% of the time? Doesn't that obviously decrease the model quality, even though "it might have picked that token anyway"?

dan-robertson 15d ago
It would be picked 10% of the time with watermarking. The randomness properties of the PRNG will be very similar to other random number generators, it is just chosen to be vulnerable to a particular cryptanalytic attack (that requires a private key known only to anthropic). I think of it like the Dual_EC_DRGB generator rather than a biased coin.
- throwuxiytayq 14d ago
  It is absolutely possible that it would not continue to be picked 10% of the time with a given fixed watermark key. The implementation literally labels tokens using a keyed hash and then modifies their scores. The entire point of the watermarking system is to bias certain tokens against others, and - as you would expect - this reportedly results in a reduced response diversity.
  - deleted 13d ago
    [deleted]
wasabi991011 14d ago
What if the token represents a high-quality answer, but would have only been picked 80% of the time, but now it's picked 90% of the time? I'm not entirely sure (haven't read the original synthID proposal), but I believe that the re-weighing is set to make both your scenarios and mine equally likely, averaging out to net Zero effect on quality.

dbdr 14d ago

To add nuance, that article does say one of the two versions does reduce text quality (and the other is worse at detectability): > SynthID-Text can be configured to be non-distortionary (preserving text quality) or distortionary (improving watermark detectability at the cost of text quality).