7 ms·

Aside from the fact that this was obviously never viable and the entire problem is clearly unsolvable if you sit down and really probe it for fifteen minutes, w

Aside from the fact that this was obviously never viable and the entire problem is clearly unsolvable if you sit down and really probe it for fifteen minutes, what I find most frustrating about this is that the false promise of preserving photos as reliable evidence is actively harmful. You will not build a perfect system, or even something near perfect. The best you're going to do is make it so that it's hard to casually present AI photos as real, leaving only the cases where it really matters. In the "best" case, you've just made the public more trusting of photos in general, so that when there's actual money or power on the line that makes jumping through the hoops to fake authenticity worth it, the public is more susceptible. The best outcome at this point is for everyone to get on the same page that photos have roughly the same probative value now as drawings. Poorly thought out snake oil efforts to prove authenticity are only going to delay that.

qurren 22d ago
Are we entering a world where if I took a picture with a film camera and scanned it, it would be rejected as not real? This is ridiculous.
- CapitalistCartr 22d ago
  It's not a matter of "rejected as not real", it's "There's no way to know if this is real or not".
  - account42 21d ago
    When 99% of people are posting pics from their big tech approved smartphones with the "200% real no way to fake this" badge it doesn't matter that absence of the badge technically doesn't mean fake, that's what it will be interpreted as.
- lelandfe 22d ago
  The defendant submitted a remarkably high quality video of you saying you generated it with Nano Banana.
  - ted_dunning 22d ago
    And their video was C2PA signed.
- spaqin 22d ago
  You can even do it wholly in an analogue process, do a darkroom print, and it could be seen as fake as well - darkroom techniques are quite flexible as well. And a negative itself? That could be a copy of a prepared printed slide.
indutny 22d ago
In my opinion, the benefits for end users are rather minimal since I doubt an average person would ever be checking C2PA provenance data, but there is a commercial incentive for Google and others to promote C2PA, since it makes preparing training material for Machine Learning significantly easier, and perhaps as a smaller benefit justifies hardware attestation that locks users down into proprietary OSes.
- chrisjj 22d ago
  > it makes preparing training material for Machine Learning significantly easier How?
  - lazide 22d ago
    A huge problem with ML training is the ‘ouroboros issue’ - training ML with input from other MLs breaks things in very deep (but difficult to stop/detect when it’s happening) ways due to the way the internal math works (model collapse). Right now, the Internet training set is becoming more and more contaminated with better and better generative AI images and video. It makes the models more screwed up, and makes it very difficult for humans to figure out what is original and not too. If there was some signal that could at least make it easier to identify ‘original’/real images…
    - ainch 22d ago
      "Model collapse" is often overstated, as this paper demonstrates: https://arxiv.org/pdf/2404.01413 https://arxiv.org/pdf/2404.01413 The original model collapse paper assumes you train networks on 100% synthetic data produced by the previous generation. But if you maintain some portion of real data then the problem is mitigated.
      - lazide 22d ago
        Not when you also including poisoning attacks. I remember the original paper showing issues with even a couple percent of certain kinds of synthetic data too, not 100%.
fwipsy 22d ago
I think it's useful even if it can be spoofed. Many people don't even bother to edit visible watermarks out of AI photos/videos. I'm fairly certain this will defeat 99.9% of malicious users, many of whom won't even know it exists until someone points out it's missing. People are concerned that the technology will lend additional credence to the last 0.1%. But anyone who thinks about the technology for 2 minutes will see you can just point the camera at the screen. In cases where it really matters (a court of law, internet arguments between nerds) people will know it's not 100% reliable. Locks can be picked, and signatures can be forged, but that doesn't make them useless. "C2PA Cameras Do Not Survive Contact With Reality" does not survive contact with reality where very, very few users would even think of rooting their phone so they can create signed fake images.
- Retr0id 22d ago
  When I search for "C2PA" on the google play store, there are more AI-watermark-removal apps than there are signing apps. Certain types will jump through ridiculous hoops if they think it will affect their algorithmic reach on social media. Malicious users don't need to root their own phones. They just need to go to fakemyimage dot com, and someone else's rooted phone in a clickfarm-type setup signs it for them. I am not operating such a service myself because I thought it was unnecessary in making my point, but perhaps I will have to reconsider.
  - fwipsy 22d ago
    You're arguing that lots of people can spoof this, the other guy is arguing that nobody will know it can be spoofed so it will do more damage. But these are contradictory -- if fakes become common, then they will also become common knowledge. The impact of any given fake is reduced if there are more of them. The technology doesn't need to provide 100% assurance. If it adds even a little friction to the slop mills then that's increasing the signal to noise ratio.
- treyd 22d ago
  It's actually worse if it is plausibly trustworthy for "99.9%", since that's enough that naive users will get accustomed to believing the verification badge is authentic. When a motivated malicious user (who doesn't actually need that much resources) will be able to convince people something is authentic because the verification passes when it shouldn't since naive users are primed to believe it by default.
Gigachad 22d ago
I don’t think there is a technical solution to this problem but I think there is a legal one. Make it a legal requirement to mark AI generated photos and enforce penalties for posting unmarked AI generations. Social media should also mark the country of origin for each post, with the knowledge that posts from your own country are covered by these laws.
- inigyou 22d ago
  The EU did this, Claude and Microsoft complied and HN was practically rioting about it literally yesterday: https://news.ycombinator.com/item?id=49421158 https://news.ycombinator.com/item?id=49421158
  - tancop 22d ago
    No, this would be literally you (the user) checking "this is AI generated" before you post something publicly. Platforms can auto mark it if they detect a watermark like SynthID but they can also leave the choice up to you. The point is making it easier to go after bot accounts that spam generated videos to influence politics. They need to disclose that its AI and lose their power or lie and get criminally prosecuted. It's not the same thing as model providers adding a mandatory watermark to everything, and even worse one you can't verify yourself and have to trust that Anthropic is telling everyone the truth. People should have the option to use undetectable AI tools in private, even if it's illegal to post the outputs on social media.
- akoboldfrying 22d ago
  I think public key cryptography offers a technical solution that is nearly as ideal here as for its existing uses for securing communication between physically remote actors -- please see my comment here for details: https://news.ycombinator.com/item?id=49444227 https://news.ycombinator.com/item?id=49444227 I say "nearly", because as soon as you need to keep a private key secure from someone with direct physical access to the device, you're entering dangerous territory. TTBOMK there's no way to make a "perfect black box", so it becomes an arms race between defensive "obfuscation" and tamper detection mechanisms in the one hand and stealth scanning techniques on the other. But this is already the case for TPMs -- that is, the situation is no worse than for an already widely accepted technology.
akoboldfrying 22d ago
What makes you so certain that this valuable research showing weaknesses in today's systems will render the C2PA concept useless forever? Yes, software LPEs are a risk -- as they are in every nontrivial computer system. New ones will appear, and old ones will be closed in time, as TFA acknowledges. Re hardware attacks: The (neat!) glitch injection attack the author describes in the linked "lighter" page only raises the implementation cost of doing image certification properly. For example, if the camera module presented only an interface that dumped raw RGB or JPEG-encoded data plus a digital signature that used a private key known only to the manufacturer, then all that would be required to verify a "downstream" image would be to keep a copy of those original bytes inside the final (potentially cropped, filtered, AI-ed, etc.) image, in the worst case roughly doubling its size on disk (though certainly more efficient schemes could be designed). Any interested third party could then compare the original and final images by eye and decide for themselves whether or not the subsequent processing materially changed the image's "meaning". Finally: Does the existence of lock picks or bolt cutters render padlocks pointless today? Does it corrode society by encouraging people to mistakenly believe that anything they put behind a $5 padlock will be safe forever? No, and no.
- hypfer 22d ago
  This is yet another of these absolutely caustic takes that come with a veneer of intellectualism, but actually just ignorantly deconstruct reality. Each of them weaponizing the rules of the platform that (for sensible reasons) demand you engage with the strongest interpretation of the message/argument you see. The asymmetry of effort there is unsustainable, and that's exactly the point. No idea how that could be solved. Maybe a meta comment like this one helps. __ I mean if you think about it, it shouldn't be possible for some anon account to drop this and sound like it's a worthy contribution to a debate against some real person with a real name, a track record and multiple thousand dollars of bricked hardware leading up to that assessment. (Nor would it make sense for a non-anon but equally empty account) It makes no sense, and the guarantees regarding protection of speech and all do not apply to these topics, because it's not an opinion that would get your real name in jail. What can we do about these social exploits. Someone tell me please. It's driving me up the walls
richdrich 22d ago
People got on fine until 200 years ago with the only means of rendering a picture being to draw it, as in Hogarth's calumnous image of the in fact really quite civilised Gin Lane (https://www.theguardian.com/artanddesign/picture/2012/sep/12/william-hogarth-gin-lane https://www.theguardian.com/artanddesign/picture/2012/sep/12...)
- account42 21d ago
  People didn't have the ability to distribute forged images to millions to form a mob in seconds though.
adabovehuman 22d ago
> and the entire problem is clearly unsolvable The problem is not identifying AI generated media The problem is identifying real media. That it can actually do.
account42 22d ago
Worse than not working, this will likely be used as yet another excuse to attack computing freedom.

qurren 22d ago

Are we entering a world where if I took a picture with a film camera and scanned it, it would be rejected as not real? This is ridiculous.

CapitalistCartr 22d ago
It's not a matter of "rejected as not real", it's "There's no way to know if this is real or not".
- account42 21d ago
  When 99% of people are posting pics from their big tech approved smartphones with the "200% real no way to fake this" badge it doesn't matter that absence of the badge technically doesn't mean fake, that's what it will be interpreted as.
lelandfe 22d ago
The defendant submitted a remarkably high quality video of you saying you generated it with Nano Banana.
- ted_dunning 22d ago
  And their video was C2PA signed.
spaqin 22d ago
You can even do it wholly in an analogue process, do a darkroom print, and it could be seen as fake as well - darkroom techniques are quite flexible as well. And a negative itself? That could be a copy of a prepared printed slide.

indutny 22d ago

In my opinion, the benefits for end users are rather minimal since I doubt an average person would ever be checking C2PA provenance data, but there is a commercial incentive for Google and others to promote C2PA, since it makes preparing training material for Machine Learning significantly easier, and perhaps as a smaller benefit justifies hardware attestation that locks users down into proprietary OSes.

chrisjj 22d ago
> it makes preparing training material for Machine Learning significantly easier How?
- lazide 22d ago
  A huge problem with ML training is the ‘ouroboros issue’ - training ML with input from other MLs breaks things in very deep (but difficult to stop/detect when it’s happening) ways due to the way the internal math works (model collapse). Right now, the Internet training set is becoming more and more contaminated with better and better generative AI images and video. It makes the models more screwed up, and makes it very difficult for humans to figure out what is original and not too. If there was some signal that could at least make it easier to identify ‘original’/real images…
  - ainch 22d ago
    "Model collapse" is often overstated, as this paper demonstrates: https://arxiv.org/pdf/2404.01413 https://arxiv.org/pdf/2404.01413 The original model collapse paper assumes you train networks on 100% synthetic data produced by the previous generation. But if you maintain some portion of real data then the problem is mitigated.
    - lazide 22d ago
      Not when you also including poisoning attacks. I remember the original paper showing issues with even a couple percent of certain kinds of synthetic data too, not 100%.

fwipsy 22d ago

I think it's useful even if it can be spoofed. Many people don't even bother to edit visible watermarks out of AI photos/videos. I'm fairly certain this will defeat 99.9% of malicious users, many of whom won't even know it exists until someone points out it's missing. People are concerned that the technology will lend additional credence to the last 0.1%. But anyone who thinks about the technology for 2 minutes will see you can just point the camera at the screen. In cases where it really matters (a court of law, internet arguments between nerds) people will know it's not 100% reliable. Locks can be picked, and signatures can be forged, but that doesn't make them useless. "C2PA Cameras Do Not Survive Contact With Reality" does not survive contact with reality where very, very few users would even think of rooting their phone so they can create signed fake images.

Retr0id 22d ago
When I search for "C2PA" on the google play store, there are more AI-watermark-removal apps than there are signing apps. Certain types will jump through ridiculous hoops if they think it will affect their algorithmic reach on social media. Malicious users don't need to root their own phones. They just need to go to fakemyimage dot com, and someone else's rooted phone in a clickfarm-type setup signs it for them. I am not operating such a service myself because I thought it was unnecessary in making my point, but perhaps I will have to reconsider.
- fwipsy 22d ago
  You're arguing that lots of people can spoof this, the other guy is arguing that nobody will know it can be spoofed so it will do more damage. But these are contradictory -- if fakes become common, then they will also become common knowledge. The impact of any given fake is reduced if there are more of them. The technology doesn't need to provide 100% assurance. If it adds even a little friction to the slop mills then that's increasing the signal to noise ratio.
treyd 22d ago
It's actually worse if it is plausibly trustworthy for "99.9%", since that's enough that naive users will get accustomed to believing the verification badge is authentic. When a motivated malicious user (who doesn't actually need that much resources) will be able to convince people something is authentic because the verification passes when it shouldn't since naive users are primed to believe it by default.

Gigachad 22d ago

I don’t think there is a technical solution to this problem but I think there is a legal one. Make it a legal requirement to mark AI generated photos and enforce penalties for posting unmarked AI generations. Social media should also mark the country of origin for each post, with the knowledge that posts from your own country are covered by these laws.

inigyou 22d ago
The EU did this, Claude and Microsoft complied and HN was practically rioting about it literally yesterday: https://news.ycombinator.com/item?id=49421158 https://news.ycombinator.com/item?id=49421158
- tancop 22d ago
  No, this would be literally you (the user) checking "this is AI generated" before you post something publicly. Platforms can auto mark it if they detect a watermark like SynthID but they can also leave the choice up to you. The point is making it easier to go after bot accounts that spam generated videos to influence politics. They need to disclose that its AI and lose their power or lie and get criminally prosecuted. It's not the same thing as model providers adding a mandatory watermark to everything, and even worse one you can't verify yourself and have to trust that Anthropic is telling everyone the truth. People should have the option to use undetectable AI tools in private, even if it's illegal to post the outputs on social media.
akoboldfrying 22d ago
I think public key cryptography offers a technical solution that is nearly as ideal here as for its existing uses for securing communication between physically remote actors -- please see my comment here for details: https://news.ycombinator.com/item?id=49444227 https://news.ycombinator.com/item?id=49444227 I say "nearly", because as soon as you need to keep a private key secure from someone with direct physical access to the device, you're entering dangerous territory. TTBOMK there's no way to make a "perfect black box", so it becomes an arms race between defensive "obfuscation" and tamper detection mechanisms in the one hand and stealth scanning techniques on the other. But this is already the case for TPMs -- that is, the situation is no worse than for an already widely accepted technology.

akoboldfrying 22d ago

What makes you so certain that this valuable research showing weaknesses in today's systems will render the C2PA concept useless forever? Yes, software LPEs are a risk -- as they are in every nontrivial computer system. New ones will appear, and old ones will be closed in time, as TFA acknowledges. Re hardware attacks: The (neat!) glitch injection attack the author describes in the linked "lighter" page only raises the implementation cost of doing image certification properly. For example, if the camera module presented only an interface that dumped raw RGB or JPEG-encoded data plus a digital signature that used a private key known only to the manufacturer, then all that would be required to verify a "downstream" image would be to keep a copy of those original bytes inside the final (potentially cropped, filtered, AI-ed, etc.) image, in the worst case roughly doubling its size on disk (though certainly more efficient schemes could be designed). Any interested third party could then compare the original and final images by eye and decide for themselves whether or not the subsequent processing materially changed the image's "meaning". Finally: Does the existence of lock picks or bolt cutters render padlocks pointless today? Does it corrode society by encouraging people to mistakenly believe that anything they put behind a $5 padlock will be safe forever? No, and no.

hypfer 22d ago
This is yet another of these absolutely caustic takes that come with a veneer of intellectualism, but actually just ignorantly deconstruct reality. Each of them weaponizing the rules of the platform that (for sensible reasons) demand you engage with the strongest interpretation of the message/argument you see. The asymmetry of effort there is unsustainable, and that's exactly the point. No idea how that could be solved. Maybe a meta comment like this one helps. __ I mean if you think about it, it shouldn't be possible for some anon account to drop this and sound like it's a worthy contribution to a debate against some real person with a real name, a track record and multiple thousand dollars of bricked hardware leading up to that assessment. (Nor would it make sense for a non-anon but equally empty account) It makes no sense, and the guarantees regarding protection of speech and all do not apply to these topics, because it's not an opinion that would get your real name in jail. What can we do about these social exploits. Someone tell me please. It's driving me up the walls

richdrich 22d ago

People got on fine until 200 years ago with the only means of rendering a picture being to draw it, as in Hogarth's calumnous image of the in fact really quite civilised Gin Lane (https://www.theguardian.com/artanddesign/picture/2012/sep/12/william-hogarth-gin-lane https://www.theguardian.com/artanddesign/picture/2012/sep/12...)

account42 21d ago
People didn't have the ability to distribute forged images to millions to form a mob in seconds though.

adabovehuman 22d ago

> and the entire problem is clearly unsolvable The problem is not identifying AI generated media The problem is identifying real media. That it can actually do.

account42 22d ago

Worse than not working, this will likely be used as yet another excuse to attack computing freedom.