5 ms·

Im the developers who actually got banned because of this dataset. I used NudeNet offline to benchmark my on-device NSFW app Punge — nothing uploaded, nothing s

Im the developers who actually got banned because of this dataset. I used NudeNet offline to benchmark my on-device NSFW app Punge — nothing uploaded, nothing shared. Your dataset wasn’t the problem. The real problem is that independent developers have zero access to the tools needed to detect CSAM, while Big Tech keeps those capabilities to itself. Meanwhile, Google and other giants openly use massive datasets like LAION-5B — which also contained CSAM — without facing any consequences at all. Google even used early LAION data to train one of its own models. Nobody bans Google. But when I touched NudeNet for legitimate testing, Google deleted 130,000+ files from my account, even though only ~700 images out of ~700,000 were actually problematic. That’s not safety — that’s a detection system wildly over firing with no independent oversight and no accountability. Big Tech designed a world where they alone have the scanning tools and the immunity when those tools fail. Everyone else gets punished for their mistakes. So yes — your dataset has done good. ANY data set is subject to this. There needs to be tools and process for all. But let’s be honest about where the harm came from: a system rigged so only Big Tech can safely build or host datasets, while indie developers get wiped out by the exact same automated systems Big Tech exempts itself from.

petee 9mo ago
700 were csam, if I'm reading this right?
- wang_li 9mo ago
  Perhaps these folks should work together to make patches to the dataset to remove the problematic images? E: But also make sure every image in the dataset is properly licensed. This would have eliminated this entirely from the get go. Playing fast and loose with the distribution rights to these images led to this problem.
- markatlarge 9mo ago
  That is right: https://medium.com/@russoatlarge_93541/canadian-child-protection-group-uncovers-abusive-content-in-academic-ai-dataset-9cc13f88701e https://medium.com/@russoatlarge_93541/canadian-child-protec...
- rolph 9mo ago
  700 CSAM images, even one is damning, but hundreds are often referred to as a cache or horde, normally anyone caught with that can wave bye-bye to thier life. google should be fully accountable for possesion and distribution, perhaps even manufacturing.
lynndotpy 9mo ago
Agreed entirely. I want to add some technical details, since this is a peeve I've also had for many years now: The standard for this is Microsoft's PhotoDNA, a paid and gatekept software-as-a-service which maintains a database of "perceptual hashes." (Unlike cryptographic hashes, these are robust against common modifications). It'd be very simple for Microsoft to release a small library which just wraps (1) the perceptual hash algorithm and provides (2) a bloom filter (or newer, similar structures, like an XOR filter) to allow developers to check set membership against it. There are some concerns that an individual perceptual hash can be reversed to a create legible image, so I wouldn't expect or want that hash database to be widely available. But you almost certainly can't do the same with something like a bloom filter. If Microsoft wanted to keep both the hash algorithm and even an XOR filter of the hash database proprietary, that's understandable. But then that's ok too, because we also have mature implementations of zero-knowledge set membership proofs. The only reason I could see is that security-by-obscurity might be a strategy that makes it infeasible for people to find adversarial ways to defeat the proprietary secret-sauce in their perceptual hash algorithm. But I that means giving up opportunities to improve the algorithm, while excluding so many ways it could be useful to combat CSAM.
- Hizonner 9mo ago
  > There are some concerns that an individual perceptual hash can be reversed to a create legible image, Yeah no. Those hashes aren't big enough to encode any real image, and definitely not an image that would actually be either "useful" to yer basic pedo, or recognizable as a particular person. Maybe they could produce something that a diffusion model could refine back into something resembling the original... if the model had already been trained on a ton of similar material. > If Microsoft wanted to keep both the hash algorithm and even an XOR filter of the hash database proprietary That algorithm leaked years ago. Third party code generates exactly the same hashes on the same input. There are open-literature publications on creating collisions (which can be totally innocent images). They have no actual secrets left.
  - lynndotpy 9mo ago
    > > There are some concerns that an individual perceptual hash can be reversed to a create legible image, > Yeah no. Well, kind of. Towards Data Science had an article on it that they've since removed: https://web.archive.org/web/20240219030503/https://towardsdatascience.com/black-box-attacks-on-perceptual-image-hashes-with-gans-cc1be11f277 https://web.archive.org/web/20240219030503/https://towardsda... And this newer paper: https://eprint.iacr.org/2024/1869.pdf https://eprint.iacr.org/2024/1869.pdf They're not very good at all (it just uses a GAN over a recovered bitmask), but it's reasonable for Microsoft to worry that every bit in that hash might be useful. I wouldn't want to distribute all those hashes on a hunch they could never be be used to recover images. I don't think any such thing would be possible, but that's just a hunch. That said, I can't speak on the latter claim without a source. My understanding is that PhotoDNA still has proprietary implementation details that aren't generally available.

petee 9mo ago

700 were csam, if I'm reading this right?

wang_li 9mo ago
Perhaps these folks should work together to make patches to the dataset to remove the problematic images? E: But also make sure every image in the dataset is properly licensed. This would have eliminated this entirely from the get go. Playing fast and loose with the distribution rights to these images led to this problem.
markatlarge 9mo ago
That is right: https://medium.com/@russoatlarge_93541/canadian-child-protection-group-uncovers-abusive-content-in-academic-ai-dataset-9cc13f88701e https://medium.com/@russoatlarge_93541/canadian-child-protec...
rolph 9mo ago
700 CSAM images, even one is damning, but hundreds are often referred to as a cache or horde, normally anyone caught with that can wave bye-bye to thier life. google should be fully accountable for possesion and distribution, perhaps even manufacturing.

lynndotpy 9mo ago

Agreed entirely. I want to add some technical details, since this is a peeve I've also had for many years now: The standard for this is Microsoft's PhotoDNA, a paid and gatekept software-as-a-service which maintains a database of "perceptual hashes." (Unlike cryptographic hashes, these are robust against common modifications). It'd be very simple for Microsoft to release a small library which just wraps (1) the perceptual hash algorithm and provides (2) a bloom filter (or newer, similar structures, like an XOR filter) to allow developers to check set membership against it. There are some concerns that an individual perceptual hash can be reversed to a create legible image, so I wouldn't expect or want that hash database to be widely available. But you almost certainly can't do the same with something like a bloom filter. If Microsoft wanted to keep both the hash algorithm and even an XOR filter of the hash database proprietary, that's understandable. But then that's ok too, because we also have mature implementations of zero-knowledge set membership proofs. The only reason I could see is that security-by-obscurity might be a strategy that makes it infeasible for people to find adversarial ways to defeat the proprietary secret-sauce in their perceptual hash algorithm. But I that means giving up opportunities to improve the algorithm, while excluding so many ways it could be useful to combat CSAM.

Hizonner 9mo ago
> There are some concerns that an individual perceptual hash can be reversed to a create legible image, Yeah no. Those hashes aren't big enough to encode any real image, and definitely not an image that would actually be either "useful" to yer basic pedo, or recognizable as a particular person. Maybe they could produce something that a diffusion model could refine back into something resembling the original... if the model had already been trained on a ton of similar material. > If Microsoft wanted to keep both the hash algorithm and even an XOR filter of the hash database proprietary That algorithm leaked years ago. Third party code generates exactly the same hashes on the same input. There are open-literature publications on creating collisions (which can be totally innocent images). They have no actual secrets left.
- lynndotpy 9mo ago
  > > There are some concerns that an individual perceptual hash can be reversed to a create legible image, > Yeah no. Well, kind of. Towards Data Science had an article on it that they've since removed: https://web.archive.org/web/20240219030503/https://towardsdatascience.com/black-box-attacks-on-perceptual-image-hashes-with-gans-cc1be11f277 https://web.archive.org/web/20240219030503/https://towardsda... And this newer paper: https://eprint.iacr.org/2024/1869.pdf https://eprint.iacr.org/2024/1869.pdf They're not very good at all (it just uses a GAN over a recovered bitmask), but it's reasonable for Microsoft to worry that every bit in that hash might be useful. I wouldn't want to distribute all those hashes on a hunch they could never be be used to recover images. I don't think any such thing would be possible, but that's just a hunch. That said, I can't speak on the latter claim without a source. My understanding is that PhotoDNA still has proprietary implementation details that aren't generally available.