10 ms·

A catalog of naturally occurring images whose Apple NeuralHash is identical

337 points by hongsy 5y ago

toxik 5y ago
Sigh, for the last time, it doesn't actually matter if the NeuralHash is identical. You need multiple images matching, and then the images are compared by another system on Apple's end, which you don't know anything about. The system is specifically designed so that colliding images does not pose a threat to the user. NeuralHash and the CSAM scanning is grotesque, but please, criticize it for what it is, not some bullshit that is easily dismissed as technical ignorance.
- collaborative 5y ago
  We still need to rely on the automated "secret backend system" that nobody supposedly knows anything about
  - toxik 5y ago
    You (or at least Apple's customers) trust in and rely on Apple's proprietary software to do its job all the time. How is this different? I find this argument very weak.
    - Hackbraten 5y ago
      1. You can at least somewhat audit the software running on an iPhone, for example by means of reverse engineering. You can’t audit the server side. 2. It’s one thing to rely on proprietary services like Find My or Siri. It’s another thing to rely on a secret server-side app that has the power to destroy your life.
      - idunnoman 5y ago
        "Hey Siri, accuse me of something that will ruin my life and reputation even if I'm not guilty".
      - fetzu 5y ago
        What I somehow fail to grasp in the first argument is that this whole system is designed specifically so that it runs client-side. AFAIK all the alternatives (as in « cloud photo services ») have been doing the exact same thing on the server side for decades. If you upload your photos to the cloud, a lot of service actually already have the power to destroy your life.
        collaborative 5y ago
        This is all about Apple users waking up to the fact that they've been had
    - strangetortoise 5y ago
      afaik none of Apples other "proprietary software" is designed to pass my personal images to a human for visual inspection if it mistakenly outputs 2 high-enough numbers after a handful of convolution operations and matrix multiplications.
      - FabHK 5y ago
        They pass a "visual derivative" to "a human", but only after some matrix multiplications etc. that result in extremely low probability false positives. It could also happen that you lose your phone and "a human" finds it and randomly puts in the correct passcode on the first try and visually inspects your personal images. In fact, that seems vastly more likely [1]. [1] About 4% of smartphones are lost or stolen every year [https://www.mcafee.com/blogs/consumer/family-safety/almost-5-of-smartphones-lost-every-year/ https://www.mcafee.com/blogs/consumer/family-safety/almost-5... ], but make it just 1/1000, so 1e-3. Then a 6 digit passcode, 1e-6, so we're at 1e-9 per year, or 1000x as likely as being falsely flagged, assuming Apple's numbers (which can easily be achieved by calibrating the threshold).
    - collaborative 5y ago
      Tell this to the victims of Pegasus. If anyone were able to get their hands on the "secret backend system" we wouldn't be talking about spy games, we'd be talking about people's lives being ruined
      - roody15 5y ago
        Apple still has not patched the security exploit in iMessage used by Pegasus. Apple has released two ios security updates since the Pegasus revelations but still has not patched it most widely used exploit… hmmmm. Now apple is getting a local client side scanning tool ready. Interesting timing.
        collaborative 5y ago
        I don't think what you are saying is far-fetched
    - read_if_gay_ 5y ago
      No. I trusted Apple’s proprietary software before this, because they maintained that they care about privacy, and there have been examples of that. Now, I don’t trust them anymore.
    - simion314 5y ago
      There is a big difference, if say Apple tracking of what you run is problematic you get a bad user experience like apps starting after 1 minute of waiting, or if Apple App Store contains malware you will probably get some annoying issue while Apple will try to silently cleanup their mess BUT with this system there is a difference, this is designed not to serve you or protect you, it is designed to do some checks then if some specific rules match send some guys on you to destroy your life, today is FBI tomorrow other authoritarians. Issues in any other Apple software will not send the police on you. Why would you install a software on your desktop/laptop that is designed to snitch on you, you would need to get some advantage or be forced by some law. For now I see only disadvantages but please let me know of any real advantage and not speculation Disadvantages: - closed software with hidden db can't be trusted, so as a user you will always have a doubt that some non CP images are in the db(Apple always collaborates with governments) - bugs in this stuff will cause you big problems(we seen in the past how false accusation destroyed peoples life) and we also seen bad actors abusing this kind of stuff. - this is also clearly a beginning, now that Apple has the capability then even if they were saints a judge could force them to add new hashes, change the configs etc.
- tsimionescu 5y ago
  Then let's get rid of the NeuralHash entirely, if it doesn't matter, right? If it's a critical part of the system, then it should be inspected thoroughly. If Apple claims a minuscule chance of a hash collision, and the reality is that collisions are relatively common, that significantly changes the requirements for the backend system, which Apple keeps secret. We have every right to believe, bbased oon ppublic info, that Apple was expecting that NeuralHash would be almost fool-proof, leaving the backend system to be a rubber stamp. This would be tragic.
  - toxik 5y ago
    The point in the NeuralHash and PSI system is to preserve user privacy as far as possible. From a technical standpoint, it is not essential - a NeuralHash function that returns 0x0000… for everything would still catch CSAM. It's just that it would upload every single image on the user's device. Now, how well this NeuralHash does preserve privacy is a different question, and /not/ one that is being answered by the original post here. In fact, I've not seen anybody look at the hash distribution over natural images, which would be an actual argument against the system.
    - Retr0id 5y ago
      It doesn't really matter whether all images are uploaded, or just 1 in x (for large value of x), due to the Panopticon effect.
      - ithkuil 5y ago
        Let's not forget what the alternative is: this is about images that are uploaded on icloud anyway. The alternative is to upload the image in clear (or with ane encryption key that apple controls), and let apple run the CSAM filter on their servers. Apple now has the ability to encrypt the images before sending them to icloud, with a private key you own. Except that some percentage of images that match the CSAM fingerprint with their neural feature extractor will be sent to a CSAM filter on the server side (whose workings we don't have many details about) This whole thing backfired on Apple entirely due to psychological effects, not because they are really doing anything more "panopticon" that they would already able to do now on their icloud storage (after all people are ready sending their photos to apple)
- draw_down 5y ago
  Come on.
- saithound 5y ago
  Discussing the preimage attack on NeuralHash is not technical ignorance. Dismissing the preimage attack as irrelevant is. 0. Most importantly: the existence of a preimage attack makes Apple's system completely useless for its original purpose. The NeuralHash collider allows the producers and distributors of CSAM material to ensure that nearly all of the next generation of CSAM will suffer from hash collisions with perfectly innocent images. Two weeks after it was deployed, Apple's CSAM scanning is now _only_ an attack vector and a privacy risk. Thanks to the preimage attack, it's now completely useless for its nominal function! Apple put a lot of effort into a system that reduced the privacy and security of all their customers, and made the company itself more exposed to the whims of governments. And for no gain whatsoever. 1. There are no known perceptual hash functions on which preimage attacks are difficult. Barring a major "secret cryptographic breakthrough", Apple's second hash function is not resistant to preimage attacks either. In fact, the second algorithm is almost certainly easier to attack than NeuralHash itself, since it has to work on the "visual derivative", a fixed-size low-resolution thumbnail of the original image. 2. But isn't Apple's second algorithm kept secret, making it difficult to perform preimage attacks against it? No. First of all, the second algorithm cannot be kept secret. Apple doesn't have its own CSAM database (the whole point is that they don't want to deal with CSAM on their servers!), so the algorithm has to be shared with multiple organizations which do have such databases, so that they can pre-compute the hashes that Apple will match against. Due to Apple's policy, some of these organizations will be located outside the US [1]. Chances are, the hash function will leak: Apple won't know if and when that happens. Secondly, this _is_ security by obscurity. Some people argue that keeping the hash algorithm secret is similar to keeping a cryptographic key secret. This is not the case. Of course, any security system relies on keeping _something_ secret, but these secret somethings are not created equal. The secret keys of cryptographic algorithms are designed to satisfy Kerckhoffs's assumption. This means that the key, as long as it remains secret, should be sufficient to protect the confidentiality and integrity of your system, even if your adversary knows everything else apart from the key, including the details of the algorithm you use, the hardware you have, and even all your previous plaintexts and ciphertexts (inputs and outputs). The second hash does not have this property at all. Keeping the algorithm secret does not ensure the confidentiality or integrity of Apple's system. E.g. if somebody gets access to a reasonable number of inputs-output examples, that allows them to train their own model which behaves similarly enough to let them find perceptual hash collisions, even if they don't know the exact details of the original algorithm. This is incredibly hard for cryptographic hashes, but very easy for perceptual hashes, since a small change in the input should cause only a small change in the output of the perceptual hash algorithm. So, to maintain security, Apple doesn't have to keep just the hash algorithm (or its configuration parameters) secret, but all the inputs and outputs as well. This is bad: the fewer and simpler the secrets that one must keep to ensure system security, the easier it is to maintain system security. Finally, the second hash algorithm is unlikely to be original (NeuralHash was original, and by all accounts it was a massive effort). If an attacker successfully guesses that Apple's secret algorithm H is closely related to a known algorithm, say PhotoDNA, they will probably be able to make a transfer attack against it. By engineering a PhotoDNA collision on the resized thumbnail (e.g. via a resizing attack, extensively discussed in a previous thread [3]), they have a reasonable chance of generating a H-collision as well. How good is fairly good? Well, something like 5% is more than enough! The attacker needs to produce a certain number of NeuralHash collisions (say 30 images) to get through the first threshold of Apple's algorithm. But after that, Apple will decode all the thumbnails in the user's safety voucher: the attacker only needs one of those 30 to get through the second hash. Given a sufficiently high probability of hash collisions, this can be achieved "blindly". 3. It's incredibly easy to come up with these kinds of attacks. Even the HN audience could come up with several reasonable plans, and could point out several reasonable issues, in two weeks. People who do malice for a living will have a much easier time with it. Even if somehow all the plans presented on HN turned out to be unviable, it will not take long for someone to stumble upon something practical. Any reassurance that Apple could provide at this point is fake. Cf. the timelines for real security: it took 17 years to come up with an analogous attack against SHA-1 [4], and two years after that to turn it into something that can be exploited in practice [5]. The existence of a preimage attack made Apple's system completely useless for its original purpose in two weeks. It's now just a security and privacy hole, with no other function. Keeping it around would be a travesty, even if it was difficult to exploit. But it's not. [1] https://www.itnews.com.au/news/apple-to-only-seek-abuse-images-flagged-in-multiple-nations-568625 https://www.itnews.com.au/news/apple-to-only-seek-abuse-imag... [2] https://en.wikipedia.org/wiki/Kerckhoffs%27s_principle https://en.wikipedia.org/wiki/Kerckhoffs%27s_principle [3] https://news.ycombinator.com/item?id=28236102 https://news.ycombinator.com/item?id=28236102 [4] https://security.googleblog.com/2017/02/announcing-first-sha1-collision.html https://security.googleblog.com/2017/02/announcing-first-sha... [5] https://www.zdnet.com/article/sha-1-collision-attacks-are-now-actually-practical-and-a-looming-danger/ https://www.zdnet.com/article/sha-1-collision-attacks-are-no...
- nullc 5y ago
  (A threshold of) matchings result in the private keys for the images being leaked to Apple, where they're vulnerable to: (1) Review by apple staff (2) Access and leaking by other apple staff (3) Access by hackers who have compromised their system (4) Access by parties coercing apple/staff, including via national security letters. All of which compromise the privacy of the user. This matters or the neuralhash comparison wouldn't exist in the first place. Totally agree that the whole system is grotesque-- but that doesn't stop it also being grotesque in every detail as well. The fact that there are false positives when they easily could have designed a system that had none (at the expense of increased false negatives) shows that Apple doesn't especially value customer privacy even if you accept their vigilante privacy invasion. The fact that it's possible to construct adversarial false positives and that their reports didn't disclose this fact shows they either don't know what they're doing or they're not being honest about the risks (or both).
hhsbz 5y ago
Is it really a catalogue when there only are two of them? I find it amusing that they probably ran this tool against a set of millions or even billions of images and this is the best they could come up with. They are practically praising Apple here lmao
valparaiso 5y ago
Lmao no one knows how works Apple production code/algorithm but already did assumptions.
yosito 5y ago
> a catalog Can two collisions really be called a catalog?
- yeldarb 5y ago
  PRs welcome!
- eesmith 5y ago
  A catalog of a thousand pages begins with the first entry.
  - nannal 5y ago
    And a story may start with the first word, but if I present the word "Octopus" and say check out my story, you're going to be well within bounds to question me on it.
    - simondotau 5y ago
      Well to be fair there were two collisions, so your story should be "Octopus imploded" and now everyone is captivated by the plight of this unlucky cephalopod.
    - eesmith 5y ago
      Wikipedia started as just an announcement of the idea (https://web.archive.org/web/20030414014355/http://www.nupedia.com/pipermail/nupedia-l/2001-January/000676.html https://web.archive.org/web/20030414014355/http://www.nupedi... ). How many entries do you think there were when the first live version was announced only a few hours later (http://www.nupedia.com/pipermail/nupedia-l/2001-January/000678.html http://www.nupedia.com/pipermail/nupedia-l/2001-January/0006... )? As a different metaphor than your "Octopus", this is "first light". "First light" in astronomy is the first time a telescope is used. It doesn't need to start with an amazing or ground-breaking image.
- dathinab 5y ago
  It's a WIP catalog where everyone who stumbles over one can put it in. It could in the future be used to e.g. improves this algorithms.
theshadowknows 5y ago
I’m glad that people are trying to figure out any technical flaws in the system as best they can, but if I’m being honest I do trust Apple’s engineers to have built something that is solid from a technical stand point. Am I correct in that the primary reason folks are so upset is that the system could (probably) be easily modified such that -any- content could invoke legal action? That the main problem is really the scanning at all, and not the chances that it could be attacked by an individual actor but instead by a government?
- read_if_gay_ 5y ago
  Governments don’t get to search your house because some people out there have CP at home. Why should your smartphone be different?
  - ryeguy_24 5y ago
    This sums up the frustration very eloquently.
- pille 5y ago
  I can’t speak for everyone, but that’s certainly a technical part of it. Another big part of the problem is that it’s insulting to presume everyone guilty, and make them to use their own resources (own phone, own battery cycles) to investigate them as if they were suspects. But that’s been discussed plenty on other threads here at HN.
- peteretep 5y ago
  Exactly that. The tech seems fine, but I live in a country with a government that has strong censorship laws, and I do not trust Apple to not bend to countries like China in extending this to political content.
- tucosan 5y ago
  It might be solid from a technical standpoint. Once you built it, governments will be coming and asking for more. Are you aware that the Chinese government already has been granted access to the infrastructure holding the keys to iCloud in China?
programmer_dude 5y ago
Can this affect people who do not use Apple products?
- theshrike79 5y ago
  No, how would it?
  - programmer_dude 5y ago
    Then why is it such a big deal on hackernews and elsewhere?
    - theshrike79 5y ago
      Because the headline says "Apple". It brings in the views and likes, which bring money.
- nullc 5y ago
  As a non-apple user you could be impacted indirectly by people you know being directly impacted or by Apple's practices being imported into the law. E.g. laws that attempt to outlaw encryption lacking apple-like backdoors.
slownews45 5y ago
"This is a false-positive rate of 2 in 2 trillion image pairs (1,431,168^2)" That is not bad. As a tool to filter down what apple human reviewers need to look at this is pretty good. Ultimately these images will make it to a human reviewer who can make a call as they would in any flagging system. Could a backend server side system do a more precise hash (96 bits is not a ton) prior to human review?
- willis936 5y ago
  2 collisions out of a million images. I'm not sure how big the CSAM database is but if it's a tens of thousands and there are millions of photos uploaded a day then Apple could have a problem on their hands. This is all extrapolating from a study that doesn't use photos representative of what people actually upload. I would suspect when most photos being uploaded are of humans the actual collision rate will be much higher.
  - xadhominemx 5y ago
    They don’t take any action unless you have 30 matches in the database, which will not happen by chance.
    - willis936 5y ago
      I would agree if we assume that this very deterministic system behaved randomly.
    - AmericanChopper 5y ago
      It could happen on purpose if I intentionally send you 30 colliding images. I don’t know how iMessage handles images, but WhatsApp for example will put them directly into your photo library (and from there directly into iCloud if you’ve got syncing enabled). Perhaps I could even do that without revealing my motives to you.
      - theshrike79 5y ago
        It would STILL just mildly inconvenience the Apple wageslave tasked with checking the notifications. If you send actual CP, then that's a wholly different matter.
nobrains 5y ago
Please help me understand. Isn't this the reason why the process involved a final manual review? If so, isn't the point of having identical hashes moot? Or is the point that having more identical hashes means reviewing more personal pictures manually, leading to a privacy issue?
- mns 5y ago
  I don't think I would trust a huge corporation with this. Plus, leaving the review to some internal classified process where some poor faceless guy needs to reach an unrealistically high quota of reviewed images per day to get his bonus, might be a bit of a risk.
  - madeofpalk 5y ago
    it's not just internal policy - the safety vouchers will not decrypt (technically impossible) unless there are ~30 matches. It is a policy encoded in cryptography.
    - zionic 5y ago
      "30 matches" just means you're one imgur album of adversarial images away from a swat team.
roody15 5y ago
Apple has yet to make a valid reason for implementing client side CSAM scanning. According to Apple only images that will be uploaded to iCloud will be scanned. If this is the case there is zero reason to scan locally and you can just scan the uploaded image once it is on the server. Apple has not implemented E2E nor has it released a statement indicating this will be implemented in the future.
- YetAnotherNick 5y ago
  Also they will be doing scanning in their server anyways as there are other ways to upload it in iCloud than using latest iOS.
- mrweasel 5y ago
  One reason for client side could be to save on datacenter compute resources. That would seem like a perfectly valid reason, if that’s their reasoning.
  - AnonC 5y ago
    If it’s going to really save a significant amount of data center resources, then it’s also probably going to reduce the battery lifespan of all these devices significantly. That may probably be good for Apple’s bottom line temporarily, but it will hurt in the long run. I’d imagine it’d be a lot easier to optimize the data center compute resources than optimizing the scanning on individual devices and not trashing battery lifespan.
    - josefx 5y ago
      > I’d imagine it’d be a lot easier to optimize the data center compute resources than optimizing the scanning on individual devices and not trashing battery lifespan. No amount of data center optimization will beat running computations on hundreds of millions of devices other people have to pay for.
  - jtbayly 5y ago
    If that’s a valid reason to steal electricity and compute resources from your customers, then why not go the whole way and use all the Mac’s as storage and compute for iCloud?
nextlevelwizard 5y ago
How are any of these "naturally occurring" when all (4) examples are things cut out of context on a white background. Yeah two sticks (ski and nail) are visually similar on a white background. Why is this news to anyone? EDIT: if you are going to downvote please leave a comment unless you are just downvoting for wrong think.
- verygoodname 5y ago
  As it is explained in the "readme" part, in this specific context, "naturally occurring" means that no one has purposefully manipulated any of the images to make them collide: that the images were already published and "out there" and happen to collide. In other words, it does not necessarily imply that the images correspond to natural photographic scenes (which seems to be your interpretation of it). Besides, you could probably "naturally" obtain such type of colliding images by photographing similar-looking objects against a white (or generally featureless) background. Furthermore, it suggests/demonstrates that similar-looking images with similar backgrounds can lead to unexpected collisions in practice (i.e. "naturally"), even if you do not assume an adversarial scenario. Are you sure that, if you take a picture of a naked body part, it won't collide with anything that looks similar in their database?
  - nextlevelwizard 5y ago
    It is unlikely unless you manage to capture some position and happen to have some background. This whole thing is a nothingburger. This is one of those weird things were many people have baseless gut reactions and then try to go and prove if flawed even though they don't have a complete picture. It is unlikely that there is a collision of benign image with the database and even if that happens it is not some automatic process that just sends cops to your house to raid it. Of course we can get bunch of collitions with essentially same images, I don't get why this is so magical just squint your eyes and I'm sure you have two objects with in your reach that could be made to collide, but that isn't a gotcha on any level
    - ogma 5y ago
supperburg 5y ago
It would be a shame if thousands of people regularly uploaded hash collisions to their iCloud overwhelming apples human review capacity
scotty79 5y ago
Why are exact collisions interesting? They are not intended to be compared exactly. This algorithm doesn't even give exact matches for the same image on different hardware. https://github.com/AsuharietYgvar/AppleNeuralHash2ONNX https://github.com/AsuharietYgvar/AppleNeuralHash2ONNX Note: Neural hash generated here might be a few bits off from one generated on an iOS device. This is expected since different iOS devices generate slightly different hashes anyway. The reason is that neural networks are based on floating-point calculations. The accuracy is highly dependent on the hardware. For smaller networks it won't make any difference. But NeuralHash has 200+ layers, resulting in significant cumulative errors.
- yeldarb 5y ago
  That is a good point; has Apple stated how many bits two images’ NeuralHashes can differ by and still be considered a “match” by their system?
- tgv 5y ago
  The hash is 96 bits long. When hashing 1 billion pictures, that gives a collision probability of 6e-12. If it were uniformly distributed. There's no way people have hashed billions of images already. It just shows that it's pretty probably there will be collisions, and on visual inspection, it looks as if the collisions will happen on visually similar images. So if there's a naked baby pic in the CSAM database, quite a few of you 100s of child pictures can be flagged.
  - Lamad123 5y ago
    The Tim cuk's law: "If it can happen, it will happen"
  - halflings 5y ago
    Clearly this is not a cryptographic hash, and hence it's known hashes are not uniformly distributed. Apple explained in their technical summary [0] that they'll only consider this an offence if a certain number of hashes match. They estimated the likelihood of false positives there (they don't explain which dataset was used, but it was non-CSAM naturally) is 1 out of a trillion [1] In the very unlikely event where that 1 in a trillion occurrence happens, they have manual operators to check each of these photos. They also have a private model (unavailable to the public) to double-check these perceptual hashes which also used before alerting authorities. [0] https://www.apple.com/child-safety/pdf/CSAM_Detection_Technical_Summary.pdf https://www.apple.com/child-safety/pdf/CSAM_Detection_Techni... [1] https://www.zdnet.com/article/apple-to-tune-csam-system-to-keep-one-in-a-trillion-false-positive-deactivation-threshold/ https://www.zdnet.com/article/apple-to-tune-csam-system-to-k...
AnonC 5y ago
A very relevant point on this entire discourse about Apple’s on-device CSAM scanning: According to the U.S. law, key snippets of which are quoted on the Stratechery blog (by Ben Thompson), Apple isn’t obligated to scan for CSAM. It’s only obligated to act on CSAM if it finds them. While it’s good for Apple to scan on its systems (iCloud) like Facebook, Google and other companies do on their servers, it’s inappropriate to do it on individual devices, which starts with the assumption that anyone who has iCloud photos enabled is a potential CSAM hoarder and needs to pay with their device’s battery life and time for the scanning to happen and report back. It’s a sort of micro-robbery that Apple is doing on the devices when there is no legal compulsion to do so. Everything else on trusting Apple’s NeuralHash or the sanctity of the NCMEC hashes come later, IMO. I sincerely hope Apple realizes that it’s got a dud solution on hand, eats humble pie (which it’s usually not capable of) and ditches this whole thing. I know a lot of egos at Apple are at stake here. But doing the right thing matters for a company that claims that “privacy is a fundamental human right” and has a CEO who’s a member of a marginalized/discriminated community and understands the risks of these efforts.
- Lamad123 5y ago
  They say only if they find 30 matching images, they'd act. So if they find 20 or 29 and don't report them, they are actually breaking the law!! I am wondering why they chose that magical number!
  - Tagbert 5y ago
    Just getting a match on the neural hash is not sufficient to declare an image CSAM. It just flags it as a candidate for review. Apple is not required to report it unless it can be verified to be CSAM. there is a multi-step process involving multiple processes and organizations to do so.
    - Lamad123 5y ago
      They are opening the door for a lot of questions like this. They didn't say (as far as I know) if this 30-number cover multiply upload sessions or it gets reset with every upload. I feel like this number gets reset to 0 each for each upload session. Somebody can deliberately safely upload 29 actual CSAM photos each time. I know Apple is not stupid, but you know the government and lawmakers will ask it for a smaller magical number...
StrLght 5y ago
I don't really get what this repository is trying to achieve and what's the point of collecting collisions. Collisions will happen, that's just how it is with hashes. It's already a public knowledge that Apple has 2 more systems (some server-side verification and a manual check later) to prevent false-positives. So what's the point of researching collisions in NeuralHash?
- woofie11 5y ago
  No. Most proper cryptographic hash systems (e.g. used for verifying files, rather than data structures) never have collisions. Try to find a SHA256 collision. Anywhere, ever, in the history of mankind. This isn't for lack of looking. A lot of very smart people have looked for them. If you find one, I bet you'll be eligible for a tenured faculty slot at a good university, if not more. A whole world of secure systems would need to be re-engineered. Hypothetical collisions of course exist, by the pigeonhole principle, just not in the real world.
  - StrLght 5y ago
    Yes, but cryptographic hashes are irrelevant here because they'd allow to easily bypass CSAM by modifying/appending a single byte.
    - woofie11 5y ago
      Apple is claiming to have a visual equivalent to a cryptographic hash -- one which won't change with a single byte, but only if the image is substantially different. At least their security analysis relies on that. From their whitepaper: "The threshold is selected to provide an extremely low (1 in 1 trillion) probability of incorrectly flagging a given account" If your claim is that their hash algorithm isn't cryptographic, their security analysis is incorrect.
      - kbelder 5y ago
        Their security analysis is obviously incorrect. "equivalent to a cryptographic hash" "change... only if the image is substantially different" Both are not true, cannot be true.
floor_ 5y ago
Why not combine this with a second different hash? E: Better yet, only run the second hash if you have a collision, which should be very rare.
- theshrike79 5y ago
  How do you know they aren't doing this on the backend after the initial on-device match?
- yeldarb 5y ago
  Apple does, and I created a proof of concept for how it might work to guard against adversarially perturbed images here: https://blog.roboflow.com/apples-csam-neuralhash-collision/ https://blog.roboflow.com/apples-csam-neuralhash-collision/
teekert 5y ago
I still think the biggest problem is that at some point a human is going to look at a false positive, this may be picture of my naked children and this human may not have the best intentions with my picture. That said, Nextcloud is my backend and I do not upload anything to iCloud (except for MS authenticator 2fa backups), so I'm safe right?
- Grustaf 5y ago
  Since the risk of that is 1 in a trillion, a lot of people are quite happy to take that risk.
  - dathinab 5y ago
    It's not 1 in a trillion, it's MUCH higher. We are not speaking about a situation where not a "arbitrary" picture is miss-classifieds. We are speaking about a situation where a innocent picture involving a naked or not fully clothed child is deemed similar to a non innocent picture of a naked or not fully clothed child. Now you might argue that there should not be a picture a a naked or not fully clothed child of any form ever on any phone, but IMHO that is short sighed, discriminating and at best shows you don't know to much about the world and other cultures. Let's list some simple reasons such a think could happen first: - Photos meant for a doctor, or living partner to ask if something is normal or a problem. In many different ways. - Photos of little children bathing or similar a e.g. dad sends to their mom who is currently on a business trip. - etc. Reasons people are less aware of is that not all countries are as stuck up about nakedness especially in the family. So it's totally normal for families that e.g. before or after taking a shower family member independent of age and gender walk through the apartment naked. Similar if you didn't got any shame about the naked body indoctrinated you might totally do thinks like visiting a "naked-beach" with your family (meeting other families and taking advantage of it often being less crowded) and in turn normal innocent beach family pictures contain naked children. And on itself that's not a problem. But with Apples approach stuff like this is like to trigger both systems Apple announced and wrongly label your while family as pedophiles...
    - 5y ago
Grustaf 5y ago
This is of course entertaining, but since Apple has already tested for this, with 100 million images, and adjusted the rules accordingly, it has no practical implications.
Lamad123 5y ago
The problem is that a company is raping my privacy!! I don't care about the details of how good or bad is their spyware is!!
foxfluff 5y ago
My take on this is that the system is by and large useless. It won't catch anything but the dumbest of dumb criminals, because those who care about CSAM can surely figure out a better way to share images, or find a way to obfuscate their images enough to bypass the system (the lower the false positive rate, the easier it must be to trick the system). So what's left when all the criminals this is supposed to catch have figured it out? False positives. Only false positives. Is it really worth turning personal devices into snitches that don't even do a good job of protecting children? Also, numbers about false positives must be taken with a grain of salt because of the non-uniform distribution of perceptual hashes. It might be that your random vacation photos and kitty pics have a 1-in-a-million chance of a fapo, but someone who happens to (say) live in an apartment that has been laid out very similarly to a scene in pictures appearing in the CSAM database may have a massively higher chance of fapos for photos taken in their home.
- Retric 5y ago
  Perceptual hashes are only used to reduce the search space for human review. Apple doesn’t have images in the CSAM database to do a comparison, but if it’s just a picture of a door their going to reject it. Also, because human review is an expense Apple’s incentives are to minimize the number of times it happens, thus the requirement for multiple collisions.
  - jdavis703 5y ago
    > Apple’s incentives are to minimize the number of times it happens, thus the requirement for multiple collisions. How can we be sure they won’t cut costs by increasing worker load? I could see them giving each reviewer less time to review individual pictures before passing it on to law enforcement.
    - fraa-orolo 5y ago
      We can't and they probably will, everyone else seems to already be doing so. There's that Swiss federal police report that only about 10% of NCMEC reports are actually relevant (https://fedpol.report/en/fedpol-in-figures/fight-against-paedophile-crime https://fedpol.report/en/fedpol-in-figures/fight-against-pae...)
JoshTko 5y ago
The threshold of collisions Apple is using before review is 40
- dathinab 5y ago
  I think no one is anymore afraid of 40 accidental natural image collisions. But un-natural image collisions or bad images in the database and similar are a different matter and had been the main critique point from the get to go as far as I can tell.
  - dathinab 5y ago
    Also given how many people use IPhones, how many pictures they have and how often they have many similar pictures, thinks are not necessary that simple. I wouldn't be surprised if some flat, small height fully adult (e.g. 30) woman does some sexting and goes from 0 to >40 collisions in a month. Not because of arbitrary collisions but because the similarity some of here sexting pictures might have with the ones from a 14y old but older looking girl (which e.g. where forced and ended up in the database).
ya3r 5y ago
Technically speaking, this does not prove that an adversarial attack is possible on the CSAM system of apple, Given that apple has another not released neural hash system on their servers which is potentially larger and works better than the one on device. The more interesting technical question for me is: do collisions transfer across models? or how to find collisions that transfer across models?
tyingq 5y ago
I guess don't upload pictures of peaches, poppy buds, phallic cacti, and so on to iCloud.
nitrogen 5y ago
The technology is not why the Apple system is unwanted. It's just extra fuel for the fire. This system is unwanted because it puts a spy literally in your house and in your hands. It's bad enough that cloud everything blurs the line between what's yours and what's mine. Placing any law enforcement tech on a user's own device takes that line between "public" and "private" and completely erases it.
- nullc 5y ago
  Absolutely. The problem is Apple introducing a spy into your home. This alone should be bad enough, but some people are rather trusting. Showing that the spy is also tripping balls both exposes additional risks and emphasizes that Apple neither has their best interest at heart nor is putting adequate care into their actions. The latter gives people reason to question apple's claims of additional protection mechanisms that are non-falsifiable.
Cyberdog 5y ago
Isn't a hash collision from similar images the point of the whole thing? At any rate, IANAL, but I'm pretty sure you can't be convicted based on a hash alone. If you get busted for possession of a picture of a nematode and you can show the jury it's just a picture of an axe that has the same value when run through this algorithm, you'll be fine. And there's a decent chance prosecutors won't chase down individuals who will just have a single collision in their photo library with this tech in the first place - people who have dozens or hundreds will be much more interesting.
erdos4d 5y ago
Is it possible for the courts to use this system to search a defendant's phone for leaked documents say? Like if NSA learns that one of a small group leaked document X, can they get a court to force Apple to add the hash of Document X to the database on that group of people's phones? If so, I bet this becomes the new norm for investigating leaks.
gok 5y ago
The "catalog" has two entries.

337 points by hongsy 5y ago

Sigh, for the last time, it doesn't actually matter if the NeuralHash is identical. You need multiple images matching, and then the images are compared by another system on Apple's end, which you don't know anything about. The system is specifically designed so that colliding images does not pose a threat to the user. NeuralHash and the CSAM scanning is grotesque, but please, criticize it for what it is, not some bullshit that is easily dismissed as technical ignorance.

collaborative 5y ago
We still need to rely on the automated "secret backend system" that nobody supposedly knows anything about
- toxik 5y ago
  You (or at least Apple's customers) trust in and rely on Apple's proprietary software to do its job all the time. How is this different? I find this argument very weak.
  - Hackbraten 5y ago
    1. You can at least somewhat audit the software running on an iPhone, for example by means of reverse engineering. You can’t audit the server side. 2. It’s one thing to rely on proprietary services like Find My or Siri. It’s another thing to rely on a secret server-side app that has the power to destroy your life.
    - idunnoman 5y ago
      "Hey Siri, accuse me of something that will ruin my life and reputation even if I'm not guilty".
    - fetzu 5y ago
      What I somehow fail to grasp in the first argument is that this whole system is designed specifically so that it runs client-side. AFAIK all the alternatives (as in « cloud photo services ») have been doing the exact same thing on the server side for decades. If you upload your photos to the cloud, a lot of service actually already have the power to destroy your life.
      - collaborative 5y ago
        This is all about Apple users waking up to the fact that they've been had
  - strangetortoise 5y ago
    afaik none of Apples other "proprietary software" is designed to pass my personal images to a human for visual inspection if it mistakenly outputs 2 high-enough numbers after a handful of convolution operations and matrix multiplications.
    - FabHK 5y ago
      They pass a "visual derivative" to "a human", but only after some matrix multiplications etc. that result in extremely low probability false positives. It could also happen that you lose your phone and "a human" finds it and randomly puts in the correct passcode on the first try and visually inspects your personal images. In fact, that seems vastly more likely [1]. [1] About 4% of smartphones are lost or stolen every year [https://www.mcafee.com/blogs/consumer/family-safety/almost-5-of-smartphones-lost-every-year/ https://www.mcafee.com/blogs/consumer/family-safety/almost-5... ], but make it just 1/1000, so 1e-3. Then a 6 digit passcode, 1e-6, so we're at 1e-9 per year, or 1000x as likely as being falsely flagged, assuming Apple's numbers (which can easily be achieved by calibrating the threshold).
  - collaborative 5y ago
    Tell this to the victims of Pegasus. If anyone were able to get their hands on the "secret backend system" we wouldn't be talking about spy games, we'd be talking about people's lives being ruined
    - roody15 5y ago
      Apple still has not patched the security exploit in iMessage used by Pegasus. Apple has released two ios security updates since the Pegasus revelations but still has not patched it most widely used exploit… hmmmm. Now apple is getting a local client side scanning tool ready. Interesting timing.
      - collaborative 5y ago
        I don't think what you are saying is far-fetched
  - read_if_gay_ 5y ago
    No. I trusted Apple’s proprietary software before this, because they maintained that they care about privacy, and there have been examples of that. Now, I don’t trust them anymore.
  - simion314 5y ago
    There is a big difference, if say Apple tracking of what you run is problematic you get a bad user experience like apps starting after 1 minute of waiting, or if Apple App Store contains malware you will probably get some annoying issue while Apple will try to silently cleanup their mess BUT with this system there is a difference, this is designed not to serve you or protect you, it is designed to do some checks then if some specific rules match send some guys on you to destroy your life, today is FBI tomorrow other authoritarians. Issues in any other Apple software will not send the police on you. Why would you install a software on your desktop/laptop that is designed to snitch on you, you would need to get some advantage or be forced by some law. For now I see only disadvantages but please let me know of any real advantage and not speculation Disadvantages: - closed software with hidden db can't be trusted, so as a user you will always have a doubt that some non CP images are in the db(Apple always collaborates with governments) - bugs in this stuff will cause you big problems(we seen in the past how false accusation destroyed peoples life) and we also seen bad actors abusing this kind of stuff. - this is also clearly a beginning, now that Apple has the capability then even if they were saints a judge could force them to add new hashes, change the configs etc.
tsimionescu 5y ago
Then let's get rid of the NeuralHash entirely, if it doesn't matter, right? If it's a critical part of the system, then it should be inspected thoroughly. If Apple claims a minuscule chance of a hash collision, and the reality is that collisions are relatively common, that significantly changes the requirements for the backend system, which Apple keeps secret. We have every right to believe, bbased oon ppublic info, that Apple was expecting that NeuralHash would be almost fool-proof, leaving the backend system to be a rubber stamp. This would be tragic.
- toxik 5y ago
  The point in the NeuralHash and PSI system is to preserve user privacy as far as possible. From a technical standpoint, it is not essential - a NeuralHash function that returns 0x0000… for everything would still catch CSAM. It's just that it would upload every single image on the user's device. Now, how well this NeuralHash does preserve privacy is a different question, and /not/ one that is being answered by the original post here. In fact, I've not seen anybody look at the hash distribution over natural images, which would be an actual argument against the system.
  - Retr0id 5y ago
    It doesn't really matter whether all images are uploaded, or just 1 in x (for large value of x), due to the Panopticon effect.
    - ithkuil 5y ago
      Let's not forget what the alternative is: this is about images that are uploaded on icloud anyway. The alternative is to upload the image in clear (or with ane encryption key that apple controls), and let apple run the CSAM filter on their servers. Apple now has the ability to encrypt the images before sending them to icloud, with a private key you own. Except that some percentage of images that match the CSAM fingerprint with their neural feature extractor will be sent to a CSAM filter on the server side (whose workings we don't have many details about) This whole thing backfired on Apple entirely due to psychological effects, not because they are really doing anything more "panopticon" that they would already able to do now on their icloud storage (after all people are ready sending their photos to apple)
draw_down 5y ago
Come on.
saithound 5y ago
Discussing the preimage attack on NeuralHash is not technical ignorance. Dismissing the preimage attack as irrelevant is. 0. Most importantly: the existence of a preimage attack makes Apple's system completely useless for its original purpose. The NeuralHash collider allows the producers and distributors of CSAM material to ensure that nearly all of the next generation of CSAM will suffer from hash collisions with perfectly innocent images. Two weeks after it was deployed, Apple's CSAM scanning is now _only_ an attack vector and a privacy risk. Thanks to the preimage attack, it's now completely useless for its nominal function! Apple put a lot of effort into a system that reduced the privacy and security of all their customers, and made the company itself more exposed to the whims of governments. And for no gain whatsoever. 1. There are no known perceptual hash functions on which preimage attacks are difficult. Barring a major "secret cryptographic breakthrough", Apple's second hash function is not resistant to preimage attacks either. In fact, the second algorithm is almost certainly easier to attack than NeuralHash itself, since it has to work on the "visual derivative", a fixed-size low-resolution thumbnail of the original image. 2. But isn't Apple's second algorithm kept secret, making it difficult to perform preimage attacks against it? No. First of all, the second algorithm cannot be kept secret. Apple doesn't have its own CSAM database (the whole point is that they don't want to deal with CSAM on their servers!), so the algorithm has to be shared with multiple organizations which do have such databases, so that they can pre-compute the hashes that Apple will match against. Due to Apple's policy, some of these organizations will be located outside the US [1]. Chances are, the hash function will leak: Apple won't know if and when that happens. Secondly, this _is_ security by obscurity. Some people argue that keeping the hash algorithm secret is similar to keeping a cryptographic key secret. This is not the case. Of course, any security system relies on keeping _something_ secret, but these secret somethings are not created equal. The secret keys of cryptographic algorithms are designed to satisfy Kerckhoffs's assumption. This means that the key, as long as it remains secret, should be sufficient to protect the confidentiality and integrity of your system, even if your adversary knows everything else apart from the key, including the details of the algorithm you use, the hardware you have, and even all your previous plaintexts and ciphertexts (inputs and outputs). The second hash does not have this property at all. Keeping the algorithm secret does not ensure the confidentiality or integrity of Apple's system. E.g. if somebody gets access to a reasonable number of inputs-output examples, that allows them to train their own model which behaves similarly enough to let them find perceptual hash collisions, even if they don't know the exact details of the original algorithm. This is incredibly hard for cryptographic hashes, but very easy for perceptual hashes, since a small change in the input should cause only a small change in the output of the perceptual hash algorithm. So, to maintain security, Apple doesn't have to keep just the hash algorithm (or its configuration parameters) secret, but all the inputs and outputs as well. This is bad: the fewer and simpler the secrets that one must keep to ensure system security, the easier it is to maintain system security. Finally, the second hash algorithm is unlikely to be original (NeuralHash was original, and by all accounts it was a massive effort). If an attacker successfully guesses that Apple's secret algorithm H is closely related to a known algorithm, say PhotoDNA, they will probably be able to make a transfer attack against it. By engineering a PhotoDNA collision on the resized thumbnail (e.g. via a resizing attack, extensively discussed in a previous thread [3]), they have a reasonable chance of generating a H-collision as well. How good is fairly good? Well, something like 5% is more than enough! The attacker needs to produce a certain number of NeuralHash collisions (say 30 images) to get through the first threshold of Apple's algorithm. But after that, Apple will decode all the thumbnails in the user's safety voucher: the attacker only needs one of those 30 to get through the second hash. Given a sufficiently high probability of hash collisions, this can be achieved "blindly". 3. It's incredibly easy to come up with these kinds of attacks. Even the HN audience could come up with several reasonable plans, and could point out several reasonable issues, in two weeks. People who do malice for a living will have a much easier time with it. Even if somehow all the plans presented on HN turned out to be unviable, it will not take long for someone to stumble upon something practical. Any reassurance that Apple could provide at this point is fake. Cf. the timelines for real security: it took 17 years to come up with an analogous attack against SHA-1 [4], and two years after that to turn it into something that can be exploited in practice [5]. The existence of a preimage attack made Apple's system completely useless for its original purpose in two weeks. It's now just a security and privacy hole, with no other function. Keeping it around would be a travesty, even if it was difficult to exploit. But it's not. [1] https://www.itnews.com.au/news/apple-to-only-seek-abuse-images-flagged-in-multiple-nations-568625 https://www.itnews.com.au/news/apple-to-only-seek-abuse-imag... [2] https://en.wikipedia.org/wiki/Kerckhoffs%27s_principle https://en.wikipedia.org/wiki/Kerckhoffs%27s_principle [3] https://news.ycombinator.com/item?id=28236102 https://news.ycombinator.com/item?id=28236102 [4] https://security.googleblog.com/2017/02/announcing-first-sha1-collision.html https://security.googleblog.com/2017/02/announcing-first-sha... [5] https://www.zdnet.com/article/sha-1-collision-attacks-are-now-actually-practical-and-a-looming-danger/ https://www.zdnet.com/article/sha-1-collision-attacks-are-no...
nullc 5y ago
(A threshold of) matchings result in the private keys for the images being leaked to Apple, where they're vulnerable to: (1) Review by apple staff (2) Access and leaking by other apple staff (3) Access by hackers who have compromised their system (4) Access by parties coercing apple/staff, including via national security letters. All of which compromise the privacy of the user. This matters or the neuralhash comparison wouldn't exist in the first place. Totally agree that the whole system is grotesque-- but that doesn't stop it also being grotesque in every detail as well. The fact that there are false positives when they easily could have designed a system that had none (at the expense of increased false negatives) shows that Apple doesn't especially value customer privacy even if you accept their vigilante privacy invasion. The fact that it's possible to construct adversarial false positives and that their reports didn't disclose this fact shows they either don't know what they're doing or they're not being honest about the risks (or both).

hhsbz 5y ago

Is it really a catalogue when there only are two of them? I find it amusing that they probably ran this tool against a set of millions or even billions of images and this is the best they could come up with. They are practically praising Apple here lmao

valparaiso 5y ago

Lmao no one knows how works Apple production code/algorithm but already did assumptions.

yosito 5y ago

> a catalog Can two collisions really be called a catalog?

yeldarb 5y ago
PRs welcome!
eesmith 5y ago
A catalog of a thousand pages begins with the first entry.
- nannal 5y ago
  And a story may start with the first word, but if I present the word "Octopus" and say check out my story, you're going to be well within bounds to question me on it.
  - simondotau 5y ago
    Well to be fair there were two collisions, so your story should be "Octopus imploded" and now everyone is captivated by the plight of this unlucky cephalopod.
  - eesmith 5y ago
    Wikipedia started as just an announcement of the idea (https://web.archive.org/web/20030414014355/http://www.nupedia.com/pipermail/nupedia-l/2001-January/000676.html https://web.archive.org/web/20030414014355/http://www.nupedi... ). How many entries do you think there were when the first live version was announced only a few hours later (http://www.nupedia.com/pipermail/nupedia-l/2001-January/000678.html http://www.nupedia.com/pipermail/nupedia-l/2001-January/0006... )? As a different metaphor than your "Octopus", this is "first light". "First light" in astronomy is the first time a telescope is used. It doesn't need to start with an amazing or ground-breaking image.
dathinab 5y ago
It's a WIP catalog where everyone who stumbles over one can put it in. It could in the future be used to e.g. improves this algorithms.

theshadowknows 5y ago

I’m glad that people are trying to figure out any technical flaws in the system as best they can, but if I’m being honest I do trust Apple’s engineers to have built something that is solid from a technical stand point. Am I correct in that the primary reason folks are so upset is that the system could (probably) be easily modified such that -any- content could invoke legal action? That the main problem is really the scanning at all, and not the chances that it could be attacked by an individual actor but instead by a government?

read_if_gay_ 5y ago
Governments don’t get to search your house because some people out there have CP at home. Why should your smartphone be different?
- ryeguy_24 5y ago
  This sums up the frustration very eloquently.
pille 5y ago
I can’t speak for everyone, but that’s certainly a technical part of it. Another big part of the problem is that it’s insulting to presume everyone guilty, and make them to use their own resources (own phone, own battery cycles) to investigate them as if they were suspects. But that’s been discussed plenty on other threads here at HN.
peteretep 5y ago
Exactly that. The tech seems fine, but I live in a country with a government that has strong censorship laws, and I do not trust Apple to not bend to countries like China in extending this to political content.
tucosan 5y ago
It might be solid from a technical standpoint. Once you built it, governments will be coming and asking for more. Are you aware that the Chinese government already has been granted access to the infrastructure holding the keys to iCloud in China?

programmer_dude 5y ago

Can this affect people who do not use Apple products?

theshrike79 5y ago
No, how would it?
- programmer_dude 5y ago
  Then why is it such a big deal on hackernews and elsewhere?
  - theshrike79 5y ago
    Because the headline says "Apple". It brings in the views and likes, which bring money.
nullc 5y ago
As a non-apple user you could be impacted indirectly by people you know being directly impacted or by Apple's practices being imported into the law. E.g. laws that attempt to outlaw encryption lacking apple-like backdoors.

slownews45 5y ago

"This is a false-positive rate of 2 in 2 trillion image pairs (1,431,168^2)" That is not bad. As a tool to filter down what apple human reviewers need to look at this is pretty good. Ultimately these images will make it to a human reviewer who can make a call as they would in any flagging system. Could a backend server side system do a more precise hash (96 bits is not a ton) prior to human review?

willis936 5y ago
2 collisions out of a million images. I'm not sure how big the CSAM database is but if it's a tens of thousands and there are millions of photos uploaded a day then Apple could have a problem on their hands. This is all extrapolating from a study that doesn't use photos representative of what people actually upload. I would suspect when most photos being uploaded are of humans the actual collision rate will be much higher.
- xadhominemx 5y ago
  They don’t take any action unless you have 30 matches in the database, which will not happen by chance.
  - willis936 5y ago
    I would agree if we assume that this very deterministic system behaved randomly.
  - AmericanChopper 5y ago
    It could happen on purpose if I intentionally send you 30 colliding images. I don’t know how iMessage handles images, but WhatsApp for example will put them directly into your photo library (and from there directly into iCloud if you’ve got syncing enabled). Perhaps I could even do that without revealing my motives to you.
    - theshrike79 5y ago
      It would STILL just mildly inconvenience the Apple wageslave tasked with checking the notifications. If you send actual CP, then that's a wholly different matter.

nobrains 5y ago

Please help me understand. Isn't this the reason why the process involved a final manual review? If so, isn't the point of having identical hashes moot? Or is the point that having more identical hashes means reviewing more personal pictures manually, leading to a privacy issue?

mns 5y ago
I don't think I would trust a huge corporation with this. Plus, leaving the review to some internal classified process where some poor faceless guy needs to reach an unrealistically high quota of reviewed images per day to get his bonus, might be a bit of a risk.
- madeofpalk 5y ago
  it's not just internal policy - the safety vouchers will not decrypt (technically impossible) unless there are ~30 matches. It is a policy encoded in cryptography.
  - zionic 5y ago
    "30 matches" just means you're one imgur album of adversarial images away from a swat team.

roody15 5y ago

Apple has yet to make a valid reason for implementing client side CSAM scanning. According to Apple only images that will be uploaded to iCloud will be scanned. If this is the case there is zero reason to scan locally and you can just scan the uploaded image once it is on the server. Apple has not implemented E2E nor has it released a statement indicating this will be implemented in the future.

YetAnotherNick 5y ago
Also they will be doing scanning in their server anyways as there are other ways to upload it in iCloud than using latest iOS.
mrweasel 5y ago
One reason for client side could be to save on datacenter compute resources. That would seem like a perfectly valid reason, if that’s their reasoning.
- AnonC 5y ago
  If it’s going to really save a significant amount of data center resources, then it’s also probably going to reduce the battery lifespan of all these devices significantly. That may probably be good for Apple’s bottom line temporarily, but it will hurt in the long run. I’d imagine it’d be a lot easier to optimize the data center compute resources than optimizing the scanning on individual devices and not trashing battery lifespan.
  - josefx 5y ago
    > I’d imagine it’d be a lot easier to optimize the data center compute resources than optimizing the scanning on individual devices and not trashing battery lifespan. No amount of data center optimization will beat running computations on hundreds of millions of devices other people have to pay for.
- jtbayly 5y ago
  If that’s a valid reason to steal electricity and compute resources from your customers, then why not go the whole way and use all the Mac’s as storage and compute for iCloud?

nextlevelwizard 5y ago

How are any of these "naturally occurring" when all (4) examples are things cut out of context on a white background. Yeah two sticks (ski and nail) are visually similar on a white background. Why is this news to anyone? EDIT: if you are going to downvote please leave a comment unless you are just downvoting for wrong think.

verygoodname 5y ago
As it is explained in the "readme" part, in this specific context, "naturally occurring" means that no one has purposefully manipulated any of the images to make them collide: that the images were already published and "out there" and happen to collide. In other words, it does not necessarily imply that the images correspond to natural photographic scenes (which seems to be your interpretation of it). Besides, you could probably "naturally" obtain such type of colliding images by photographing similar-looking objects against a white (or generally featureless) background. Furthermore, it suggests/demonstrates that similar-looking images with similar backgrounds can lead to unexpected collisions in practice (i.e. "naturally"), even if you do not assume an adversarial scenario. Are you sure that, if you take a picture of a naked body part, it won't collide with anything that looks similar in their database?
- nextlevelwizard 5y ago
  It is unlikely unless you manage to capture some position and happen to have some background. This whole thing is a nothingburger. This is one of those weird things were many people have baseless gut reactions and then try to go and prove if flawed even though they don't have a complete picture. It is unlikely that there is a collision of benign image with the database and even if that happens it is not some automatic process that just sends cops to your house to raid it. Of course we can get bunch of collitions with essentially same images, I don't get why this is so magical just squint your eyes and I'm sure you have two objects with in your reach that could be made to collide, but that isn't a gotcha on any level
  - ogma 5y ago

supperburg 5y ago

It would be a shame if thousands of people regularly uploaded hash collisions to their iCloud overwhelming apples human review capacity

scotty79 5y ago

Why are exact collisions interesting? They are not intended to be compared exactly. This algorithm doesn't even give exact matches for the same image on different hardware. https://github.com/AsuharietYgvar/AppleNeuralHash2ONNX https://github.com/AsuharietYgvar/AppleNeuralHash2ONNX Note: Neural hash generated here might be a few bits off from one generated on an iOS device. This is expected since different iOS devices generate slightly different hashes anyway. The reason is that neural networks are based on floating-point calculations. The accuracy is highly dependent on the hardware. For smaller networks it won't make any difference. But NeuralHash has 200+ layers, resulting in significant cumulative errors.

yeldarb 5y ago
That is a good point; has Apple stated how many bits two images’ NeuralHashes can differ by and still be considered a “match” by their system?
tgv 5y ago
The hash is 96 bits long. When hashing 1 billion pictures, that gives a collision probability of 6e-12. If it were uniformly distributed. There's no way people have hashed billions of images already. It just shows that it's pretty probably there will be collisions, and on visual inspection, it looks as if the collisions will happen on visually similar images. So if there's a naked baby pic in the CSAM database, quite a few of you 100s of child pictures can be flagged.
- Lamad123 5y ago
  The Tim cuk's law: "If it can happen, it will happen"
- halflings 5y ago
  Clearly this is not a cryptographic hash, and hence it's known hashes are not uniformly distributed. Apple explained in their technical summary [0] that they'll only consider this an offence if a certain number of hashes match. They estimated the likelihood of false positives there (they don't explain which dataset was used, but it was non-CSAM naturally) is 1 out of a trillion [1] In the very unlikely event where that 1 in a trillion occurrence happens, they have manual operators to check each of these photos. They also have a private model (unavailable to the public) to double-check these perceptual hashes which also used before alerting authorities. [0] https://www.apple.com/child-safety/pdf/CSAM_Detection_Technical_Summary.pdf https://www.apple.com/child-safety/pdf/CSAM_Detection_Techni... [1] https://www.zdnet.com/article/apple-to-tune-csam-system-to-keep-one-in-a-trillion-false-positive-deactivation-threshold/ https://www.zdnet.com/article/apple-to-tune-csam-system-to-k...

AnonC 5y ago

A very relevant point on this entire discourse about Apple’s on-device CSAM scanning: According to the U.S. law, key snippets of which are quoted on the Stratechery blog (by Ben Thompson), Apple isn’t obligated to scan for CSAM. It’s only obligated to act on CSAM if it finds them. While it’s good for Apple to scan on its systems (iCloud) like Facebook, Google and other companies do on their servers, it’s inappropriate to do it on individual devices, which starts with the assumption that anyone who has iCloud photos enabled is a potential CSAM hoarder and needs to pay with their device’s battery life and time for the scanning to happen and report back. It’s a sort of micro-robbery that Apple is doing on the devices when there is no legal compulsion to do so. Everything else on trusting Apple’s NeuralHash or the sanctity of the NCMEC hashes come later, IMO. I sincerely hope Apple realizes that it’s got a dud solution on hand, eats humble pie (which it’s usually not capable of) and ditches this whole thing. I know a lot of egos at Apple are at stake here. But doing the right thing matters for a company that claims that “privacy is a fundamental human right” and has a CEO who’s a member of a marginalized/discriminated community and understands the risks of these efforts.

Lamad123 5y ago
They say only if they find 30 matching images, they'd act. So if they find 20 or 29 and don't report them, they are actually breaking the law!! I am wondering why they chose that magical number!
- Tagbert 5y ago
  Just getting a match on the neural hash is not sufficient to declare an image CSAM. It just flags it as a candidate for review. Apple is not required to report it unless it can be verified to be CSAM. there is a multi-step process involving multiple processes and organizations to do so.
  - Lamad123 5y ago
    They are opening the door for a lot of questions like this. They didn't say (as far as I know) if this 30-number cover multiply upload sessions or it gets reset with every upload. I feel like this number gets reset to 0 each for each upload session. Somebody can deliberately safely upload 29 actual CSAM photos each time. I know Apple is not stupid, but you know the government and lawmakers will ask it for a smaller magical number...

StrLght 5y ago

I don't really get what this repository is trying to achieve and what's the point of collecting collisions. Collisions will happen, that's just how it is with hashes. It's already a public knowledge that Apple has 2 more systems (some server-side verification and a manual check later) to prevent false-positives. So what's the point of researching collisions in NeuralHash?

woofie11 5y ago
No. Most proper cryptographic hash systems (e.g. used for verifying files, rather than data structures) never have collisions. Try to find a SHA256 collision. Anywhere, ever, in the history of mankind. This isn't for lack of looking. A lot of very smart people have looked for them. If you find one, I bet you'll be eligible for a tenured faculty slot at a good university, if not more. A whole world of secure systems would need to be re-engineered. Hypothetical collisions of course exist, by the pigeonhole principle, just not in the real world.
- StrLght 5y ago
  Yes, but cryptographic hashes are irrelevant here because they'd allow to easily bypass CSAM by modifying/appending a single byte.
  - woofie11 5y ago
    Apple is claiming to have a visual equivalent to a cryptographic hash -- one which won't change with a single byte, but only if the image is substantially different. At least their security analysis relies on that. From their whitepaper: "The threshold is selected to provide an extremely low (1 in 1 trillion) probability of incorrectly flagging a given account" If your claim is that their hash algorithm isn't cryptographic, their security analysis is incorrect.
    - kbelder 5y ago
      Their security analysis is obviously incorrect. "equivalent to a cryptographic hash" "change... only if the image is substantially different" Both are not true, cannot be true.

floor_ 5y ago

Why not combine this with a second different hash? E: Better yet, only run the second hash if you have a collision, which should be very rare.

theshrike79 5y ago
How do you know they aren't doing this on the backend after the initial on-device match?
yeldarb 5y ago
Apple does, and I created a proof of concept for how it might work to guard against adversarially perturbed images here: https://blog.roboflow.com/apples-csam-neuralhash-collision/ https://blog.roboflow.com/apples-csam-neuralhash-collision/

teekert 5y ago

I still think the biggest problem is that at some point a human is going to look at a false positive, this may be picture of my naked children and this human may not have the best intentions with my picture. That said, Nextcloud is my backend and I do not upload anything to iCloud (except for MS authenticator 2fa backups), so I'm safe right?

Grustaf 5y ago
Since the risk of that is 1 in a trillion, a lot of people are quite happy to take that risk.
- dathinab 5y ago
  It's not 1 in a trillion, it's MUCH higher. We are not speaking about a situation where not a "arbitrary" picture is miss-classifieds. We are speaking about a situation where a innocent picture involving a naked or not fully clothed child is deemed similar to a non innocent picture of a naked or not fully clothed child. Now you might argue that there should not be a picture a a naked or not fully clothed child of any form ever on any phone, but IMHO that is short sighed, discriminating and at best shows you don't know to much about the world and other cultures. Let's list some simple reasons such a think could happen first: - Photos meant for a doctor, or living partner to ask if something is normal or a problem. In many different ways. - Photos of little children bathing or similar a e.g. dad sends to their mom who is currently on a business trip. - etc. Reasons people are less aware of is that not all countries are as stuck up about nakedness especially in the family. So it's totally normal for families that e.g. before or after taking a shower family member independent of age and gender walk through the apartment naked. Similar if you didn't got any shame about the naked body indoctrinated you might totally do thinks like visiting a "naked-beach" with your family (meeting other families and taking advantage of it often being less crowded) and in turn normal innocent beach family pictures contain naked children. And on itself that's not a problem. But with Apples approach stuff like this is like to trigger both systems Apple announced and wrongly label your while family as pedophiles...
  - 5y ago

Grustaf 5y ago

This is of course entertaining, but since Apple has already tested for this, with 100 million images, and adjusted the rules accordingly, it has no practical implications.

Lamad123 5y ago

The problem is that a company is raping my privacy!! I don't care about the details of how good or bad is their spyware is!!

foxfluff 5y ago

My take on this is that the system is by and large useless. It won't catch anything but the dumbest of dumb criminals, because those who care about CSAM can surely figure out a better way to share images, or find a way to obfuscate their images enough to bypass the system (the lower the false positive rate, the easier it must be to trick the system). So what's left when all the criminals this is supposed to catch have figured it out? False positives. Only false positives. Is it really worth turning personal devices into snitches that don't even do a good job of protecting children? Also, numbers about false positives must be taken with a grain of salt because of the non-uniform distribution of perceptual hashes. It might be that your random vacation photos and kitty pics have a 1-in-a-million chance of a fapo, but someone who happens to (say) live in an apartment that has been laid out very similarly to a scene in pictures appearing in the CSAM database may have a massively higher chance of fapos for photos taken in their home.

Retric 5y ago
Perceptual hashes are only used to reduce the search space for human review. Apple doesn’t have images in the CSAM database to do a comparison, but if it’s just a picture of a door their going to reject it. Also, because human review is an expense Apple’s incentives are to minimize the number of times it happens, thus the requirement for multiple collisions.
- jdavis703 5y ago
  > Apple’s incentives are to minimize the number of times it happens, thus the requirement for multiple collisions. How can we be sure they won’t cut costs by increasing worker load? I could see them giving each reviewer less time to review individual pictures before passing it on to law enforcement.
  - fraa-orolo 5y ago
    We can't and they probably will, everyone else seems to already be doing so. There's that Swiss federal police report that only about 10% of NCMEC reports are actually relevant (https://fedpol.report/en/fedpol-in-figures/fight-against-paedophile-crime https://fedpol.report/en/fedpol-in-figures/fight-against-pae...)

JoshTko 5y ago

The threshold of collisions Apple is using before review is 40

dathinab 5y ago
I think no one is anymore afraid of 40 accidental natural image collisions. But un-natural image collisions or bad images in the database and similar are a different matter and had been the main critique point from the get to go as far as I can tell.
- dathinab 5y ago
  Also given how many people use IPhones, how many pictures they have and how often they have many similar pictures, thinks are not necessary that simple. I wouldn't be surprised if some flat, small height fully adult (e.g. 30) woman does some sexting and goes from 0 to >40 collisions in a month. Not because of arbitrary collisions but because the similarity some of here sexting pictures might have with the ones from a 14y old but older looking girl (which e.g. where forced and ended up in the database).

ya3r 5y ago

Technically speaking, this does not prove that an adversarial attack is possible on the CSAM system of apple, Given that apple has another not released neural hash system on their servers which is potentially larger and works better than the one on device. The more interesting technical question for me is: do collisions transfer across models? or how to find collisions that transfer across models?

tyingq 5y ago

I guess don't upload pictures of peaches, poppy buds, phallic cacti, and so on to iCloud.

nitrogen 5y ago

The technology is not why the Apple system is unwanted. It's just extra fuel for the fire. This system is unwanted because it puts a spy literally in your house and in your hands. It's bad enough that cloud everything blurs the line between what's yours and what's mine. Placing any law enforcement tech on a user's own device takes that line between "public" and "private" and completely erases it.

nullc 5y ago
Absolutely. The problem is Apple introducing a spy into your home. This alone should be bad enough, but some people are rather trusting. Showing that the spy is also tripping balls both exposes additional risks and emphasizes that Apple neither has their best interest at heart nor is putting adequate care into their actions. The latter gives people reason to question apple's claims of additional protection mechanisms that are non-falsifiable.

Cyberdog 5y ago

Isn't a hash collision from similar images the point of the whole thing? At any rate, IANAL, but I'm pretty sure you can't be convicted based on a hash alone. If you get busted for possession of a picture of a nematode and you can show the jury it's just a picture of an axe that has the same value when run through this algorithm, you'll be fine. And there's a decent chance prosecutors won't chase down individuals who will just have a single collision in their photo library with this tech in the first place - people who have dozens or hundreds will be much more interesting.

erdos4d 5y ago

Is it possible for the courts to use this system to search a defendant's phone for leaked documents say? Like if NSA learns that one of a small group leaked document X, can they get a court to force Apple to add the hash of Document X to the database on that group of people's phones? If so, I bet this becomes the new norm for investigating leaks.

gok 5y ago

The "catalog" has two entries.