7 ms·

I think my "cautious optimism" mostly came from the fact that Gemini found subtle problems with the counterfeit packages that I (someone who looks at fakes as p

by wgrover 19d ago

I think my "cautious optimism" mostly came from the fact that Gemini found subtle problems with the counterfeit packages that I (someone who looks at fakes as part of my job) didn't notice - like mismatched info between the tube and box, and a malformed Irish postal code. If I didn't notice those, then I seriously doubt that a consumer glancing at a package would notice them. In that case, Gemini's insights would be valuable info for a consumer. Gemini wrongly called an authentic product a fake, but that was mainly because there really were typographical errors in the authentic product's ingredient list. That's more of an indictment of the manufacturer than it is Gemini. Finally, a lot of the things that Gemini got wrong seemed to me like they could reasonably be attributed to things like optical artifacts in the photos (glare, shadows, stuff like that). Better/more photos might improve that. All that being said, this is obviously a tiny study of a single AI tool with a single brand of cosmetic product, so it's probably premature for me to optimistic, even cautiously. I've edited the last section of the writeup accordingly.

hn_throwaway_99 19d ago
Thanks for the response, and I was overly harsh, I apologize. I did appreciate the detail and thoroughness you went to explain your experiences. Yet I still think your study was a microcosm of the greatest dangers I think about AI. That is, it was astoundingly good at doing small scale feature detection, but astoundingly bad at synthesizing an overall conclusion, and worse, it did so with characteristic "AI certainty". Also, in the real world just like you found, pictures have glare, and real manufacturers have mistakes. The horrifyingly scary thing is that even if you think Gemini did a fairly good job at feature detection, in the real world people have and will just follow the AI conclusions blindly because they're "mostly" correct, even when they lead to completely wrong outcomes. Heck, a Tesla already killed its passenger when it rammed into the side of a truck a few years ago because it mistook glare on the truck for the sun. Military planners blew up a school of young children based on old data, yet the Pentagon tried to blacklist Anthropic because Anthropic didn't want to provide autonomous kill capabilities. I don't mean to sound over dramatic, but again, to me your study highlights everything I think is wrong with AI and the extreme dangers it will cause if society relies on it too much, which it has already begun to do.

by wgrover 19d ago

hn_throwaway_99 19d ago

Thanks for the response, and I was overly harsh, I apologize. I did appreciate the detail and thoroughness you went to explain your experiences. Yet I still think your study was a microcosm of the greatest dangers I think about AI. That is, it was astoundingly good at doing small scale feature detection, but astoundingly bad at synthesizing an overall conclusion, and worse, it did so with characteristic "AI certainty". Also, in the real world just like you found, pictures have glare, and real manufacturers have mistakes. The horrifyingly scary thing is that even if you think Gemini did a fairly good job at feature detection, in the real world people have and will just follow the AI conclusions blindly because they're "mostly" correct, even when they lead to completely wrong outcomes. Heck, a Tesla already killed its passenger when it rammed into the side of a truck a few years ago because it mistook glare on the truck for the sun. Military planners blew up a school of young children based on old data, yet the Pentagon tried to blacklist Anthropic because Anthropic didn't want to provide autonomous kill capabilities. I don't mean to sound over dramatic, but again, to me your study highlights everything I think is wrong with AI and the extreme dangers it will cause if society relies on it too much, which it has already begun to do.