6 ms·

The effort to fact check with LLMs is also high. Here's one from a few days ago. Someone used AI to generate an image in the style of a Charles Schulz Peanuts

by dalke 6mo ago

The effort to fact check with LLMs is also high. Here's one from a few days ago. Someone used AI to generate an image in the style of a Charles Schulz Peanuts cartoon. Someone else observed that there were 5 fingers on the characters, and quoted as Google AI as saying “Charlie Brown, along with other Peanuts characters, is generally depicted with four fingers on each hand (three fingers and one thumb) ...” Yet if you go to the Wikipedia entry at https://en.wikipedia.org/wiki/Peanuts https://en.wikipedia.org/wiki/Peanuts you'll see the kids have 5 fingers. Or take a look at the actual cartoons. Or read the TVTropes entry https://tvtropes.org/pmwiki/pmwiki.php/Main/FourFingeredHands https://tvtropes.org/pmwiki/pmwiki.php/Main/FourFingeredHand... under "Comic Strips". Fact checking this with human sources is easy and not ambiguous. While LLMs are being trained that many cartoon characters only have a thumb and three fingers - it is a trope for a reason - so isn't it logical for LLMs to give the wrong answer for a comic where the human characters are actually drawn with 5 fingers? My experience with LLMs is they keep getting things wrong, when details matter. Do you ask the LLM to fact check everything? (In which case, why isn't that part of the standard prompt?) Or do you only ask to fact check things where you are unsure about the answer? (In which case, is it the algorithm telling you what you want to hear?) When do you stop the fact checking?

lovelearning 6mo ago
> When do you stop the fact checking? Exactly the same calculus as fact checking anything else from any other source. What are the social/economic/ethical consequences to me if the answer is wrong or inaccurate or incomplete? How much time do I have to check? How thorough should I be? I imagine this calculus isn't really that different for most people. Or is it? As for your example, I believe it. But I also feel it's a rather outlier example involving image comprehension of an obscure factoid. That isn't typical of how I use LLMs which is mostly as text-based question answering engines and not what I had in mind when writing the comment. I guess LLMs for image comprehension need a much higher level of skepticism.
- dalke 6mo ago
  Well, in my case going to a Peanuts comic and looking at hands was pretty easy, and didn't involve any questions about negative environment or labor consequences, the massive hammering of web sites to gather data, centralization of power, and the like. Like, "!w Peanuts" in my search bar, look at the image, and count fingers. "a rather outlier example" You wrote that you use AI to find "obscure connections" - aren't those all by definition outliers? "mostly as text-based question" I just now asked Google AI "how many fingers are on charlie brown's hand?" It replied "In the Peanuts comic strip, Charlie Brown and the rest of the gang are traditionally drawn with four fingers (or three fingers and a thumb) on each hand." No image comprehension, exactly as you had in mind. And completely false. And that's from a training corpus which almost certainly includes statements that the kids are drawn with 5 fingers, since I confirmed that info on TVTropes and Reddit comments, like https://www.reddit.com/r/pics/comments/swod8/charlie_brown_has_4_fingers_and_a_thumb/ https://www.reddit.com/r/pics/comments/swod8/charlie_brown_h... .
  - lovelearning 6mo ago
    Even as a human, I find whatever sources Google shows to be inconsistent. I can't give any confident answer about the number of fingers. I think the answer is actually "4 sometimes and 5 other times." So I'm not sure how much LLMs can handle this kind of inconsistency between "reputable" visual sources and text sources, nor how representative this example is. A "reputable source" like Variety says this... https://variety.com/2015/film/spotlight/charlie-brown-steve-martino-peanuts-blue-sky-studios-1201657959/ https://variety.com/2015/film/spotlight/charlie-brown-steve-...: > “The rig would automatically move the features around so it would match the way Charles Schulz drew the character,” Heller says....In some drawings, Charlie Brown has just three fingers, while in others, he has five Images from another website... https://cartoonresearch.com/index.php/cartoons-at-bat-part-17-peanuts-at-the-plate/ https://cartoonresearch.com/index.php/cartoons-at-bat-part-1... : 1. https://cartoonresearch.com/wp-content/uploads/2025/09/Lost-Ballpark-new.jpg https://cartoonresearch.com/wp-content/uploads/2025/09/Lost-... -> 4 fingers 2. https://cartoonresearch.com/wp-content/uploads/2025/09/images.jpeg https://cartoonresearch.com/wp-content/uploads/2025/09/image... -> 4 fingers Anyway this wasn't the type of obscure connections I was referring to though I can understand you interpreting it that way. Personally I think this example supports what I said about "reputable sources." They can't be blindly trusted either because they may be inconsistent with each other and which one we choose to believe (Reddit.com or TVTropes.com or Variety.com) becomes entirely subjective.

by dalke 6mo ago

lovelearning 6mo ago

> When do you stop the fact checking? Exactly the same calculus as fact checking anything else from any other source. What are the social/economic/ethical consequences to me if the answer is wrong or inaccurate or incomplete? How much time do I have to check? How thorough should I be? I imagine this calculus isn't really that different for most people. Or is it? As for your example, I believe it. But I also feel it's a rather outlier example involving image comprehension of an obscure factoid. That isn't typical of how I use LLMs which is mostly as text-based question answering engines and not what I had in mind when writing the comment. I guess LLMs for image comprehension need a much higher level of skepticism.

dalke 6mo ago
Well, in my case going to a Peanuts comic and looking at hands was pretty easy, and didn't involve any questions about negative environment or labor consequences, the massive hammering of web sites to gather data, centralization of power, and the like. Like, "!w Peanuts" in my search bar, look at the image, and count fingers. "a rather outlier example" You wrote that you use AI to find "obscure connections" - aren't those all by definition outliers? "mostly as text-based question" I just now asked Google AI "how many fingers are on charlie brown's hand?" It replied "In the Peanuts comic strip, Charlie Brown and the rest of the gang are traditionally drawn with four fingers (or three fingers and a thumb) on each hand." No image comprehension, exactly as you had in mind. And completely false. And that's from a training corpus which almost certainly includes statements that the kids are drawn with 5 fingers, since I confirmed that info on TVTropes and Reddit comments, like https://www.reddit.com/r/pics/comments/swod8/charlie_brown_has_4_fingers_and_a_thumb/ https://www.reddit.com/r/pics/comments/swod8/charlie_brown_h... .
- lovelearning 6mo ago
  Even as a human, I find whatever sources Google shows to be inconsistent. I can't give any confident answer about the number of fingers. I think the answer is actually "4 sometimes and 5 other times." So I'm not sure how much LLMs can handle this kind of inconsistency between "reputable" visual sources and text sources, nor how representative this example is. A "reputable source" like Variety says this... https://variety.com/2015/film/spotlight/charlie-brown-steve-martino-peanuts-blue-sky-studios-1201657959/ https://variety.com/2015/film/spotlight/charlie-brown-steve-...: > “The rig would automatically move the features around so it would match the way Charles Schulz drew the character,” Heller says....In some drawings, Charlie Brown has just three fingers, while in others, he has five Images from another website... https://cartoonresearch.com/index.php/cartoons-at-bat-part-17-peanuts-at-the-plate/ https://cartoonresearch.com/index.php/cartoons-at-bat-part-1... : 1. https://cartoonresearch.com/wp-content/uploads/2025/09/Lost-Ballpark-new.jpg https://cartoonresearch.com/wp-content/uploads/2025/09/Lost-... -> 4 fingers 2. https://cartoonresearch.com/wp-content/uploads/2025/09/images.jpeg https://cartoonresearch.com/wp-content/uploads/2025/09/image... -> 4 fingers Anyway this wasn't the type of obscure connections I was referring to though I can understand you interpreting it that way. Personally I think this example supports what I said about "reputable sources." They can't be blindly trusted either because they may be inconsistent with each other and which one we choose to believe (Reddit.com or TVTropes.com or Variety.com) becomes entirely subjective.