6 ms·

> Is anthropomorphizing a real problem? The paper argues that pretending that the so-called thinking traces represent real reasoning can lead users into trusti

by ergl 27d ago

> Is anthropomorphizing a real problem? The paper argues that pretending that the so-called thinking traces represent real reasoning can lead users into trusting wrong answers, if the thinking traces appear convincing enough. Researchers might inspect these traces to try to determine the “intent” of a model, as well. For an example of the latter, when OpenAI spoke about the hacking of HuggingFace at Black Hat, they repeatedly showed the thinking traces of their model as “proof” of what the model was “thinking” as it performed the attack, calling out “surprise” moments, etc. Now, it’s possible that the employees presenting didn’t truly believe that the thinking traces would give them useful clues, and presented them only for a “wow” factor, but I wouldn’t discount the possibility that even the people working at frontier companies can fall for this tendency to anthropomorphize LLMs.

brookst 27d ago
But how is that any different than people being misled by real humans saying words that reflect real thinking, but which are actually dead wrong? The fallacy here is "thinking == correct", not "tokens == thinking"
- randomImmigrant 27d ago
  Because the real thinking still cost the other human the same-ish energy it costs you to put words together, and because after all, the source is a human and not a machine, no, this is very different. Being mislead may be the shared outcome. But why is different category of source of the mistake and the cost to producer of making the mistake not relevant in this discussion? Where else in science do you brush aside all differences this way?
  - lern_too_spel 27d ago
    Why aren't humans simply biological machines? There is no "science" that GP is brushing aside. You need to provide repeatable observations or experiments that GP is ignoring.
    - randomImmigrant 27d ago
      Please define "simply biological machines". I'm not sure "biological machine" had a proper definition. What's machine like about biology exactly?
      - lern_too_spel 27d ago
        A "machine" is a term that is well defined in science. https://en.wikipedia.org/wiki/Machine https://en.wikipedia.org/wiki/Machine
        randomImmigrant 26d ago
        And biological machine is? Don’t get me wrong. Biology I is full of molecules that we call machines. But you’re making a broader claim, saying that biology is only this. This needs you to answer some questions: 1. Why do the machine parts in biology show such flexible application? A gear cog won’t ever moonlight as a signaling chip, but in biology you often have molecules doing double and triple duty. 2. How is the biological machine able to build itself? What does self assembly imply for the machines function? 3. Where does this machine get its inner drive? No LLM has been found that starts outputting text unprompted. A car doesn’t decide to move to a shady parking spot. Why? Where in the machine to biological machine continuum does the ability to make internally driven decisions come in? Why does it come in for biology? A bacterium is able to make such agentic decisions unprompted. Why is no manufactured machine able to do this?
        lern_too_spel 26d ago
        > And biological machine is? "Biological," too, is well defined. It relates to living things and their processes. > Why do the machine parts in biology show such flexible application? Evolution. > A gear cog won’t ever moonlight as a signaling chip A gear cog was purpose built for that purpose, but you will find that people often recycle parts into other systems, often in completely different roles. > How is the biological machine able to build itself? Protein synthesis. > What does self assembly imply for the machines function? The way that a machine is built has no bearing on how the machine functions. I could build the same machine using a 3d printer or a CNC router. > Where does this machine get its inner drive? Evolution selected for organisms that survive long enough to reproduce. Different biological systems handle this differently. > No LLM has been found that starts outputting text unprompted. If you give an agent a goal, it will perform actions to achieve that goal. This is just as true for artificial agents as it is for biological agents. > Why is no manufactured machine able to do this? Many do. Even robotic vacuum cleaners will charge themselves without human prompting.
ux266478 27d ago
FWIW I think the paper's argumentation is extremely weak to begin with. Like in section 4.1, it opens by expressing a sound position of skepticism: > there are significant questions on whether these traces have any valid semantic import to the end user. Which it contradicts in the very next paragraph, taking a stance that there are no valid semantics present in the trace: > the false idea that derivational traces are semantically meaningful It's really not a high quality paper worth taking seriously. And that's before we get into the complete and total breakdown of objective analysis. It rejects distributional semantics as a theory, while also explicitly stating the results that have been produced under its auspices are "undeniable". Never elaborated on, and at no point in the paper am I given the impression the authors are even aware of the problem with this. It's just more unempirical slop that wants its pound of flesh without putting the work in. Frankly, whoever let this through peer review should be ashamed of themselves.

by ergl 27d ago

brookst 27d ago

But how is that any different than people being misled by real humans saying words that reflect real thinking, but which are actually dead wrong? The fallacy here is "thinking == correct", not "tokens == thinking"

randomImmigrant 27d ago
Because the real thinking still cost the other human the same-ish energy it costs you to put words together, and because after all, the source is a human and not a machine, no, this is very different. Being mislead may be the shared outcome. But why is different category of source of the mistake and the cost to producer of making the mistake not relevant in this discussion? Where else in science do you brush aside all differences this way?
- lern_too_spel 27d ago
  Why aren't humans simply biological machines? There is no "science" that GP is brushing aside. You need to provide repeatable observations or experiments that GP is ignoring.
  - randomImmigrant 27d ago
    Please define "simply biological machines". I'm not sure "biological machine" had a proper definition. What's machine like about biology exactly?
    - lern_too_spel 27d ago
      A "machine" is a term that is well defined in science. https://en.wikipedia.org/wiki/Machine https://en.wikipedia.org/wiki/Machine
      - randomImmigrant 26d ago
        And biological machine is? Don’t get me wrong. Biology I is full of molecules that we call machines. But you’re making a broader claim, saying that biology is only this. This needs you to answer some questions: 1. Why do the machine parts in biology show such flexible application? A gear cog won’t ever moonlight as a signaling chip, but in biology you often have molecules doing double and triple duty. 2. How is the biological machine able to build itself? What does self assembly imply for the machines function? 3. Where does this machine get its inner drive? No LLM has been found that starts outputting text unprompted. A car doesn’t decide to move to a shady parking spot. Why? Where in the machine to biological machine continuum does the ability to make internally driven decisions come in? Why does it come in for biology? A bacterium is able to make such agentic decisions unprompted. Why is no manufactured machine able to do this?
        lern_too_spel 26d ago
        > And biological machine is? "Biological," too, is well defined. It relates to living things and their processes. > Why do the machine parts in biology show such flexible application? Evolution. > A gear cog won’t ever moonlight as a signaling chip A gear cog was purpose built for that purpose, but you will find that people often recycle parts into other systems, often in completely different roles. > How is the biological machine able to build itself? Protein synthesis. > What does self assembly imply for the machines function? The way that a machine is built has no bearing on how the machine functions. I could build the same machine using a 3d printer or a CNC router. > Where does this machine get its inner drive? Evolution selected for organisms that survive long enough to reproduce. Different biological systems handle this differently. > No LLM has been found that starts outputting text unprompted. If you give an agent a goal, it will perform actions to achieve that goal. This is just as true for artificial agents as it is for biological agents. > Why is no manufactured machine able to do this? Many do. Even robotic vacuum cleaners will charge themselves without human prompting.

ux266478 27d ago

FWIW I think the paper's argumentation is extremely weak to begin with. Like in section 4.1, it opens by expressing a sound position of skepticism: > there are significant questions on whether these traces have any valid semantic import to the end user. Which it contradicts in the very next paragraph, taking a stance that there are no valid semantics present in the trace: > the false idea that derivational traces are semantically meaningful It's really not a high quality paper worth taking seriously. And that's before we get into the complete and total breakdown of objective analysis. It rejects distributional semantics as a theory, while also explicitly stating the results that have been produced under its auspices are "undeniable". Never elaborated on, and at no point in the paper am I given the impression the authors are even aware of the problem with this. It's just more unempirical slop that wants its pound of flesh without putting the work in. Frankly, whoever let this through peer review should be ashamed of themselves.