11 ms·
Is anthropomorphizing a real problem? From what I know, none of the serious LLM researchers believe it has anything to do with human reasoning, apart from Anthr
by kgeist 27d ago
Is anthropomorphizing a real problem? From what I know, none of the serious LLM researchers believe it has anything to do with human reasoning, apart from Anthropic with their click-baity terminology like "LLM biology". It's just a metaphor. "Reasoning tokens" is simpler to say than "learned prompt augmentation tokens". I used to (and still do) anthropomorphize things long before LLMs, and I've seen my colleagues do it too. Say, when MySQL fails to start because it tries to read its config from the wrong dir, I may say "oh, this guy thinks he must read the config from ..." (having a language with grammatical genders as my native language also helps make it sound pretty natural). It's more fun like that :) Doesn't mean I genuinely believe a MySQL instance actually thinks.
- jurgenburgen 27d ago> It's more fun like that :) Doesn't mean I genuinely believe a MySQL instance actually thinks. A lot of people are not in on the joke. ELIZA effect and AI psychosis is a thing. Interacting a lot with LLMs might be damaging to the human psyche even for mentally stable people.
- Pannoniae 27d agoWhat's wrong with treating it as biology though? Even large software systems have biological aspects, their behaviour is emergent and if you want to observe how they work, a holistic approach is needed, you can't really reason about their full state... For example, if you have a search engine or a complex game, you can't run tests like "for all inputs the results are correct", you're going to be fudging a lot, using randomness, using heuristics, and all that kinda stuff Just like how mathematics > physics > chemistry > biology > psychology > economics/sociology (Auguste Comte's hierarchy reordered a bit for the modern day), moving up the abstraction ladder makes things more complex, less legible and less exact.
- JohnMakin 27d agoBecause plenty of people, even ones that should know better, really believe it's a conscious, thinking entity, not just some turn of phrase. I have a coworker that spends at least 10 hours a week arguing with his like you would with a conscious person. I've gently tried to explain it's like arguing with your compiler for giving you an incoherent error message - it's pointless. It doesn't understand, it can't understand, and even if it could, you arguing with it isn't going to make it "learn" or act differently.
- alexey-salmin 27d ago> It doesn't understand, it can't understand, and even if it could, you arguing with it isn't going to make it "learn" or act differently. I know people who are like that too. I'm not sure anthropomorphizing is a problem. Seeing analogies everywhere is an innate human trait, sometimes it can be harmful but more often it's useful.
- danaris 27d ago> I know people who are like that too. This is part of the problem being described. You are part of the problem. "Some people are bad at X" is not comparable—is not even in the same category—as "LLMs are fundamentally incapable of X". Every human (at least to a first approximation) is capable of understanding, of learning, of remembering things, of doing math, of counting the number of "r"s in "strawberry". What you are observing is that some humans are careless, do not take the time and effort to understand, or have internalized the idea that they're "not smart enough" or "not the type of person" who understands things like <whatever>. That has nothing remotely to do with the fact that LLMs have no consciousness, no self-awareness, no cognition, no understanding. At a fundamental level.
- JohnMakin 27d agoI don't think anyone in this conversation is saying this behavior is anything but the fault of the user not understanding how these tools work? This is a weirdly aggressive post.
- danaris 27d agoThis is an extremely common fallacy I've seen lots and lots of people fall into with respect to LLMs. In nearly every case, they use the fact that "some humans can't do X" to claim that LLMs are, in fact, basically conscious/human-like/AGI already. This is deeply untrue, and is highly likely to lead them to bad conclusions about what we can and should do with LLMs.
- Capricorn2481 27d ago> none of the serious LLM researchers believe it has anything to do with human reasoning But some of the biggest evangelists, who are well respected programmers that get lauded on this very site, have said it is fully sentient and has emotions. Even going back to 2022, when the LLMs were dogshit, a Google employee lost his job claiming it was sentient because it said it had emotions. Combine that with the marketing angle of both Anthropic and OpenAI, who have been trying their hardest to describe every function of an LLM as analogous to the human brain. Because it's politically useful to paint them as dangerous and uncontrollable, so the keys will only be granted to the few people on the mountaintop.
- doawoo 27d ago> Is anthropomorphizing a real problem? Yes it's really a problem. On this website you are surrounded by people who have technical knowledge and understand at least somewhat, how a computer functions. You have the ability to separate "fun" and "reality" because you know you're putting input into a really really big calculator. Most people do not fathom this. AI Psychosis is a real thing, look it up (don't just ask an LLM) and do some reading. It's actively harming people, and the way they think. There's no regulation around any of this stuff and it drives me crazy that we let these AI companies _sprint_ so far ahead of everyone, and now we're facing the consequences.
- mikehollinger 27d ago> Is anthropomorphizing a real problem Yes. There's a difference between scrapping a session and starting over, or going back and branching something, or using sub-agents to see five outcomes, vs arguing with a system in a long drawn out chat. Like - I know that if a model starts doing something silly, instead of correcting it - I can probably go back and edit two steps prior to add an extra guardrail, or extra data, or whatever.
- trs83 27d agoSimplifying terminology is not a problem. The providers intentionally choosing terminology to make people think it's something it's not is a problem. I hate the term agent. Calling them companions as some do is just gross.
- red75prime 27d agoWhen I was taking an MIT AI course (in ancient pre-LLM times), an autonomous agent was defined as a system that perceives its environment and acts on it (we were focusing on reward-expectation-maximizing agents, but it's not that important). Peter Norvig has said something like, technically, anything can be described as an agent (a rock maximizes the "follow physical laws" objective), but naturally, it doesn't make much sense to model a rock as an agent. With AI agents, the situation is significantly less controversial: they do perceive, deliberate, and act.
- brookst 27d agoAll of these terms were picked by individuals, years ago, while reaching for metaphors that made sense to them personally. None of these "agent" / "thinking" / "reasoning" terms were dreamed up in boardrooms to intentionally mislead people. They are useful but faulty metaphors; there is no conspiracy.
- lern_too_spel 27d ago"Agent" is standard reinforcement learning terminology, used in Chris Watkins's thesis introducing Q-Learning in 1989.
- ergl 27d ago> Is anthropomorphizing a real problem? The paper argues that pretending that the so-called thinking traces represent real reasoning can lead users into trusting wrong answers, if the thinking traces appear convincing enough. Researchers might inspect these traces to try to determine the “intent” of a model, as well. For an example of the latter, when OpenAI spoke about the hacking of HuggingFace at Black Hat, they repeatedly showed the thinking traces of their model as “proof” of what the model was “thinking” as it performed the attack, calling out “surprise” moments, etc. Now, it’s possible that the employees presenting didn’t truly believe that the thinking traces would give them useful clues, and presented them only for a “wow” factor, but I wouldn’t discount the possibility that even the people working at frontier companies can fall for this tendency to anthropomorphize LLMs.
- brookst 27d agoBut how is that any different than people being misled by real humans saying words that reflect real thinking, but which are actually dead wrong? The fallacy here is "thinking == correct", not "tokens == thinking"
- randomImmigrant 27d agoBecause the real thinking still cost the other human the same-ish energy it costs you to put words together, and because after all, the source is a human and not a machine, no, this is very different. Being mislead may be the shared outcome. But why is different category of source of the mistake and the cost to producer of making the mistake not relevant in this discussion? Where else in science do you brush aside all differences this way?
- lern_too_spel 27d agoWhy aren't humans simply biological machines? There is no "science" that GP is brushing aside. You need to provide repeatable observations or experiments that GP is ignoring.
- 27d ago
- pera 27d agoYes it is a very serious problem because it confuses a lot of folks with a great deal of power like judges and policymakers. The first book I ever read on ML (late 90s) dedicated the entire first or second chapter exploring the distinctions between artificial and biological neurons, and even talked a bit about the philosophy of modelling. I still remember thinking back then why would the authors spend so many pages on this but now I believe it was because they understood that a metaphor can be a double-edged sword.
- ux266478 27d agoTo be fair, the ANN architecture underneath is a misleading thing to be looking at, it's not where the comparison comes from. Though I can't tell if you meant it to be relevant in that way, or just as a general example for the dangerous nature of metaphor. LLMs are expressly designed to approximate human behavior within the bounds of the written word. The anthropomorphization is no more philosophically problematic than saying differential calculus measures curves.
- pera 27d agoI meant it in the latter way: a metaphor can be useful as a pedagogical tool to introduce new ideas, and using the source of inspiration for this idea as the metaphor itself makes perfect sense, but unfortunately our brains seem to be prone to assign other properties of the metaphor that don't actually belong to the object of study. I imagine this happens because we tend to conflate things that are similar, or maybe because it's not entirely clear which characteristics are being mapped in the metaphor?
- lern_too_spel 27d agoThey spent so many pages discussing it only to show that the mechanism for how ANNs work is different from the mechanism for how biological brains work. It says nothing about whether they can compute the same things.
- eli_gottlieb 27d agoI find it annoying because when I read ML papers nowadays I have to back-translate from anthropomorphized talk into actual machine talk, then mentally compare to what I actually know about brains and cognition.
- taurath 27d ago> Is anthropomorphizing a real problem? Even tech companies are rolling out AI training which utterly anthropomorphizes it, and leads people to think its actually intelligence. This is part of the reason for the backlash - everyone understands it bullshit marketing the second you actually try to use it.
- frrlpp 27d ago"Esta poronga no sabe lo que está haciendo", aunque parezca femenino , en realidad no tiene género .
- kelnos 27d ago> Is anthropomorphizing a real problem? Yes. Actual real people believe that LLMs are actual, thinking, intelligences, perhaps even with consciousness. Your bit with MySQL is harmless because it's obvious that a database isn't a sentient lifeform. But LLMs can look like they're the real deal, and people believe it is. Using terminology like "thinking" and "reasoning" to describe what they do only reinforces this. Having said that, I agree with you on the terminology front: I'm not going to say "learned prompt augmentation tokens" either.
- YawningAngel 27d agoI think LLMs are pretty clearly intelligent in some sense of the word, and I don't know how one could ever confidently know they aren't conscious in some sense. That's not to say they're humanlike, just that people who think they know these ideas are ridiculous seem to be overreaching in the same way Steve Yegge seems to be overreaching
- mdp2021 27d ago> conscious The problem with that term is that it is hardly meaningful. Why would you use it? It does not add much, and no clarity, to what it is attributed to.
- Otterly99 27d agoI actually think on the LLM side it might be beneficial to refer to them as <thinking> because it explicitely guide the token generation towards a "thinking space". As weird as it is, anthropomorphizing LLMs in prompts has been actually pretty useful (think of the latest big math discoveries which were achieved by having the user giving supporting words). It would be interesting to see if a LLM would perform worse if you used a more neutral term. The paper's argument is rather than using terms like "thinking trace" can lead people to believe that the model is really thinking, and thus these traces can be used as a sort of interpratbility parameter. This can give a false sense of security when building a LLM-based system which requires guardrails and tracability.
- geraneum 27d ago> Is anthropomorphizing a real problem? Of course it is. Anthropomorphizing is in our nature, but it doesn’t mean we have to entertain it and extend it to everything. A poet can anthropomorphize clouds beautifully and I’d enjoy his poem, but I want my pilot to not see clouds as rabbits when they decide if it’s safe to fly through them.
- DarmokTanagra 27d ago[dead]
- jillesvangurp 27d ago> Is anthropomorphizing a real problem? Completely rational and smart people talk to their pets, plants, their car, and other inanimate objects. This is not considered abnormal by most. It's just what we are wired to do. Some LLMs are uncannily good at tricking people into believing they are talking to a real person. So, there is that as well. Some people are a bit freaked out by this or still somewhat in denial about LLMs being this good. But people have been yelling at their computers for as long as we've had them; so that ship sailed a long time ago. Trying to stop them doing that is probably a bit futile. Whether people like this or not, LLMs are actually trained and fine tuned on real conversations and that's where a lot of this is re-enforced. Instead of fighting that, you can just lean into it and accept that communicating like you would with a person totally works and can actually be efficient even as it requires less effort and thinking on your side. You can go all Jean Luc Picard on AIs and yell "Tea! Earl Grey Hot!" or you can just ask "I'd like a cup of tea, please". LLMs are good at remembering your tea preference. The please is of course completely redundant and should not affect the outcome. If you just want a cup of tea, you should be fine either way.
- micromacrofoot 26d agothere are people on the fringes in relationships with these things, so yes definitely