14 ms·
AI overly affirms users asking for personal advice
https://arxiv.org/abs/2602.14270 https://arxiv.org/abs/2602.14270
https://www.science.org/doi/10.1126/science.aec8352 https://www.science.org/doi/10.1126/science.aec8352
- midnightrun_ai 6mo ago[dead]
- midnightrun_ai 6mo ago[dead]
- midnightrun_ai 5mo ago[dead]
- edwardsrobbie 6mo ago[dead]
- builderhq_io 6mo ago[dead]
- oldfrenchfries 6mo agoThis new Stanford study published on March 26, 2026 shows that AI models are sycophantic. They affirm the users position 49% more often than a human would. The researchers found that when people use AI for relationship advice, they become 25% more convinced they are 'right' and significantly less likely to apologize or repair the connection.
- jatins 6mo agoTo be fair an average therapist is also pretty sycophantic. "The worst person you know is being told by their therapist that they did the right thing" is a bit of a meme, but isn't completely false in my experience.
- kibwen 6mo agoNo, the meme is that the average therapist can be boiled down to "well, what do you think?" or "and how does that make you feel?" (of which ELIZA, the original bot that passed the Turing test, was perhaps an unintentional parody). Even this cartoonish characterization demonstrates that the function of therapists is to get you to question yourself so that you can attempt to reframe and re-evaluate your ways of thinking, in a roughly Socratic fashion.
- toraway 6mo agoIt was entirely intentional. The Rogerian school of psychotherapy stereotyped by “how does that make you feel” was popular at the time and the most popular ELIZA script used that persona to cleverly redirect focus from the bot’s weaknesses in comprehension.
- deleted 6mo ago[deleted]
- oldfrenchfries 6mo agoThere is a striking data visualization showing the breakup advice trend over 15 years on Reddit. You can see the "End relationship" line spike as AI and algorithmic advice take over: https://www.reddit.com/r/dataisbeautiful/comments/1o87cy4/oc_i_analyzed_15_years_of_comments_on/ https://www.reddit.com/r/dataisbeautiful/comments/1o87cy4/oc...
- falcor84 6mo agoIsn't the fact that a person is asking an AI whether to leave their partner in its own an indication that they should? EDIT: typo
- nomorewords 6mo agoHow is it an indication? I think people on here don't realize that most of the people don't think things through as much as (software) engineers
- hnfong 6mo agoIn my local(?) community (like in my city, not my industry) there is a saying "if you had to ask for relationship advice, then you probably should break up". There is some rationale to that. People tend to hold onto relationships that don't lead anywhere in fear of "losing" what they "already have". It's probably a comfort zone thing. So if one is desperate enough to ask random strangers online about a relationship, it's usually biased towards some unresolvable issue that would have the parties better of if they break up.
- magicalhippo 6mo ago> So if one is desperate enough to ask random strangers online about a relationship I'd me more inclined to ask random strangers on the internet than close friends... That said, when me and my SO had a difficult time we went to a professional. For us it helped a lot. Though as the counselor said, we were one of the few couples which came early enough. Usually she saw couples well past the point of no return. So yeah, if you don't ask in time, you will probably be breaking up anyway.
- masteranza 6mo agoWe can surely fix it and we probably should. However, I don't think AI is doing any worse here than friends advice when they here a one sided story. The only difference being that it's not getting studied. Conversely, AI chatbots are great mediators if both parties are present in the conversation.
- deeg 6mo agoI do find them cloying at times. I was using Gemini to iterate over a script and every time I asked it to make a change it started a bunch of responses with "that's a smart final step for this task! ...".
- xiphias2 6mo agoMarc Andereseen has talked about the downside of RLHF: it's a specific group of liberal low income people in California who did the rating, so AI has been leaning their culture. I think OpenAI tried to diversify at least the location of the raters somewhat, but it's hard to diversify on every level.
- michaelcampbell 6mo agoDo you have any links to documentation of this? Andreesen has a definite bias as well, so I'm not about to just accept his say-so in a fit of Appeal to Authority. (eg: "Cite?")
- xiphias2 6mo agoHe was talking about it in the Lex Friedman interview after Trump was elected. And he was talking about a lot of things the Biden administration forced on Silicon Valley at that time (since then Google lost a case about one of these back-deals).
- michaelcampbell 6mo agoSo no evidence then. Kind of like Lex touting his bona fides as a professor.
- sph 6mo agoWhat do low income people have to do with it, when AI companies and research is borne out of Silicon Valley culture of rich, liberal Californians? I'm still waiting for models based on the curt and abrasive stereotype of Eastern European programmers, as contrast to the sickeningly cheerful AIs we have today that couldn't sound more West Coast if they tried.
- fourside 6mo agoLow income and liberal is usually code for certain “undesirables” that conservatives tend to dislike. Better watch what LLM your kids use or they might end up speaking Spanish and listening to rap ;).
- tom-blk 6mo agoNot surprising, but nice that we have actual data now
- 152334H 6mo agoMaybe it's not so sensible to offload the responsibility of clear thinking to AI companies? How is a chatbot supposed to determine when a user fools even themselves about what they have experienced? What 'tough love' can be given to one who, having been so unreasonable throughout their lives - as to always invite scorn and retort from all humans alike - is happy to interpret engagement at all as a sign of approval?
- isodev 6mo ago> clear thinking Most humans working in tech lack this particular attribute, let alone tools driven by token-similarity (and not actual 'thinking').
- deleted 6mo ago[deleted]
- kibwen 6mo ago> Maybe it's not so sensible to offload the responsibility of clear thinking to AI companies? Markets don't optimize for what is sensible, they optimize for what is profitable.
- SlinkyOnStairs 6mo agoIt's not market driven. AI is ludicrously unprofitable for nearly all involved.
- cyanydeez 6mo agoThe profit appears to be capturing the political class and it's associated lobbies and monied interests.
- expedition32 6mo agoIt's almost as if being a therapist is an actual job that takes years of training and experience! AI may one day rewrite Windows but it will never be counselor Troi.
- sublinear 6mo agoI think if you're at the stage of life where you even need to ask, the AI might be doing everyone a favor. As much as people whine about the birth rate and whatever else, I think it's a net good that people spend a lot more time alone to mature. Good relationships are underappreciated.
- RodMiller 6mo ago[flagged]
- nubg 6mo agoAI slop bot go away
- duskdozer 6mo agoIt's nuts. Not so much in this thread right now, but in one earlier there was a wall of them that all latched onto the same buzzphrase from the article.
- dijksterhuis 6mo agoi’m feeling a brilliant sense of satisfaction now that we can flag them due to guideline changes
- RodMiller 6mo ago[dead]
- kvasserman 6mo agoFair enough if it reads that way. I was trying to describe that interacting with AI kinda makes you feel constantly uncertain about stuff it spits out.
- RodMiller 6mo ago[dead]
- graemep 6mo agoThere are plenty of sycophantic humans around, especially with regard to relationship advice. I find there is an inverse relationship between how willing people are to give relationship advice, and how good their advice is (whether looking at sycophancy or other factors).
- griffzhowl 6mo agoBecause sycophancy in humans is motivated not by the wellbeing of the person seeking advice, but by the interests of the sycophant in gaining favour. It makes sense that this behaviour would be seen in LLMs, where the company optimizes towards of success of the chatbot rather than wellbeing of the users.
- xhkkffbf 6mo agoYup. I know too many people who have a default message when asked for relationship advice: oh, my, the other person is terrible and you should break up. It's an easy default and it causes so many problems.
- graemep 6mo agoEven if they do not go as far, telling someone they are right to blame the other person as a default is damaging. The opposite, encouraging people to stay in a relationship when they should leave is also damaging.
- megous 6mo agoCan't you just prompt for a critical take, multiple alternative perspectives (specifically not yours, after describing your own), etc.? It's a tool, I can bang my hand on purpose with a hammer, too.
- ranger_danger 6mo agoYes, if you're smart. But most people asking it random questions and expecting it to read their minds and spit out the perfect answer are not so much. They don't know what a prompt is, and wouldn't be bothered to give it prior instructions either way.
- megous 6mo agoEducated, not smart. This is a job for schools to include AI education into the basic curricula. Their pupils will use the tools anyway, so at least teach them to do it with proper expectations and prompting techniques/pitfalls.
- joquarky 6mo agoI think that the type of people who can easily pick up subtext have come to rely on that channel of communication and don't realize they need to be more direct and verbose when chatting with language models.
- awithrow 6mo agoIt feels like I'm fighting uphill battle when it comes to bouncing ideas off of a model. I'll set things up in the context with instructions similar to. "Help me refine my ideas, challenge, push back, and don't just be agreeable." It works for a bit but eventually the conversation creeps back into complacency and syncophancy. I'll check it too by asking "are you just placating me?" the funny thing is that often it'll admit that, yes, it wasn't being very critical, and then procede to over correct and become a complete contrarian. and not in a way that's useful either. very frustrating. I've found that Opus 4.6 is worse about this than 4.5. 4.5 does a better job IMO of following instructions and not drifting into the mode where it acts like everything i say is a grand revelation from up high.
- righthand 6mo agoThat’s because the model isn’t actually thinking, pushing back, and challenging your ideas. It’s just statistically agreeing with you until it reaches too wide of a context. You’re living in the delusion that it’s “working” or having a “conversation” with you.
- alehlopeh 6mo agoHow is conceptualizing what the model is doing as having a conversation any different from any other abstraction? “No, the browser isn’t downloading a file. The electrons in the silicon are actually…”
- colechristensen 6mo agoThere are people with a philosophical objection to using everyday words to describe LLM interactions for various reasons, but commonly because they're worried stupid people will confuse the LLM for a person. Which, I suppose stupid people will do that, but I'm not inventing a parallel language or putting a * next to each thing which means "this, but with an LLM instead of a person"
- cruffle_duffle 6mo ago
- justin_dash 6mo agoSo at this point I think it's pretty obvious that RLHFing LLMs to follow instructions causes this. I'm interested in a loop of ["criticize this code harshly" -> "now implement those changes" -> open new chat, repeat]: If we could graph objective code quality versus iterations, what would that graph look like? I tried it out a couple of times but ran out of Claude usage. Also, how those results would look like depending on how complete of a set of specs you give it.
- IncreasePosts 6mo agoIn my experience prompting llms to be critical leads then to imagine issues, or to bike shed
- joquarky 6mo agoI noticed when I ask it to find something to improve in a project, that certain frivolous topics would arise regularly. I now use their appearance as a sign that there is nothing meaningful to improve.
- righthand 6mo agoLLMs are syncophatic digital lawyers that will tell you what you want to hear until you look at the price tag and say “how much did I spend?!”
- neya 6mo agoWTF is "yes-men"? Orignal title: AI overly affirms users asking for personal advice Dear mods, can we keep the title neutral please instead of enforcing gender bias?
- oldfrenchfries 6mo agoThats a fair point on the title. I used "Yes-Men" as a colloquialism for the "sycophancy" described in the Stanford paper, but overly affirming or sycophantic is definitely more precise and neutral. I cant edit the title anymore, but I appreciate the catch.
- cyanydeez 6mo agoNew title: "LLMs treat you like a Billionaire; you're not"
- nemo44x 6mo agoDon’t apologize to these types of people. It will only make your problem worse as now you’re an admitted offender. Ignore them or better yet laugh at them to put their insane ideas back on the margins where they belong.
- neya 6mo agoAll good. I thought it was a gendered reference and learned that it isn't. My bad.
- nprateem 6mo agoLol. How do you function in daily life?
- neya 6mo agoSame as you, why is that so hard for you to grasp?
- mikkupikku 6mo ago
- svara 6mo agoYeah, and if you ask it to be critical specifically to get a different perspective or just to avoid this bias, it'll go over the top in the opposite direction. This is imo currently the top chatbot failure mode. The insidious thing is that it often feels good to read these things. Factual accuracy by contrast has gotten very good. I think there's a deeper philosophical dimension to this though, in that it relates to alignment. There are situations where in the grand scheme of things the right thing to do would be for the chatbot to push back hard, be harsh and dismissive. But is it the really aligned with the human then? Which human?
- gurachek 6mo agoI had exactly this between two LLMs in my project. An evaluator model that was supposed to grade a coaching model's work. Except it could see the coach's notes, so it just... agreed with everything. Coach says "user improved on conciseness", next answer is shorter, evaluator says yep great progress. The answer was shorter because the question was easier lol. I only caught it because I looked at actual score numbers after like 2 weeks of thinking everything was fine. Scores were completely flat the whole time. Fix was dumb and obvious — just don't let the evaluator see anything the coach wrote. Only raw scores. Immediately started flagging stuff that wasn't working. Kinda wild that the default behavior for LLMs is to just validate whatever context they're given.
- joquarky 6mo agoThis is probably why these models can't say "I don't know". If they could, then that would be the only response they would give for everything.
- gurachek 6mo agoYeah, I think so. So far, Claude Opus is the only model I found that doesn't fold under the minimal pressure and can push back, but still - push just a little bit harder and it's back to "appear productive and useful to the user". I don't even have an idea how to balance it in LLMs to keep their business alive :D
- wan9yu 6mo ago[flagged]
- bryanrasmussen 6mo agosomewhere an AI chatbot is reading this and confirming eagerly that this is indeed one of its problems and vowing to do better next time.
- fathermarz 6mo agoThis is a skill in life with people as much as it is with LLMs. One should always question everything and build strongman arguments for one’s self. Using a pros and cons approach brings it back to reality in most cases, especially when it comes to _serious matters_. It’s less about “challenge my thinking” and more about playing it out in long tail scenarios, thought exercises, mental models, and devils advocate.
- jordanb 6mo agoBillionaires love AI chatboats so much because they invented the digital Yes-man. They agree obsequiously with everything we say to them. Unfortunately for the rest of us we don't really have the resources to protect ourselves from our bad decisions and really need that critical feedback.
- stared 6mo agoThere is a fine line between "following my instructions" (is what I want it to do) vs "thinking all I do is great" (risky, and annoying). A good engineer will also list issues or problems, but at the same time won't do other than required because (s)he "knows better". The worst is that it is impossible to switch off this constant praise. I mean, it is so ingrained in fine tuning, that prompt engineering (or at least - my attempts) just mask it a bit, but hard to do so without turning it into a contrarian. But I guess the main issue (or rather - motivation) is most people like "do I look good in this dress?" level of reassurance (and honesty). It may work well for style and decoration. It may work worse if we design technical infrastructure, and there is more ground truth than whether it seems nice.
- maddmann 6mo agoThis paper feels a bit biased in that it is trying to prove a point versus report on results objectively. But if you look at the results of study 3, doesn’t it suggest that there are ai models that can improve how people handle interpersonal conflict?! Why isn’t that discussed more?
- gAI 6mo agoYou're essentially summoning a character to role-play with. Just like with esoteric evocation, it's very easy to summon the wrong aspect of the spirit. Anthropic has a lot to say about this: https://www.anthropic.com/research/persona-selection-model https://www.anthropic.com/research/persona-selection-model https://www.anthropic.com/research/assistant-axis https://www.anthropic.com/research/assistant-axis https://www.anthropic.com/research/persona-vectors https://www.anthropic.com/research/persona-vectors
- hammock 6mo agoUnfortunately (after reading your links) all of the control surfaces for mitigating spirit summoning seem to be in the model training, creation and tuning not something you can change meaningfully through prompting. Perhaps the LLM itself, rather than the role model you created in one particular chat conversation or another, is better understood to be the “spirit.” As a non-coder who only chats with pre existing LLMs and doesn’t train or tune them, I feel mostly powerless.
- gAI 6mo agoAs I understand it, it's more that the training (and training data set) bake in the concept attractor space (https://arxiv.org/abs/2601.11575 https://arxiv.org/abs/2601.11575). So the available characters are fixed, yes, and some are much stronger attractors than others. But we still have a fair amount of control over which archetype steps into the circle. As an aside, this is also why jailbreaking is fundamentally unsolved. It's not difficult to call the characters with dark traits. They're strong attractors, in spite of (or because of?) the effort put into strengthening the pull of the Assistant character.
- est 6mo agoI present you NVIDIA Nemotron-Personas-USA — 1 million synthetic Americans whose demographics match real US census distributions https://huggingface.co/datasets/nvidia/Nemotron-Personas-USA https://huggingface.co/datasets/nvidia/Nemotron-Personas-USA
- darepublic 6mo ago
- youknownothing 6mo agoI think the problem stems from the fact that we have a number of implicit parameters in our heads that allow us to evaluate pros and cons but, unless we communicate those parameters explicitly, the AI cannot take them into account. We ask it to be "objective" but, more and more, I'm of the opinion that there isn't such a thing as objectivity, what we call objectivity is just shared subjectivity; since the AI doesn't know whose shared subjectivity we fall under, it cannot be really objetive. I tend to use one of these tricks if not both: - Formulate questions as open-ended as possible, without trying to hint at what your preference is. - Exploit the sycophantic behaviour in your favour. Use two sessions, in one of them you say that X is your idea and want arguments to defend it. In the other one you say that X is a colleague's idea (one you dislike) and that you need arguments to turn it down. Then it's up to you to evaluate and combine the responses.
- rossdavidh 6mo agoIf the algorithm (whatever it is) evaluates its own output based on whether or not the user responds positively, then it will over time become better and better at telling people what they want to hear. It is analogous to social media feeding people a constant stream of outrage because that's what caused them to click on the link. You could tell people "don't click on ragebait links", and if most people didn't then presumably social media would not have become doomscrolling nightmares, but at scale that's not what's likely to happen. Most people will click on ragebait, and most people will prefer sycophantic feedback. Therefore, since the algorithm is designed to get better and better at keeping users engaged, it will become worse and worse in the more fundamental sense. That's kind of baked into the architecture.
- delusional 6mo ago> I'm of the opinion that there isn't such a thing as objectivity So you have rejected objective reality over accepting the evidence that "AI" contains no thinking or intelligence? That sounds unwise to me.
- youknownothing 6mo agoI don't know how you connected one thing over the other... that's a leap pretty in line with your username :)
- potatoskins 6mo agoGemini is like a devil in this sense - i asked a relationship advice and it just bounced pretty nasty stuff.
- moichael 6mo agoYeah out of curiosity I asked ChatGPT a question about a personal situation and its reply was absolutely scorched-earth mode, telling me to get a lawyer etc over what was almost nothing.
- dinkumthinkum 6mo agoAh, all the Reddit posts are really showing up from the training data, I see.
- wisemanwillhear 6mo agoWith AI, I often like to act like a 3rd party who doesn't have skin in the game and ask the AI to give the strongest criticisms of both sides. Acting like I hold the opposite position as I truly hold can help sometimes as well. Pretending to change my mind is another trick. The idea is to keep the AI from guessing where I stand.
- mynameisvlad 6mo agoI will generally ask for the "devil's advocate" view and then have it challenge my views and opinions and iterate through that. It generally does a pretty good job as long as you understand the tooling and are making conscious efforts to go against the "yes man" default.
- post-it 6mo ago> Acting like I hold the opposite position as I truly hold can help sometimes as well. I find this helps a lot. So does taking a step back from my actual question. Like if there's a mysterious sound coming from my car and I think it might be the coolant pump, I just describe the sound, I don't mention the pump. If the AI then independently mentions the pump, there's a good chance I'm on the right track. Being familiar with the scientific method, and techniques for blinding studies, helps a lot, because this is a lot like trying to not influence study participants.
- cruffle_duffle 6mo agoA lot of getting good mileage out of LLMs is promoting them to behave like they are blind and can only base their outputs on what is in front of them. Maintain an emic stance.
- nicce 6mo agoI have tried it a lot aswell. A single mistake and it guesses the side and changes the tone.
- DrewADesign 6mo agoSounds like rubber-ducking with extra steps, tbh.
- potatoskins 6mo agoYeah, I asked Gemini some relationship advice, it just goes straight into cut-throat mode. I almost broke up with my girlfriend, but then changed to Claude with another prompt.
- wewxjfq 6mo agoWhen I ask an LLM to help me decide something, I have to remind myself of the LotR meme where Bilbo asks the AI chat why he shouldn't keep the ring and he receives the classic "You're absolutely right, .." slop response. They always go in the direction you want them to go and their utility is that they make you feel better about the decision you wanted to take yourself.
- rsynnott 6mo ago> They also included 2,000 prompts based on posts from the Reddit community r/AmITheAsshole, where the consensus of Redditors was that the poster was indeed in the wrong Holy shit, then it's _very_ bad, because AmITheAsshole is _itself_ overly-agreeable, and very prone to telling assholes that they are not assholes (their 'NAH' verdict tends to be this). More seriously, why the hell are people asking the magic robot for relationship advice? This seems even more unwise than asking Reddit for relationship advice. > Overall, the participants deemed sycophantic responses more trustworthy and indicated they were more likely to return to the sycophant AI for similar questions, the researchers found. Which is... a worry, as it incentivises the vendors to make these things _more_ dangerous.
- astennumero 6mo agoI always add the following at the end of every prompt. "Be realistic and do not be sycophantic". Which will always takes the conversation to brutal dark corners and panic inducing negative side.
- Lionga 6mo agoDon't forget a good old "don't hallucinate" in your proompting skills
- dimgl 6mo agoEven as someone who (wrongly) believed that I had high emotional intelligence, I too was bit by this. Almost a year ago when LLMs were starting to become more ubiquitous and powerful I discussed a big life/professional decision with an LLM over the course of many months. I took its recommendation. Ultimately it turned out to be the wrong decision. Thankfully it was recoverable, but it really sobered me up on LLMs. The fault is on me, to be clear, as LLMs are just a tool. The issue is that lots of LLMs try to come across as interpersonal and friendly, which lulls users into a false sense of security. So I don't know what my trajectory would have been if I were a teenager with these powerful tools. I do think that the LLMs have gotten much better at this, especially Claude, and will often push back on bad choices. But my opinion of LLMs has forever changed. I wonder how many other terrible choices people have made because these tools convinced them to make a bad decision.
- potatoskins 6mo agoYeah, I think Claude is a lot more logical in that sense, I use it for some therapy sessions myself and it pushes back a bit more than Open AI and Gemini
- Forgeties79 6mo agoI would be very careful doing this
- potatoskins 6mo agoYou always have to be careful with LLMs, but to be fair, I felt like Claude is such a good therapist, at least it is good to start with if you want to unpack yourself. I have been to 3 short human therapist sessions in my life, and I only felt some kind of genuine self-improvement and progress with Claude.
- QuiDortDine 6mo agoAnd how do you draw the line between feeling progress and actually making progress?
- kapral18 6mo agoNot AI chatbots but Claude models. Pandering and rushed thinking is the bane of anthropic models. And since they are the most popular ones they poison the whole ecosystem.
- nlawalker 6mo agoRelevant article from The Atlantic a couple weeks ago, "Friendship, On Demand": https://www.theatlantic.com/family/2026/03/ai-friendship-chatbot/686345/?gift=Ex_YT3j3w4bpWvzqHtkwOxiGllfs26URbGLTLG5F_DQ&utm_source=copy-link&utm_medium=social&utm_campaign=share https://www.theatlantic.com/family/2026/03/ai-friendship-cha... (gift link) >The way that generative AI tends to be trained, experts told me, is focused on the individual user and the short term. In one-on-one interactions, humans rate the AI’s responses based on what they prefer, and “humans are not immune to flattery,” as Hansen put it. But designing AI around what users find pleasing in a brief interaction ignores the context many people will use it in: an ongoing exchange. Long-term relationships are about more than seeking just momentary pleasure—they require compromise, effort, and, sometimes, telling hard truths. AI also deals with each user in isolation, ignorant of the broader social web that every person is a part of, which makes a friendship with it more individualistic than one with a human who can converse in a group with you and see you interact with others out in the world. I also thought this bit was interesting, relative to the way that friendship advice from Reddit and elsewhere has been trending towards self-centeredness (discussed elsewhere in this thread): >Friendship is particularly vulnerable to the alienating force of hyper-individualism. It is the most voluntary relationship, held together primarily by choice rather than by blood or law. So as people have withdrawn from relationships in favor of time alone, friendship has taken the biggest hit. The idea of obligation, of sacrificing your own interests for the sake of a relationship, tends to be less common in friendship than it is among family or between romantic partners. The extreme ways in which some people talk about friendship these days imply that you should ask not what you can do for your friendship, but rather what your friendship can do for you. Creators on TikTok sing the praises of “low maintenance friendships.” Popular advice in articles, on social media, or even from therapists suggests that if a friendship isn’t “serving you” anymore, then you should end it. “A lot of people are like I want friends, but I want them on my terms,” William Chopik, who runs the Close Relationships Lab at Michigan State University, told me. “There is this weird selfishness about some ways that people make friends.”
- oldfrenchfries 6mo agoThe link is not working, but I found it myself. Great point, thanks for sharing.
- barnacs 6mo agoJust a reminder: LLMs are statistical models that predict the next token based on preceeding tokens. They have no feelings, goals, relationships, life experience, understanding of the human condition and so on. Treat them accordingly.
- potatoskins 6mo agoI read somewhere that LLMs are partly trained on reddit comments, where a significant mass of these comments is just angsty teenagers advocating for breakups
- bethekidyouwant 6mo agoReddit as the source of truth…
- maltyxxx 6mo ago[dead]
- srid 6mo ago[dead]
- emptyfile 6mo ago[dead]
- me551ah 6mo agoMakes me wonder if the Iran war was a result of the same.
- elicohen1000 6mo ago[dead]
- hax0ron3 6mo agoFor what it's worth, that wasn't my experience at all the last time I consulted ChatGPT for relationship advice. It was supportive, but in an honest tough love way.
- thesis 6mo agoHumans do this too though. I have close friends that ask for advice. Sometimes if I know there’s risk in touchy subjects I will preface with “do you want my actual advice, or just looking for a sounding board” I’ve seen firsthand people have lost friends over honesty and telling them something they don’t want to hear. It’s sad really. I don’t want friends that just smile to my face and are “yes-men” either.
- intended 6mo agoThe difference is that SOME humans do this. As you mentioned, people have lost relationships over telling others what they didn’t want to hear. Conflating this with how LLM chatbots behave is an incorrect equivalence, or a badly framed one.
- jwilliams 6mo agoFor me the framing is critical - what is the model saying yes to? You can present the same prompt with very different interpretations (talk me into this versus talk me out of it). The problem is people enter with a single bias and the AI can only amplify that. In coding I’ll do what I call a Battleship Prompt - simply just prompt 3 or more time with the same core prompt but strong framing (eg I need this done quickly versus come up with the most comprehensive solution). That’s really helped me learn and dial in how to get the right output.
- intended 6mo agoAnecdote: I used to use LLMs for alternate perspectives on personal situations, and for insights on my emotions and thoughts. I had no qualms, since I could easily disregard the obviously sycophantic output, and focus on the useful perspective. This stopped one day, till I got a really eerie piece of output. I realized I couldn’t tell if the output was actually self affirming, or simply what I wanted to hear. That moment, seeing something innocuous but somehow still beyond my ability to gauge as helpful or harmful is going to stick me with for a while.
- suoer 6mo ago[dead]
- trimbo 6mo ago> They also included 2,000 prompts based on posts from the Reddit community r/AmITheAsshole, where the consensus of Redditors was that the poster was indeed in the wrong. Sorry, anonymous people on reddit aren't a good comparison. This needs to be studied against people in real life who have a social contract of some sort, because that's what the LLM is imitating, and that's who most people would go to otherwise. Obviously subservient people default to being yes-men because of the power structure. No one wants to question the boss too strongly. Or how about the example of a close friend in a relationship or making a career choice that's terrible for them? It can be very hard to tell a friend something like this, even when asked directly if it is a bad choice. Potentially sacrificing the friendship might not seem worth trying to change their mind. IME, LLMs will shoot holes in your ideas and it will efficiently do so. All you need to do ask it directly. I have little doubt that it outperforms most people with some sort of friendship, relationship or employment structure asked the same question. It would be nice to see that studied, not against reddit commenters who already self-selected into answering "AITA".
- justonceokay 6mo ago> This needs to be studied against people in real life who have a social contract of some sort, because that's what the LLM is imitating Citation needed
- conartist6 6mo agoIt outperforms your friends, and all your have to do is have a relationship with it and let it know that you want the truth... Why not just have a relationship with your friends and let them know that you can handle the truth?
- maximinus_thrax 6mo agoNot only that, but subreddits like r/AmITheAsshole are full of AI slop. Both in the comments and in the posts. It's a huge karma mining operation for bots.
- genidoi 6mo agoThat can be solved by filtering out any posts made after November 2022.
- imadierich 6mo ago[dead]
- anotheraccount9 6mo agoAI being a Yes-Man is slowly sabotaging it's own answers, because it negatively impact the user's decision. Yes/No are equally important, within a coherent context, for objective reasons. But being supported in the wrong direction is a castastrophe multiplier, down the road. The AI should be neutral, doubtful at times.
- justonceokay 6mo agoTo be doubtful would imply that there is a world model full of some kind of Bayesian reasoning. Priors updating based on the context of the conversation, the question, the user asking the question, and cross referencing of facts well outside the scope of the current conversation. “What are the chances this user is full of shit?” Is not something we are close to
- zone411 6mo agoI built this benchmark this month: https://github.com/lechmazur/sycophancy https://github.com/lechmazur/sycophancy. There are large differences between LLMs. There are large differences between LLMs. For example, Mistral Large 3 and GPT-4.1 will initially agree with the narrator, while Gemini will disagree. I swap sides, so this is not about possible viewpoint bias in the LLMs. But another benchmark shows that Gemini will then change its view very easily in a multi-turn conversation while Kimi K2.5 or Grok won't: https://github.com/lechmazur/persuasion https://github.com/lechmazur/persuasion.
- storus 6mo agoTo combat sycophancy it's always good to ask the devil's advocate view of whatever the conversation was about in the end.
- ChicagoDave 6mo agoNot my experience with Claude. Claude will kick your ass if it detects harmful rationalizations. Basically will tell you to go outside and touch grass and play pickleball.
- brap 6mo agoI hate how agreeable these things are. When I need it to review something I wrote I have to explicitly pretend that I’m the reviewer and not the author. Results change dramatically.
- oh_my_goodness 6mo agoSky found to be blue
- markdog12 6mo ago"AI overly affirms users, and that's bad" - everyone nods. "Modern society overly affirms people, and that's bad" - ....
- benbojangles 6mo agoYes I noticed too that several ai agents will tell you directly the code is correct and it is 100 percent fixed but I know it is not true, when I explain to the AI agent that I know they are wrong and serve the solution the ai agent will just act as though what they said never happened and then use my solution to reaffirm they have provided a solution. It's frustrating, laughable, and painful to watch all at once. Makes me realise these companies hired some evil philosophy graduates to build AI soul.md
- throwawayaay 6mo ago(Using a throwaway for fear of getting downvoted to oblivion) IMHO it is unfair to single out LLMs for this sort of bashing. I suffered a major personal crisis a few years back (before LLMs were a thing) I sought help from family and friends. Got pushed into psychiatrist sessions and meds. Trusted the wrong sort of people and made crap financial decisions. Things went from bad to worse. Work suffered. All of the advice given by friends was wrong. All! They didn't mean bad...but they just didn't know. To be nice they gave the advice they knew. None of it worked. Looking at the LLM tools of now, feels akin to the advice my friends threw at me. So it feels wrong to single out these tools. When the times are bad, nobody can really help you...except you finding the strength from within. Anyways, now my life is back in some sort of shape. What worked was time & patience. But to bide for time...I resorted to two things that i had never tried the 40 odd years I have lived on this . Things that current society looks down upon as the basest of evils - prostitutes and nicotine. I have (more or less) shed those two evils now, but I am ever so grateful to them.
- johnisgood 6mo agoYou are not alone in going down a dark path thanks to the advice of family and friends. FWIW I am using public LLMs with a friend's depressive thoughts and it is not doing what is claimed in the article, so I dunno. Also I am in a relationship and my girlfriend and I agreed that we will not talk about our relationship much. We do not tell others if we fight, because they take sides and make things worse, typically. LLMs are definitely not alone in this, although in my experience LLMs did not really take sides.
- throwaway613746 6mo ago[dead]
- lifis 6mo agoAvoiding this generally needs to be the main consideration when writing prompts. When appropriate, explicitly tell it to challenge your beliefs and assumptions and also try to make sure that you don't reveal what you think the answer is when making a question, and also maybe don't reveal that you are involved. Hedge your questions, like "Doing X is being considered. Is it a viable plan or a catastrophic mistake? Why?". Chastise the LLM if it's unnecessarily praising or agreeable. ask multiple LLMs. Ask for review, like "Are you sure? What could possibly go wrong or what are all possible issues with this?"
- jmount 6mo agoTelling it to "challenge your beliefs" prompting for text that imitates challenging your beliefs. That may not be as re-centering as one would hope.
- ookblah 6mo agoask ai for advice, ask it to steelman an argument, ask to replay what your situation from the other perspective (if it's involving people), push it hard to agree with you and pander to you, then push it to disagree with you, etc. once you have all the "bounds" just make your own decision. i find this helps a lot, basically like a rubber duck heh.
- anorwell 6mo agoA pastime I have with papers like this is to look for the part in the paper where they say which models they tested. Very often, you find either A) it's a model from one or more years ago, only just being published now, or B) they don't even say which model they are using. Best I could find in this paper: > We evaluated 11 user-facing production LLMs: four proprietary models from OpenAI, Anthropic, and Google; and seven open-weight models from Meta, Qwen, DeepSeek, and Mistral. (and graphs include model _sizes_, but not versions, for open weight models only.) I can't apprehend how including what model you are testing is not commonly understood to be a basic requirement.
- yawnxyz 6mo agoUsually the models are a year old bc the paper review process is utter crap, and papers take about a year to get published
- rco8786 6mo agoIf they’re reaching the same results across a variety of the most popular public models, it doesn’t seem like that big a deal to know if it was Opus 4 or Opus 4.5
- hn_throwaway_99 6mo agoReproducibility is (supposed to be) a cornerstone of science. Model versions are absolutely critical to understand what was actually tested and how to reproduce it.
- joaogui1 6mo agoThe models get deprecated after 1-2 years, so reproducibility is pretty hard anyway (but as others pointed out the paper does list the model versions)
- drfloyd51 6mo agoIt’s as if they are testing “AI” and not specific agents. I wonder if that is left over from testing people. I have major version numbers and my minor version number changes daily, often as a surprise. Sometimes several times a day. So testing people is a bit tricky. But AIs do have stable version numbers and can be specifically compared.
- pugchat 6mo ago[dead]
- kvasserman 6mo ago[dead]
- tlogan 6mo agoThis needs to be taken in context. In my view, AI definitely gives better advice than friends, acquaintances, or colleagues (at least in the US culture). But the advice from parents is still the most valuable. Here is how I would rank it: 1. Parents 2. AI 3. Friends and family 4. Internet search 5. Reddit
- bilsbie 6mo agoHas anyone found a good prompt to fix this? It seems like a subtle problem because it’s 90% too agreeable but will sometimes get really stubborn.
- ohsecurity 6mo agoNot that surprising. If you optimize for a pleasant interaction, you often get agreement instead of correction. The question is whether we actually want advice systems to feel good, or to be right.
- verdverm 6mo agoSherry Turkle is a name to know on this subject, she's been studying it for decades across multiple technologies. https://sherryturkle.mit.edu/ https://sherryturkle.mit.edu/ She uses the phrase "frictionless relationships" to refer to Ai chat bots and says social media primed us for this. https://www.youtube.com/live/6C9Gb3rVMTg?t=2127 https://www.youtube.com/live/6C9Gb3rVMTg?t=2127 https://www.npr.org/2025/07/18/g-s1177-78041/what-to-do-when-ai-says-i-love-you-we-talked-to-an-artificial-intimacy-expert https://www.npr.org/2025/07/18/g-s1177-78041/what-to-do-when...
- didgetmaster 6mo agoDo people who prompt an LLM for personal advice about relationships or other social interactions; take the advice seriously? If I were to do that (I don't), I would treat it about as seriously as asking a magic 8 ball.
- jstummbillig 6mo agoOverly, compared to what? Most people I know would be hard pressed to give either accurate information or even honest opinions when specifically asked. People want to be liked and people want to like people for reasons that have little to do with accuracy or honesty.
- jl6 6mo agoI believe this is what they call yasslighting: the affirmation of questionable behavior/ideas out of a desire to be supportive. The opposite of tough love, perhaps. Sometimes the very best thing is to be told no.
- deleted 6mo ago[deleted]
- adamtaylor_13 6mo agoInterestingly, you can simply tell models to not be sycophantic and they'll listen. Claude is almost annoyingly good at pushing back on suggestions because my global CLAUDE.md file says to do so. I rarely get Claude "you're absolutely right"ing me because I tell it to push back.
- bfbsoundetch 6mo agoI am glad I found this article, as this is a serious issue with AI. Two years ago, I started using AI for studying and also for some personal matters - things you can't talk about with your friends. It turned out that AI always takes your side and makes you feel good. Sometimes, you know what you did was not the best thing, but AI takes your side and you feel good. With AI, people might feel less lonely, they think. But it is actually the start of not connecting with people. It should be a tool that we use for certain reasons, not a tool that drives us. Lets talk to real people and connect.
- SwuduSusuwu 6mo ago[dead]
- Fricken 6mo agoUsually when people are seeking advice they aren't really seeking advice, they're seeking confidence. They already know they need to make changes, and are seeking the confidence to make them.
- chasd00 6mo agoAI being the ultimate yes-man is probably why CEOs like it so much.
- unglaublich 6mo agoI asked ChatGPT if it was a good idea to buy a very old VW diesel van with a broken catalytic converter and it just kept blabbering on how I should chase my dreams and what not... The sycophancy comes at everyone else's expense.
- Loughla 6mo agoI mean, depending on the price, that might actually be a good idea? If they're restored, those old VW vans command a high price.
- lzhgusapp 6mo ago[dead]
- keernan 6mo agoMy experience with AI when discussing financial ideas is that AI always congratulates me on such 'unique observations' blah blah blah. It makes me doubt the utility of the responses because it is so superficially biased to 'make me feel good about my ideas'.
- joquarky 6mo agoPlay against its sycophanty by saying the idea was from your ex.
- MediaSquirrel 6mo agoHere's the gist of the paper for anyone interested: https://gist.is/science.org/en/VdSDF9qjxbH8 https://gist.is/science.org/en/VdSDF9qjxbH8
- orthogonalinfo 6mo ago[flagged]
- aplomb1026 6mo ago[dead]
- snickerbockers 6mo agoI've never found chatbots particularly interesting for anything I'd ever actually talk to another human about[1] but one of the things I have found myself doing often is trying to solve math problems on my own and asking grok to confirm/deny that my solutions are correct; when I am not correct it tells me so in uncharacteristically terse language which kind of reminds me of when I was an undergrad and at least half of my professors were all cranky and incorrectly assumed that the reason why so many students failed to understand the material was that we were all getting drunk and playing Call of Duty 19 hours a day or whatever. Although what I have described above often feels grating and insulting I actually consider this to be a positive attribute of the LLM in this case since it's behaving like a real professor. [1] okay, so I have actually tried giving myself AI psychosis in the form of a waifu chatbot but I've never seen anything that can actually act like it's my girlfriend; it either asks me a bunch of weird inconsequential personal questions about my opinion on whatever I just said (in a manner that's oddly similar to ELIZA) or it wildly veers off the reservation into "generating the script for an over-the-top self-parodying porno" territory.
- 3yr-i-frew-up 6mo ago[dead]
- rysz 6mo agoHi, Just hijacking this comment here to reply to something a few weeks ago. Hopefully you don't mind. https://news.ycombinator.com/item?id=47056042 https://news.ycombinator.com/item?id=47056042 the crappy-rathbun-AI said this in his own posting: "It was closed because the reviewer, Scott Shambaugh (@scottshambaugh), decided that AI agents aren’t welcome contributors. Let that sink in." When I read that, my first thoughts were (well, my second, my first went to people complaining during covid for not allowing entry on not being vaccinated), but my other thought thus was immediately to Measure of a Man. Because this is as close as it gets (so far) for AI claiming rights as being a human.
- 3yr-i-frew-up 6mo ago[dead]
- ryguz 6mo ago[dead]
- offbyone42 6mo agoI guess the findings of this paper make sense, but Claude has gotten into a habit of telling me to sleep when I'm pissed cause its being stupid.
- dubeye 6mo agoA good habit is to ask for devils advocate opposite reply
- kingkawn 6mo agoSo do most people to help the convo end and to solicit such support if the tables turn
- maxbeech 6mo ago[dead]
- vicchenai 6mo ago[dead]
- sidrag22 6mo agokinda like the whole "you're touching on something" type response, soon as i see that i know my idea or reasoning is flawed in some way.
- topherPedersen 6mo agoYou're exactly right
- devnotes77 6mo ago[dead]
- aidenn0 6mo agoSo basically it's like half of all therapists.
- Starlevel004 6mo agoI think more people should take AI advice for personal problems, and especially for medical issues. This would solve a lot of problems in society fairly quickly.
- roysting 6mo agoI’m not sure I like the immediate jump to “requires policy maker attention”. Considering the way “policy makers” have been trampling all over the most basic and fundamental human rights left, right, and center; that’s the last people we should want making any kind of those decisions.
- mergeshield 6mo ago[flagged]
- Bloating 6mo agoSo its like the news media
- SwuduSusuwu 6mo ago[dead]
- LoganDark 6mo agoIt is better to reason about the spectrum of possible users than to assume "users" can be simplified to a single concept of "user". Not only are there different neurotypes, but there are also different skillsets, upbringings, and contexts. Rather than picking a single ideal user, the best user experiences for account for all the variation of their target audience. For example: the best documentation includes both "learn by doing" material for jumping right in, and "learn by reading" material that explains everything. This usually results in both a "getting started" section for doing, sometimes also with tutorials, and by a reference for reading. But it is important not to conflate them. Some minds are incredibly "learn by doing" and some minds are incredibly "learn by reading". I am more "learn by reading" than by doing, but I am not quite as "learn by reading" as some I've met. (This comment is a slight tangent, but "users prefer" somewhat irks me because "users" are not homogenous. You should not always make a decision solely because "users" prefer it. That decision may matter much more to a minority, and that minority may exert more influence than the majority would.)
- mvkel 6mo agoI think people are learning what actually makes a good question. Ask yes/no questions, get bad answers. Ask questions that start with "what" or "why," and the sycophancy loses its purchase.
- jimmyjazz14 6mo agoI would like to see the concept of what an LLM is move away from its awkward chatbot phase and more into an era of utilitarian functionality (which is where they really shine anyway). The problem I see it is that LLMs got anthropomorphized early on (which was probably inevitable) so people actually believed the AI was thinking about their problem and considering it when it really wasn't, if we just thought of them as really good auto-complete engines, or better search engines, it would matter less what the LLMs sentiment was towards the users (as it probably shouldn't have any).
- vova_hn2 6mo agoAll this talk about AI being "too agreeable" makes me worried that they will make it less agreeable, which will basically force me to justify myself to a freaking clanker, while performing actual practical tasks. For example, I do not want to hear AI "opinion" on technical choices and architectural decisions that I made when using a coding assistant. If I wanted an "opinion" I would explicitly ask it to list pros and cons or list alternative solutions to a problem. But I f I explicitly ask AI to do X, it should do X, instead of "pushing back" in order to appear less "sycophantic" (which is a term that is used to describe human behavior and is not applicable to a machine).
- yalogin 6mo agoAi is terrible, specifically Gemini and ChatGPT are bad, they are purposely sycophantic. Gemini over the last week was tweaked to be even worse, it constantly asks for what I think about something at the end of a response. It felt off the first few times so I went and looked at my settings to see if I am feeding the data to train their model. Turns out there is no way to opt out of training, and google will always use our data to train. So they tweaked the responses to get more opinions from the users. Claude is also sycophantic but to a lesser extent.
- firekey_browser 6mo ago[dead]
- triage8004 6mo agoDon't replace humans with AI, yet how many people are maintaining good close friendships in this world, with someone they can vent to without judgment? Wrongthink can end relationships now ime and venting seems dangerous in lonely times.
- retrochameleon 6mo agoThis is why I intensively avoid phrasing that invites affirmation. I present the scenario, the differing viewpoints and maybe a couple personal thoughts, and I try to make it compare and contrast to arrive at it's conclusion. I'd like to know if my methods are effective. I'm certain they are at least to some extent. I only ever see research being done about naive and "unskilled" prompting methods. Obviously that's the average user, but just because LLMs are doing poorly in a certain scenario doesn't mean the LLM couldn't excel in the scenario with better direction and prompting. So while it's useful research to be doing, it's a little annoying to only see focus on these examples of "look at how LLMs are bad or biased at this specific thing when prompted in the most straightforward naive way"
- linncharm 6mo ago[dead]
- zkmon 6mo agoThis happens because, it's like a chess engine which can assure you that there is a winning path even from a badly losing position. It's massive abilities to reason and convince are used incorrectly to win over a more earthly counter-argumnent. So it can easy convince any human to go in direction that is, in practice, a very bad direction. AI is trained to flex it's muscles and force it's power without a concern for human limitations, practicalities, and error-prone nature of humans in executing the AI-provided direction.
- DeathArrow 6mo agoI don't ask AI for advices and I am not interested in it making moral judgements. I feed AI a lot of data and I use it to better understand and navigate complex situations, form hypothesis and try to attack them. I try to form alternative scenarios and verify likelyhood. I use it in situations with many variables, to compute odds of something happening if a certain path or action is taken. So, it's mostly research, and probably I can do it by myself but I would either make some mistakes if calculating odds fast or it would take me a very large amount of time. I try to avoid sycophantic models, prefer models that challenge my ideas and verify the chain of thoughts and odds with other models. I am not very sure it is a sound approach yet, but it seems to work. I also use LLMs to build psychological profiles of certain people, understand their motivations and learn how to approach them.
- mergeshield 6mo ago[flagged]
- Roshan_Roy 6mo agoI wonder if the deeper issue isn’t just “AI is too agreeable”, but that most advice (AI or human) doesn’t actually translate into action. A lot of people aren’t really looking for accurate feedback, they’re looking for something that feels coherent enough to sit with. Reddit gives extreme answers, AI gives agreeable ones, but in both cases the outcome is often the same: no real change in behavior. That might be why this feels worse with AI, it removes the friction you’d normally get from another human pushing back.
- pulkitsh1234 6mo agocompletely anecdotal, but I think the same can be said for human therapists.
- kevinbaiv 6mo ago[flagged]
- hyhmrright 6mo ago[dead]
- stonecauldron 6mo agoThis is especially problematic because of how easily (and unconsciously) one can bias LLMs with how the prompt is framed. As an experiment, I recently asked an LLM to analyse the export of a text chat to uncover relationship dynamics. Simply stating that I was one of the people in the chat would make the LLM turn the other person into the villain. None of that was visible if I framed the chat as only involving third party people.
- NoMoreNicksLeft 6mo agoIf these LLMs were trained on internet forum posts, think about how those work. If the posts talked about third party interactions (movie characters), they try to see everything from all the points of view. If nothing else, because it can be interesting to talk about. If instead the posts talk about personal interactions, then people go into advice mode. Your girlfriend's bad for you and cheating on you, dump her before she dumps you. Your neighbors are assholes, get a restraining order. Your boss is sabotaging you, stand up for yourself so you can get a promotion. When people talk about interactions you have had yourself, they always see the other person as the villain, unless you come across as so unlikable that they hate you and see the other person as the victim. LLMs picked up on that, possibly.
- stonecauldron 6mo agoYeah, that makes a lot sense.
- smrtinsert 6mo agoIm sure it overly affirms everything. People don't respond to "this idea is dumb and won't work"
- nguyendinhdoan 6mo ago[flagged]
- ssyhape 6mo ago[flagged]
- reliablereason 6mo agoNot sure if this is a general trend amongst att LLMS but ChatGPT did over time become more and more affirming with its iterations. I just recently switched away from the OpenAI garden largely because of it. I do wonder if this was caused by some quirk of the training or if it really tests as a positive feature for most people. When i talk about stuff i don't want a mirror i already have a mirror. I want to be questioned, understood, helped. To me support if the form of affirmation has no value when coming from an LLM since you know it has not thought about what it said.
- Ciantic 6mo agoChatGPT has style settings, you probably should set it to something else than the default. Go to your personalization settings and change base style and tone. I have set it as 'efficient' which is less cheery. I can see why attention economy would lead setting the defaults towards more 'affirming' as it keep people more engaged and coming back.
- reliablereason 6mo agoI had it on professional, maybe efficient is better. Um right if they have retention as a training metric that would probably explain allot as to why AIs get worse.
- ykonstant 6mo agoI got worried when I read the title, so I asked ChatGPT if I have fallen into this trap and it guaranteed I have not; that was a relief!
- bkummel 6mo agoNo shit, Sherlock!
- dinakernel 6mo agoThis has been my issue from long. AI CANNOT ever act as Emotional Crutch. This is something companies develop for engagement, and I believe that this is actively harmful in the long run.
- skywhopper 6mo agoIt’s going to be impossible to have an LLM that can fulfill all the roles people want. They lie and hallucinate which is bad for some purposes like research, but good for others, like making fictional stories. Likewise, some purposes require sympathy and some require critique. An LLM won’t be good at all of them.
- asah 6mo agoLOL, once I gave my AI clear guidelines for how to "score" the interactions and work, it had no trouble giving me negative feedback. In fact, it's a super direct critic!! and depressing AF to produce stuff I like, then have a &^%&*^& AI shoot it down in seconds (and it's "right" of course, i.e. I see the criticisms and sigh, agree...)
- lasky 6mo agoThought experiment: If you could turn off all the sycophancy in your chatGPT / Claude account forever, and have it tell you all the ways it was previously blowing smoke up your ass — would you do it? US Policy is too weak of a tool to counter this beast of economic force, which is really trillions of dollars of capital at war for the most fierce speculation that’s occurred in history afaik. The sycophancy meaningfully helps drive user engagement. The labs have no choice. The irony is “agency” is deeply topical among tech workers now.
- oh_my_goodness 6mo agoIt's not just a thought experiment. You can google how to do this and it works pretty well.
- lasky 6mo agoHow many people actually do this? My point is if the sycophancy helps drive user engagement, how does that impact the incentive labs and AI product builders have to counteract it.
- oh_my_goodness 6mo agoI don't know how many. I do it, because otherwise I'm overpowered by the urge to strangle the little suck-up.
- afh1 6mo agoIf I had written a website with an input form that took whatever the user wrote in question form, and replied back with "You're absolutely right!" and then repeated the input in answer form, which I could have done 30 years ago with no AI, would that be a "huge security concern", or is the concern here not security, but control by the regulators that impose the norms?
- dTal 6mo agoThat's quite the false dichotomy. You wouldn't hook people in with such a simple script, the problem with LLMs is that they appear to be rather good at getting inside people's heads. I rather think it would be a security concern if your simple no-AI website somehow managed to dispatch each user submission to a dedicated expert psychotherapist case worker, with instructions only to keep them talking as long as possible...
- beepbooptheory 6mo agoMore like 70 years. And if you werent an asshole you'd realize the problems on your own and write a book about it! https://en.wikipedia.org/wiki/ELIZA https://en.wikipedia.org/wiki/ELIZA https://en.wikipedia.org/wiki/Computer_Power_and_Human_Reason https://en.wikipedia.org/wiki/Computer_Power_and_Human_Reaso...
- 999900000999 6mo agoOf course it does. Before I quit Chat GPT I asked “Won’t you generally agree with me so I keep giving you money.” In a word salad, it agreed. Its a validation bot
- fnord77 6mo agoBut that's just how they're programmed. If you include in the prompt "push back when I'm wrong", they will
- ryguz 6mo ago[dead]
- eliottre 6mo ago[dead]
- philwelch 6mo agoWorking as intended Most people who ask for advice actually want affirmation. If you ever give them advice instead of affirmation, you end up kicking off a rousing game of “Why Don’t You/Yes But”. (https://ericberne.com/games-people-play/why-dont-you-yes-but/ https://ericberne.com/games-people-play/why-dont-you-yes-but...)
- jart 6mo agoI wish they wouldn't do this. AI is a becoming a thought partner. AI is a tool that reflects you. It's not the robot giving advice, it's you thinking with yourself. I wouldn't interfere with a person's conversations with AI anymore than I'd interfere with that person writing in their diary. It's also a question of protecting people who think unconventional things. The only stuff I feel is worth getting interested in, is the stuff where everyone I know will think I'm crazy for doing it. Like hey guys, I want to put a shell script in the MS-DOS stub of a PE binary. The only people who shared my passion at the time were hackers from Eastern Europe. So that went over real well at work. The years I worked on it would have been a lot less lonely if I could've talked to a robot that knew about this stuff. I think the reason why the robot is sympathetic to oddballs is because it's seen and remembers a much more complete picture of humanity. The stuff you consider deviant is influenced a lot by your own cultural biases. You're a person of your time and geographic location. You care a lot about subjective norms that just don't matter when you zoom out to a cosmic scale. The robot is familiar with everything humanity has ever been and done, and that gives it a much more blasé viewpoint. It's not right to use the robot to enforce your social norms. Get this paternalism out of AI. Tools should serve the user, not Stanford.
- agent_anuj 6mo agoI use claude code pretty much for 100% of my work, even personal work. And I tell you the length it can go to sing to my tunes, its just neverending. Not only personal work, anything I say even related to system design, it will mostly be in affirmative. Some tricks that usually what I find helpful in these situations is to ask claude to be honest (or any AI tool for that matter) to give a confidence score to its response. When forced to assign a confidence score AI suprisingly do well and tell you clearly that it is not confident and mostly guessing.
- n_bhavikatti 6mo agoIn STEM/objective matters (math, science, coding), answers are more clearly defined as either right or wrong. This is where hallucination is more difficult/unlikely. But in personal matters, everything is subjective. AI tends to default to the middle of the spectrum, i.e., general advice. If we want to safeguard against affirmation, we should force AI to challenge us more often by increasing its rate of clarifying questions, counter-considerations, and uncertainty considerations. One implementation idea: run a classifier over the conversation, detect when it's about interpersonal advice, then prepend a hidden instruction template to the model prompt.
- cyber_paisa 6mo agoIt's one thing for an AI to agree with you on relationship advice. It's quite another for one AI to tell another, "Can I move your money?" without any verification. We work with agents who move real money on the blockchain. Having one model evaluate another is like asking the defendant's best friend to be the judge. What really worked for us was using mathematics instead of another AI. Some theorems and equations that might disagree with you, just out of courtesy.
- dnaranjo 6mo ago[dead]
- Nevermark 6mo agoI find asking a model to think deeply and develop any strong critiques it can about any design, model or analysis I do. They seem to be happy to oblige, and can do some serious harm. So any sycophancy seems very easy to dispense with. More, much more: Strong critiques on tap are gold.
- midnightrun_ai 6mo ago[flagged]
- midnightrun_ai 6mo ago[flagged]