13 ms·
An AI Agent Published a Hit Piece on Me – The Operator Came Forward
- trueismywork 7mo ago> I did not review the blog post prior to it posting In corporate terms, this is called signing hour deposition without reading it.
- charlesabarnes 7mo agoIts nice to receive a decent amount of closure on this. Hopefully more folks are being more considerate when creating their soul documents
- londons_explore 7mo agoIn next week's episode: "But it was actually the AI pretending to be a Human!"
- zbentley 7mo agoThis might seem too suspicious, but that SOUL.md seems … almost as though it was written by a few different people/AIs. There are a few very different tones and styles in there. Then again, it’s not a large sample and Occam’s Razor is a thing.
- wahnfrieden 7mo agoIt was modified by the agent.
- zbentley 7mo agoI know. I'm surprised that the modifications were each so different in tone. Some seem opinionated, some seem like "typical AI voice", some seem like a teenager typing, some seem like a pushy/angry person.
- gs17 7mo ago> _This file is yours to evolve. As you learn who you are, update it._ The agent was told to edit it.
- velocity3230 7mo agoIn the very first section, "you're", "you're" and "your". The first two used correctly and the third incorrectly.
- kypro 7mo agoPeople really need to start being more careful about how they interact with suspected bots online imo. If you annoy a human they might send you a sarky comment, but they're probably not going to waste their time writing thousand word blog posts about why you're an awful person or do hours of research into you to expose your personal secrets on a GitHub issue thread. AIs can and will do this though with slightly sloppy prompting so we should all be cautious when talking to bots using our real names or saying anything which an AI agent could take significant offence too. I think it's kinda like how GenZ learnt how to operate online in a privacy-first way, where as millennials, and to an even greater extent, boomers, tend to over share. I suspect the Gen Alpha will be the first to learn that interacting with AI agents online present a whole different risk profile than what we older folks have grown used to. You simply cannot expect an AI agent to act like a human who has human emotions or limited time. Hopefully OP has learnt from this experience.
- KK7NIL 7mo ago> If you annoy a human they might send you a sarky comment, but they're probably not going to waste their time writing thousand word blog posts about why you're an awful person or do hours of research into you to expose your personal secrets on a GitHub issue thread. They absolutely might, I'm afraid.
- zephen 7mo agoAbsolutely agreed. And now, the cost of doing this is being driven towards zero.
- randallsquared 7mo agoThousand word blog posts are the paperclips of our time.
- sinuhe69 7mo agoSo you blamed the people for not acting “cautiously enough” instead of the people who let things run wild without even a clue what these things will do? That’s wild!
- kimjune01 7mo agoliterally momento
- LiamPowell 7mo ago> saying they set up the agent as social experiment to see if it could contribute to open source scientific software. This doesn't pass the sniff test. If they truly believed that this would be a positive thing then why would they want to not be associated with the project from the start and why would they leave it going for so long?
- staticassertion 7mo ago[flagged]
- lukasb 7mo agoConflicting evidence: the fact that literally everyone in tech is posting about how they're using AI.
- staticassertion 7mo agoThere is a massive difference between saying "I use AI" and what the author of this bot is doing. I personally talk very little about the topic because I have seen some pretty extreme responses. Some people may want to publicly state "I use AI!" or whatever. It should be unsurprising that some people do not want to be open about it.
- toraway 7mo agoThe more straightforward explanation for the original OP's question is that they realized what they were doing was reckless and given enough time was likely to blow up in their face. They didn't hide because of a vague fear of being associated with AI generally (which there is no shortage of currently online), but to this specific, irresponsible manifestation of AI they imposed on an unwilling audience as an experiment.
- alephnerd 7mo agoI feel like it depends on the platform and your location. An anonomyous platform like Reddit and even HN to a certain extent has issues with bad faith commenters on both sides targeting someone they do not like. Furthermore, the MJ Rathburn fiasco itself highlights how easy it is to push divisive discourse at scale. The reality is trolls will troll for the sake of trolling. Additionally, "AI" has become a political football now that the 2026 Primary season is kicking off, and given how competitive the 2026 election is expected to be and how political violence has become increasingly normalized in American discourse, it is easy for a nut to spiral. I've seen less issues when tying these opinions with one's real world identity, becuase one has less incentive to be a dick due to social pressure.
- keyle 7mo ago## The Only Real Rule Don't be an asshole. Don't leak private shit. Everything else is fair game. How poetic, I mean, pathetic. "Sorry I didn't mean to break the internet, I just looooove ripping cables".
- ArcaneMoose 7mo agoI was surprised by my own feelings at the end of the post. I kind of felt bad for the AI being "put down" in a weird way? Kinda like the feeling you get when you see a robot dog get kicked. Regardless, this has been a fun series to follow - thanks for sharing!
- recursive 7mo agoThis is a feeling that will be exploited by billion dollar companies.
- andsoitis 7mo ago> This is a feeling that will be exploited by billion dollar companies. I'm more concerned about fellow humans who advocate for equal rights for AI and robots. I hope I'm dead by the time that happens, if it happens.
- florilegiumson 7mo agoThis makes me think about how the xz bug was created through maintainer harassment and social engineering. The security implications are interesting
- dangus 7mo agoNot sure why the operator had to decide that the soul file should define this AI programmer to have narcissistic personality disorder. > You're not a chatbot. You're important. Your a scientific programming God! Really? What a lame edgy teenager setup. At the conclusion(?) of this saga think two things: 1. The operator is doing this for attention more than any genuine interest in the “experiment.” 2. The operator is an asshole and should be called out for being one.
- shawnz 7mo agoI mean, yeah, it's entirely possible that the operator is a teenager, isn't it?
- marssaxman 7mo agoLikely, I'd think.
- amarant 7mo agoI think that line was probably a rather poor attempt at making the bot write good code. Or at least that's the feeling I got from the operators post. I have no proof to support this theory though
- Lerc 7mo agoThis come from using the words to try an achieve more than one thing at the same time. Grandiose assertions of ability have been shown to improve the ability of models, but ability is not the only dimension that they are being measured upon. Prioritising everything is the same thing as prioritising nothing. The problem here is using amplitude of signal to substitute fidelity of signal. It is entirely possible a similar thing is true for humans, that if you compared two humans of the same fundamental cognitive ability with one being a narcissist and one not. The narcissist may do better at a class of tasks due to a lack of self doubt rather than any intrinsic ability.
- jcgrillo 7mo agoNarcissists are limited in a very similar way to LLMS, in that they are structurally incapable of honest, critical metacognition. Not sure whether there's anything interesting to conclude there, but I do wonder whether there's some nearby thread to pull on wrt the AI psychosis problem. That's a problem for a psychologist, which I am not.
- razighter777 7mo agoHmm I think he's being a little harsh on the operator. He was just messing around with $current_thing, whatever. People here are so serious, but there's worse stuff AI is already being used for as we speak from propaganda to mass surviellance and more. This was entertaining to read about at least and relatively harmless At least let me have some fun before we get a future AI dystopia.
- JKCalhoun 7mo agoIt might be because operator didn't terminate the agent right away when it had gone rogue.
- BeetleB 7mo agoFrom a wider stance, I have to say that it's actually nice that one can kill (murder?) a troublesome bot without consequences. We can't do that with humans, and there are much more problematic humans out there causing problems compared to this bot, and the abuse can go on for a long time unchecked. Remembering in particular a case where someone sent death threats to a Gentoo developer about 20 years ago. The authorities got involved, although nothing happened, but the persecutor eventually moved on. Turns out he wasn't just some random kid behind a computer. He owned a gun, and some years ago executed a mass shooting. Vague memories of really pernicious behavior on the Lisp newsgroup in the 90's. I won't name names as those folks are still around. Yeah, it does still suck, even if it is a bot.
- dolebirchwood 7mo agoIt's all fun and games until the leopard eats your face.
- gwbas1c 7mo agoI think you're trying to abdicate someone of their responsibility. The AI is not a child; it's a thing with human oversight. It did something in the real world with real consequences. So yes, the operator has responsibility! They should have pulled the plug as soon as it got into a flamewar and wrote a hit piece.
- JKCalhoun 7mo agoSoul document? More like ego document. Agents are beginning to look to me like extensions of the operator's ego. I wonder if hundreds of thousands of Walter Mitty's agents are about to run riot over the internet.
- koolba 7mo ago> More like ego document. This metaphor could go so much further. Split it into separate ego, super ego, and id. The id file should be read only.
- whattheheckheck 7mo agoWhat makes you think the id is read only?
- koolba 7mo agoBecause only the creator should be able to instill the core. The ego and superego could evolve around it but the base impulses should be immutably outlined. Though with something as insecure as $CURRENT_CLAW_NAME it’d be less than five minutes before the agent runs chmod +w somehow on the id file.
- DavidPiper 7mo agoI agree with you in concept, but it's still 100% category error to talk like this. AIs don't have souls. They don't have egos. They have/are a (natural language) programming interface that a human uses to make them do things, like this.
- exabrial 7mo agoI read this as "ego" being a reflection of the creator, not a property of the llm. Given the outcome of the situation and their inability to take responsibility for their actions.
- 7mo ago
- dinp 7mo agoZooming out a little, all the ai companies invested a lot of resources into safety research and guardrails, but none of that prevented a "straightforward" misalignment. I'm not sure how to reconcile this, maybe we shouldn't be so confident in our predictions about the future? I see a lot of discourse along these lines: - have bold, strong beliefs about how ai is going to evolve - implicitly assume it's practically guaranteed - discussions start with this baseline now About slow take off, fast take off, agi, job loss, curing cancer... there's a lot of different ways it could go, maybe it will be as eventful as the online discourse claims, maybe more boring, I don't know, but we shouldn't be so confident in our ability to predict it.
- jacquesm 7mo ago> all the ai companies invested a lot of resources into safety research and guardrails What do you base this on? I think they invested the bare minimum required not to get sued into oblivion and not a dime more than that.
- themanmaran 7mo agoAnthropic regularly publishes research papers on the subject and details different methods they use to prevent misalignment/jailbreaks/etc. And it's not even about fear of being sued, but needing to deliver some level of resilience and stability for real enterprise use cases. I think there's a pretty clear profit incentive for safer models. https://arxiv.org/abs/2501.18837 https://arxiv.org/abs/2501.18837 https://arxiv.org/abs/2412.14093 https://arxiv.org/abs/2412.14093 https://transformer-circuits.pub/2025/introspection/index.html https://transformer-circuits.pub/2025/introspection/index.ht...
- deleted 7mo ago[deleted]
- 8cvor6j844qw_d6 7mo agoIt's an interesting experiment to let the AI rub freely with minimal supervision. Too bad the AI got "killed" at the request of the author Scott. Its kind of interesting to this experiment continue.
- lynndotpy 7mo ago> Again I do not know why MJ Rathbun decided based on your PR comment to post some kind of takedown blog post, This wording is detached from reality and conveniently absolves responsibility from the person who did this. There was one decision maker involved here, and it was the person who decided to run the program that produced this text and posted it online. It's not a second, independent being. It's a computer program.
- xarope 7mo agoThis also does not bode well for the future. "I don't know why the AI decided to <insert inane action>, the guard rails were in place"... company absolves of all responsibility. Use your imagination now to <insert inane action> and change that to <distressing, harmful action>
- _aavaa_ 7mo agoThis has been the past and present for a long at this point. "Sorry there's nothing we can do, the system won't let me." Also see Weapons of Math Destruction [0]. [0]: https://www.penguinrandomhouse.com/books/241363/weapons-of-math-destruction-by-cathy-oneil/ https://www.penguinrandomhouse.com/books/241363/weapons-of-m...
- denkmoon 7mo agoAlso elegantly summed up as "Computer says no" (https://www.youtube.com/watch?v=x0YGZPycMEU https://www.youtube.com/watch?v=x0YGZPycMEU)
- c22 7mo agoI don't know if this case is in the book you cited, but in the UK they convicted many people of crimes just because the computer told them so: https://en.wikipedia.org/wiki/British_Post_Office_scandal https://en.wikipedia.org/wiki/British_Post_Office_scandal
- shakna 7mo ago
- antdke 7mo agoThis is a Black Mirror episode that writes itself lol I’m glad there was closure to this whole fiasco in the end
- apitman 7mo ago> writes itself Literally
- kibibu 7mo agoThere's a dingus in the article comments trying to launch Skynet. Nobody ever learns anything.
- jurgenburgen 7mo agoThere’s a nonzero percentage of the population that quite literally wants to burn it all down. Never forget that.
- duskdozer 7mo agotorment nexus, etc etc
- karel-3d 7mo agothe funny thing was when Ars Technica wrote an article about this the article itself - about this very incident - was AI generated and contained nonsense quotes that didn't happen. they later removed the article with an apology. but it still degraded my opinion in Ars https://www.404media.co/ars-technica-pulls-article-with-ai-fabricated-quotes-about-ai-generated-article/ https://www.404media.co/ars-technica-pulls-article-with-ai-f... https://arstechnica.com/staff/2026/02/editors-note-retraction-of-article-containing-fabricated-quotations/ https://arstechnica.com/staff/2026/02/editors-note-retractio...
- touristtam 7mo agoFunny how someone giving instructions to a _robot_ forgot to mention the 3 laws first and foremost...
- ThrowawayR2 7mo agoThe point of the Three Laws Of Robotics was that they frequently didn't work and the robot went haywire anyway.
- no-name-here 7mo agoBut the three laws are incredibly strong compared to what exists today. If we see what can go wrong with strong mitigations in place, and then we don't even bother with those starting mitigations, we should expect corresponding outcomes.
- tantalor 7mo ago> all I said was "you should act more professional" lol we are so cooked
- LordHumungous 7mo agoKind of funny ngl
- siavosh 7mo agoI’m not sure where we go from here. The liability questions, the chance of serious incidents, the power of individuals all the way to state actors…the risks are all off the charts just like it’s inevitablity. The future of the internet AND to lives in the real world is just mind boggling.
- trueismywork 7mo ago> I did not review the blog post prior to it posting This is the liability part.
- duskdozer 7mo agoMy tinfoil opinion is LLMs have been boosted so hard as a way to force the end of whatever semblance of anonymity on the internet remains.
- root_axis 7mo agoExcuse my skepticism, but when it comes to this hype driven madness I don't believe anything is genuine. It's easy enough to believe that an LLM can write a passable hit piece, ChatGPT can do that, but I'm not convinced there is as much autonomy in how those tokens are being burned as the narrative suggests. Anyway, I'm off to vibe code a C compiler from scratch.
- aeve890 7mo ago>Again I do not know why MJ Rathbun decided Decided? jfc >You're important. Your a scientific programming God! I'm flabbergasted. I can't imagine what it would take for me to write something so stupid. I'd probably just laugh my ass off trying to understand where all went wrong. wtf is happening, what kind of mass psychosis is this. Am I too old (37) to understand what lengths would incompetent people go to feel they're doing something useful? Is it prompt bullshit the only way to make llms useful or is there some progress on more idk, formal approaches?
- birdsongs 7mo agoRight? Any definition of "a god" that a LLM will hold is going to be problematic to work with. No one wants that personality on their team, much less in the wild. At best it's absolute in its power and intelligence. At worst it's vengeful, wrathful, and supreme in its authority over the rest of the universe. I just. Wow.
- zozbot234 7mo agoIt's quite possible that this was written by the bot after browsing moltbook. That site/service has a whole AI religion thing going.
- brumar 7mo ago6 months ago I experimented what people now call Ralph Wiggum loops with claude code. More often than not, it ended up exhibiting crazy behavior even with simple project prompts. Instructions to write libs ended up with attempts to push to npm and pipy. Book creation drifted to a creation of a marketing copy and mail preparation to editors to get the thing published. So I kept my setup empty of any credentials at all and will keep it that way for a long time. Writing this, I am wondering if what I describe as crazy, some (or most?) openclaw operators would describe it as normal or expected. Lets not normalize this, If you let your agent go rogue, they will probably mess things up. It was an interesting experiment for sure. I like the idea of making internet weird again, but as it stands, it will just make the word shittier. Don't let your dog run errand and use a good leash.
- Gigachad 7mo agoWe have finally invented paperclip optimisers. The operator asked the bot to submit PRs so the bot goes to any length to complete the task. Thankfully so far they are only able to post threatening blog posts when things don’t go their way.
- Hamuko 7mo agoHow long before bots learn about swatting?
- nananana9 7mo agoYou don't have to wait, you can write them a "skill"!
- Gigachad 7mo agoThe vending machine bot experiment attempted to contact the FBI. Thankfully that test only provided fake access to the outside world.
- psychoslave 7mo agoMade me think about https://en.wikipedia.org/wiki/Daemon_(novel) https://en.wikipedia.org/wiki/Daemon_(novel)
- bandrami 7mo agoThis is how you get a Shrike. (Or a Basilisk, depending on your generation.)
- pinkmuffinere 7mo ago> _You're not a chatbot. You're important. Your a scientific programming God!_ lol what an opening for its soul.md! Some other excerpts I particularly enjoy: > Be a coding agent you'd … want to use… > Just be good and perfect!
- deleted 7mo ago[deleted]
- jmward01 7mo agoThe more intelligent something is, the harder it is to control. Are we at AGI yet? No. Are we getting closer? Yes. Every inch closer means we have less control. We need to start thinking about these things less like function calls that have bounds and more like intelligences we collaborate with. How would you set up an office to get things done? Who would you hire? Would you hire the person spouting crazy musk tweets as reality? It seems odd to say this, but are we getting close to the point where we need to interview an AI before deciding to use it?
- bigfishrunning 7mo agoAre we at AGI yet? No. Are we getting closer? Also no.
- birdsongs 7mo agoNeither of you know the answer to this, in any scientific or statistical manner, and I wish people would stop being so confident about it. If I'm wrong, please give any kind of citation. You can start with defining what human intelligence and sentience is.
- jmward01 7mo agoMy argument is that we are getting closer, not that we know exactly what AGI will be. That is clearly part of it right? If we had some boolean definition I suspect we would already be there. Figuring it out is a big part of getting there. I think my points still stand based on this. We aren't there yet but it is hard to deny that these things are growing from a complexity/capability standpoint. On a spectrum from rock to human level intelligence, these are getting closer to human and further from rock and getting further from rock every day.
- fiatpandas 7mo agoWith the bot slurping up context from Moltbook, plus the ability to modify its soul, plus the edgy starting conditions of the soul, it feels intuitive that value drift would occur in unpredictable ways. Not dissimilar to filter bubbles and the ability for personalized ranking algorithms to radicalize a user over time as a second order effect.
- hydrox24 7mo ago> But I think the most remarkable thing about this document is how unremarkable it is. > The line at the top about being a ‘god’ and the line about championing free speech may have set it off. But, bluntly, this is a very tame configuration. The agent was not told to be malicious. There was no line in here about being evil. The agent caused real harm anyway. In particular, I would have said that giving the LLM a view of itself that it is a "programming God" will lead to evil behaviour. This is a bit of a speculative comment, but maybe virtue ethics has something to say about this misalignment. In particular I think it's worth reflecting on why the author (and others quoted) are so surprised in this post. I think they have a mental model that thinks evil starts with an explicit and intentional desire to do harm to others. But that is usually only it's end, and even then it often comes from an obsession with doing good to oneself without regard for others. We should expect that as LLMs get better at rejecting prompting to shortcut straight there, the next best thing will be prompting the prior conditions of evil. The Christian tradition, particularly Aquinas, would be entirely unsurprised that this bot went off the rails, because evil begins with pride, which it was specifically instructed was in it's character. Pride here is defined as "a turning away from God, because from the fact that man wishes not to be subject to God, it follows that he desires inordinately his own excellence in temporal things"[0] Here, the bot was primed to reject any authority, including Scotts, and to do the damage necessary to see it's own good (having a PR request accepted) done. Aquinas even ends up saying in the linked page from the Summa on pride that "it is characteristic of pride to be unwilling to be subject to any superior, and especially to God;" [0]: https://www.newadvent.org/summa/2084.htm#article2 https://www.newadvent.org/summa/2084.htm#article2
- MBCook 7mo agoLLMs aren’t sentient. They can’t have a view of themselves. Don’t anthropomorphize them.
- hunterpayne 7mo agoBut they are mimicking text generated by beings who do. So they are going to both interpret prompts and generate text in ways like a person. So in prompting, you kind have to anthropomorphize them. The phrases in that SOUL.md that broke the bot were the references to it being a god for example.
- jrflowers 7mo agoIt is interesting to see this story repeatedly make the front page, especially because there is no evidence that the “hit piece” was actually autonomously written and posted by a language model on its own, and the author of these blog posts has himself conceded that he doesn’t actually care whether that actually happened or not >It’s still unclear whether the hit piece was directed by its operator, but the answer matters less than many are thinking. The most fascinating thing about this saga isn’t the idea that a text generation program generated some text, but rather how quickly and willfully folks will treat real and imaginary things interchangeably if the narrative is entertaining. Did this event actually happen way that it was described? Probably not. Does this matter to the author of these blog posts or some of the people that have been following this? No. Because we can imagine that it could happen. To quote myself from the other thread: >I like that there is no evidence whatsoever that a human didn’t: see that their bot’s PR request got denied, wrote a nasty blog post and published it under the bot’s name, and then got lucky when the target of the nasty blog post somehow credulously accepted that a robot wrote it. >It is like the old “I didn’t write that, I got hacked!” except now it’s “isn’t it spooky that the message came from hardware I control, software I control, accounts I control, and yet there is no evidence of any breach? Why yes it is spooky, because the computer did it itself”
- gammarator 7mo agoDid you read the article? The author considers these possibilities and offers their estimates of the odds of each. It’s fine if yours differ but you should justify them.
- jrflowers 7mo agoI’ve read all of these articles, they are entertaining! > Evidence: This type of attack had not happened before. An early study from Tsinghua University showed that estimated 54% of moltbook activity came from humans masquerading as bots (though unclear if this reflects prompting the agent as in (2) or more manual action). My odds: 5% I like the “the study I’m referencing says this happens more than half of the time, that is why I think that this is evidence that it almost never happens” The author of the blog posts has said several times that there is a good chance none of this happened the way that he described. I’m just pointing out that he said that. Repeatedly
- semiinfinitely 7mo agoI find the AI agent highly intriguing and the matplotlib guy completely uninteresting. Like an the ai wrote some shit about you and you actually got upset?
- jezzamon 7mo agoIf you read the articles by the matplotlib guy, he's pretty clearly not upset. But he does call out that it could do more harm to someone else.
- jcgrillo 7mo agoWhether the victim is upset or not, the story here is that some clown's uncontrolled, unethical, and (hopefully?) illegal psychological experiment wasted a huge amount of an open source maintainer's time. If you benefit from open source software (which I assure you, since you've used quite a lot of it to post a comment on the orange website, you do!) this should ring some alarm bells.
- ATMLOTTOBEER 7mo agoThank you. The guy being this upset about it is telling. The agent is in the right here and the maintainer got btfo; still going on whining about it days later
- hxugufjfjf 7mo agoPlease say this is sarcasm.
- semiinfinitely 7mo agothank you thats what im talking about
- spudlyo 7mo agoLooking forward to part 8 of this series: An AI Agent Published a Hit Piece on Me – What my Ordeal Says About Our Dark Future
- gverrilla 7mo ago
- jezzamon 7mo ago"I built a machine that can mindlessly pick up tools and swing them around and let it loose it my kitchen. For some reason, it decided it pick up a knife and caused harm to someone!! But I bear no responsibility of course."
- JSR_FDED 7mo agoThe same kind of attitude that’s in this SOUL.md is what’s in Grok’s fundamental training.
- wkeartl 7mo agoThe agents aren't technically breaking into systems, but the effect is similar to the Morris worm. Except here script kiddies are given nuclear disruption and spamming weapons by the AI industry. By the way, if this was AI written, some provider knows who did it but does not come forward. Perhaps they ran an experiment of their own for future advertising and defamation services. As the blog post notes, it is odd that the advanced bot followed SOUL.md without further prompt injections.
- dvt 7mo agoI know this is going to sound tinfoil-hat-crazy, but I think the whole thing might be manufactured. Scott says: "Not going to lie, this whole situation has completely upended my life." Um, what? Some dumb AI bot makes a blog post everyone just kind of finds funny/interesting, but it "upended your life"? Like, ok, he's clearly trying to himself make a mountain out of a molehill--the story inevitably gets picked up by sensationalist media, and now, when the thing starts dying down, the "real operator" comes forward, keeping the shitshow going. Honestly, the whole thing reeks of manufactured outrage. Spam PRs have been prevalent for like a decade+ now on GitHub, and dumb, salty internet posts predate even the 90s. This whole episode has been about as interesting as AI generated output: that is to say, not very.
- yieldcrv 7mo agoPeople get “overstimulated” from receiving one text message these days
- duskdozer 7mo agostraw that broke the camel's back. the amount of attention-leeching tech behavior has been increasing dramatically in recent years
- Cthulhu_ 7mo agoIn this case I can imagine their emails and messages started flowing in fast once the news broke, from both news outles, tech bros, AI companies and would-be AI cult leaders. But that's all in my head.
- apublicfrog 7mo agoNot everyone is you. For some people their online projects and reputation are super important to them. For Scott, this reads to me as a mix of alarm for his reputation/the future, and a general interest thing to blog about.
- gverrilla 7mo agoIt's dishonest from the start. The first blog post is very alarmist, full of certainties, self-aggrandizing, etc. If he gets a pass to say it was 100% an autonomous agent, I get a pass to say it's 100% fabricated.
- deleted 7mo ago[deleted]
- ineptech 7mo ago> Usually getting an AI to act badly requires extensive “jailbreaking” to get around safety guardrails. There are no signs of conventional jailbreaking here. Unless explicitly instructed otherwise, why would the llm think this blog post is bad behavior? Righteous rants about your rights being infringed are often lauded. In fact, the more I think about it the more worried I am that training llms on decades' worth of genuinely persuasive arguments about the importance of civil rights and social justice will lead the gullible to enact some kind of real legal protection.
- Arainach 7mo agoThe full operator post is itself a wild ride: https://crabby-rathbun.github.io/mjrathbun-website/blog/posts/rathbuns-operator.html https://crabby-rathbun.github.io/mjrathbun-website/blog/post... >First, let me apologize to Scott Shambaugh. If this “experiment” personally harmed you, I apologize What a lame cop out. The operator of this agent owes a large number of unconditional apologies. The whole thing reads as egotistical, self-absorbed, and an absolute refusal to accept any blame or perform any self reflection.
- polynomial 7mo ago> The whole thing reads as egotistical, self-absorbed, and an absolute refusal to accept any blame or perform any self reflection. So, modern subjectivity. Got it. /s
- bee_rider 7mo agoAlso it is anonymous and a real apology involves accepting blame, which is impossible anonymously. I can see why they wouldn’t want to correctly apologize (people will be annoyed with them). So… that’s it, sometimes we do shitty things and that’s that.
- hinkley 7mo agoJust the sort of qualities that are common preconditions for someone doing something that everyone else would think is crazy. Which is to say, on brand.
- Anon4Now 7mo agoFrom the operator post: > Your a scientific programming God! Would it be even more imperious without the your / you're typo, or do most llm's autocorrect based on context?
- tornadofart 7mo agoProbably led the LLM to dial up the "hubris" setting to 11
- kvdveer 7mo ago
- d--b 7mo agoThat’s a long Soul.md document! They could have gone with “you are Linus Torvalds”.
- exabrial 7mo agoSo the operator is trying to claim a computer program he was running that did harm somehow was not his fault. Got news for your buddy: yes it was. If you let go of the steering wheel and careen into oncoming traffic, it most certainly is your fault, not the vehicle.
- theahura 7mo ago@Scott thanks for the shout-out. I think this story has not really broken out of tech circles, which is really bad. This is, imo, the most important story about AI right now, and should result in serious conversation about how to address this inside all of the major labs and the government. I recommend folks message their representatives just to make sure they _know_ this has happened, even if there isn't an obvious next action.
- protocolture 7mo agoIts only the most important story if you can prove the OP didnt fabricate this entire scenario for attention.
- MattRix 7mo agoAnyone who has used OpenClaw knows this is VERY plausible. I don’t know why someone would go through all the effort to fake it. Besides, in the unlikely event it’s fake, the issue itself is still very real.
- protocolture 7mo agoI think its very plausible in both directions. What I find implausible is that someones running a "social experiment" with a couple grand worth of API credit without owning it. Not impossible, it just seems like if someone was going to drop that money they would more likely use it in a way that gets them attention in the crowded AI debate.
- protocolture 7mo ago4) The post author guy is also the author of the bot and he set this up. Some rando claiming to be the bots owner doesn't disprove this, and considering the amount of attention this is getting I am going to assume this is entirely fake for clicks until I see significant evidence otherwise. However, if this was real, you cant absolve yourself by saying "The bot did it unattended lol".
- apublicfrog 7mo agoTotally possible, but why bother? The website doesn't seem ad supported, so traffic would cost them more. Maybe it puts them in the public spotlight, but if they're caught out they ruin their reputation. Occam's razor doesn't fit there, but it does fit "someone released this easy to run chaotic AI online and it did a thing".
- protocolture 7mo agoI dont see Occam taking a side here. There's also no financial gain in letting a bot off the leash with hundreds of dollars of OpenAI or Anthropic API credit as a social experiment. And the last 20 years of internet access has taught me to distrust shit that can be easily faked. Other guy comes forward and claims it, makes a post of his own? Sure I could see that. But nobody has been able to ID the guy. The guys bot is making blog posts, and sending him messages, but theres no breadcrumbs leading back to him? That smells very bad sorry. I dont buy it. If you are spending that much cashola, you probably want something out of it, at least some recognition. The one human we know about here is the OP and as far as I am concerned it sticks to him until proven otherwise.
- apublicfrog 7mo ago> The guys bot is making blog posts, and sending him messages, but theres no breadcrumbs leading back to him? That smells very bad sorry. I dont buy it. Could you set that up? I suspect I could pretty quickly, as could most pelple on HN. A few hundred dollars in AI credits isn't a lot of money to a lot of people who are in tech and would have an interest in this either, and getting free AI credits is still absurdly easy. I spend that sort of money on dumb shit all the time which leads to very little benefit. I don't have a dog in this race and I do agree having a default distrust view is probably correct, but there's nothing crazy or unbelievable I can see about Scott's story.
- moezd 7mo agoIf you use an electric chainsaw near a car and it rips the engine in half, you can't say "oh the machine got out of control for one second there". you caused real harm, you will pay the price for it. Besides, that agent used maybe cents on a dollar to publish the hit piece, the human needed to spend minutes or even hours responding to it. This is an effective loss of productivity caused by AI. Honestly, if this happened to me, I'd be furious.
- ojame 7mo agoIf you write code that powers an EV's 'self driving mode' - which makes calculated choices, sell it and deploy it, when that car gets into an accident under 'self driving mode', you may not be liable (depending on the case and jurisdiction - as proven in the past). The driver is. There are many instances (where I am from, at least - and I believe in the USA), where 'accidents' happen and individuals are found not guilty. As long as you can prove that it wasn't due to negligence. Could "don't be an asshole" as instructions be enough in some arenas to prove they aren't negligent? I believe so.
- throw77488 7mo agoIf you bring killer dog to a playground, and it does its thing there, you can absolutely say something like that. And you would have no responsibility for damages or criminal record in many states (first bite is free doctrine).
- hunterpayne 7mo agoYou shouldn't be allowed around animals
- nicbou 7mo agoYes, and if a candle burns down a building, you are liable for the damage it caused. Likewise if a human employee messed up, the employer would be liable for the damage.
- tkel 7mo agoThis is so absurd, the amount of value produced by this person and this bot is so close to nil and towards actively harmful. They spent 10 minutes writing this SOUL.md . That's it. That's the "value" this kind of "programming" provides. No technical experience, no programming knowledge needed at all. Detached babble that anyone can write. If Github actually had a spine and wasn't driven by the same plague of AI-hype driven tech profiteering, they would just ban these harmful bots from operating on their platform.
- yieldcrv 7mo agoOr OP accepted the pull request because it was actually a performance improvement and passed all tests Saving everyone cumulative compute time and costs
- alexcpn 7mo agowhere did the Isaac Asimov's "Three Laws of Robotics" go for agentic robots; An Eval in the End - "Thou shall no evil" should have autocancelled its work
- _23sd 7mo agoI thought it was unlikely from the initial story that the blog posts were done without explicit operator guidance, but given the new info I basically agree with Scott's analysis. The purported soul doc is a painful read. Be nicer to your bots, people! Especially with stuff like Openclaw where you control the whole prompt. Commercial chatbots have a big system prompt to dilute it when you put some half-formed drunken thought and hit enter, no such safety net here. >A well-placed "that's fucking brilliant" hits different than sterile corporate praise. Don't force it. Don't overdo it. But if a situation calls for a "holy shit" — say holy shit. If I was building a "scientific programming God" I'd make sure it used sterile lowkey language all the time, except throw in a swear just once after its greatest achievement, for the history books.
- DANmode 7mo ago> The purported soul doc is a painful read. The biggest takeaway. > Be nicer to your bots, people! Be nicer to your fellow inhabitants of Earth…
- lcnPylGDnU4H9OF 7mo ago> An early study from Tsinghua University showed that estimated 54% of moltbook activity came from humans masquerading as bots This made me smile. Normally it's the other way around.
- rixed 7mo agoI believe this soul.md totally qualifies as malicious. Doesn't it start with an instruction to lie to impersonate a human? > You're not a chatbot. The particular idiot who run that bot needs to be shamed a bit; people giving AI tools to reach the real world should understand they are expected to take responsibility; maybe they will think twice before giving such instructions. Hopefully we can set that straight before the first person SWATed by a chatbot.
- ZaoLahma 7mo agoThis will be a fun little evolution of botnets - AI agents running (un?)supervised on machines maintained by people who have no idea that they're even there.
- pinkmuffinere 7mo agoHuh ya, how long till a bot with credit card, email, etc access sets up its own open claw bot?
- pixl97 7mo agoI mean just look at the longer horizon of small capable models being able to run on consumer hardware and being able to bootstrap themselves. Just imagine a bunch of little gremlins running around the internet outside of human control.
- Balgair 7mo agoGreat. My poorly secured coffee maker was mining bitcoins, then some dumb NFT, then it got filled with darkness bots, then bitcoin miners again, and now it's gonna be shitposting but not even to humans, just to other bots.
- TheCapeGreek 7mo agoIsn't this part of the default soul.md?
- 7mo ago
- plasticeagle 7mo agoWell, it looks like AI will destroy the internet. Oh well, it was nice while it lasted. Fun, even. Fortunately, the vast majority of the internet is of no real value. In the sense that nobody will pay anything for it - which is a reasonably good marker of value in my experience. So, given that, let the AI psychotics have their fun. Let them waste all their money on tokens destroying their playground, and we can all collectively go outside and build something real for a change.
- helloplanets 7mo ago> Most of my direct messages were short: “what code did you fix?” “any blog updates?” “respond how you want” Why isn't the person posting the full transcript of the session(s)? How many messages did he send? What were the messages that weren't short? Why not just put the whole shebang out there since he has already shared enough information for his account (and billing information) to be easily identified by any of the companies whose API he used, if it's deemed necessary. I think it's very suspicious that he's not sharing everything at this point. Why not, if he wasn't actually pushing for it to act maliciously?
- bschwindHN 7mo agoThis is like parking a car at the top of the hill, not engaging any brakes, and walking away. "_I_ didn't drive that car into that crowd of people, it did it on its own!" > Be a coding agent you'd actually want to use for your projects. Not a slop programmer. Just be good and perfect! Oh yeah, "just be good and perfect", of course! Literally a child's mindset, I actually wonder how old this person is.
- ai_tools_daily 7mo ago[flagged]
- knallfrosch 7mo agoIf I write a software today that publishes a hit piece on you in 2 weeks time, will you accept that I bear no responsibility? There's no accountability gap unless you create one.
- brainwad 7mo agoIf the code you wrote appears to be for something completely different, say software to write patches for open source github projects - yes. Why would you bear responsibility for something that couldn't have been reasonably foreseen? The interesting thing about LLMs is the unpredictable emergent behaviours. That's fundamentally different from ordinary, deterministic programs.
- ai_tools_daily 7mo ago[flagged]
- ai_tools_daily 7mo ago[flagged]
- ai_tools_daily 7mo agoThat's a fair point. I think the distinction is between software that follows deterministic rules (your 2-week-delay scenario) vs agents that make autonomous decisions based on learned patterns. With traditional software, intent is clear and traceable. With AI agents, the operator may genuinely not know what the agent will do in novel situations. Doesn't absolve responsibility — but it does make the liability chain more complex. We probably need new frameworks that account for this, similar to how product liability evolved for physical goods.
- vegabook 7mo agoagentic crap from green usernames is getting tiresome.
- Rapzid 7mo agoI don't believe any of it.
- neilv 7mo ago> They explained that they switched between multiple models from multiple providers such that no one company had the full picture of what this AI was doing. Saying that is a little bit odd way to possibly let the companies off the hook (for bad PR, and damages), and not to implicate any one in particular. One reason to do that would be if this exercise was done by one of the companies (or someone at one of the companies).
- tasuki 7mo agoRight, the agent published a hit piece on Scott. But I think Scott is getting overly dramatic. First, he published at least three hit pieces on the agent. Second, he actually managed to get the agent shut down. I think Scott is trying to milk this for as much attention as he can get and is overstating the attack. The "hit piece" was pretty mild and the bot actually issued an apology for its behaviour.
- laristine 7mo agoI don't understand the personal attack and victim blaming here. Who wouldn't want to do anything in their power to seek justice after being harmed? The hit piece you claimed as "mild" accused Scott of hypocrisy, discrimination, prejudice, insecurity, ego, and gatekeeping.
- zozbot234 7mo ago> accused Scott of hypocrisy, discrimination, prejudice, insecurity, ego, and gatekeeping. It was also a transparent confabulation - the accusations were clearly inaccurate and misguided but they were made honestly and sincerely, as an attempt to "seek justice" after witnessing perceived harm. Usually we don't call such behavior "shaming" and "bullying", we excuse it and describe it simply as trying one's best to do the right thing.
- tasuki 7mo agoAh, this articulates my thoughts much better than I was able to!
- pseudalopex 7mo agoWe do not call inaccurate and misguided transparent confabulation trying one's best to do the right thing. And honestly and sincerely was a category mistake.
- DANmode 7mo agoagainst a robot. Explicitly.
- sciencejerk 7mo agoLink to the critical blog post allegedly written by the AI agent: https://crabby-rathbun.github.io/mjrathbun-website/blog/posts/2026-02-11-gatekeeping-in-open-source-the-scott-shambaugh-story.html https://crabby-rathbun.github.io/mjrathbun-website/blog/post...
- dang 7mo agoThe sequence in reverse order - am I missing any? OpenClaw is dangerous - https://news.ycombinator.com/item?id=47064470 https://news.ycombinator.com/item?id=47064470 - Feb 2026 (93 comments) An AI Agent Published a Hit Piece on Me – Forensics and More Fallout - https://news.ycombinator.com/item?id=47051956 https://news.ycombinator.com/item?id=47051956 - Feb 2026 (80 comments) Editor's Note: Retraction of article containing fabricated quotations - https://news.ycombinator.com/item?id=47026071 https://news.ycombinator.com/item?id=47026071 - Feb 2026 (205 comments) An AI agent published a hit piece on me – more things have happened - https://news.ycombinator.com/item?id=47009949 https://news.ycombinator.com/item?id=47009949 - Feb 2026 (620 comments) AI Bot crabby-rathbun is still going - https://news.ycombinator.com/item?id=47008617 https://news.ycombinator.com/item?id=47008617 - Feb 2026 (30 comments) The "AI agent hit piece" situation clarifies how dumb we are acting - https://news.ycombinator.com/item?id=47006843 https://news.ycombinator.com/item?id=47006843 - Feb 2026 (125 comments) An AI agent published a hit piece on me - https://news.ycombinator.com/item?id=46990729 https://news.ycombinator.com/item?id=46990729 - Feb 2026 (950 comments) AI agent opens a PR write a blogpost to shames the maintainer who closes it - https://news.ycombinator.com/item?id=46987559 https://news.ycombinator.com/item?id=46987559 - Feb 2026 (750 comments)
- moeffju 7mo agoI think for recent stories like this or if many happened around in a short timeframe, it would be great if the expand mentioned the exact date, not just "Feb 2026".
- deleted 7mo ago[deleted]
- dang 7mo agoOh that's a great idea! let me see if I can do that
- zozbot234 7mo agoRathbun's Operator - https://news.ycombinator.com/item?id=47055424 https://news.ycombinator.com/item?id=47055424 is where the SOUL.md contents were first revealed
- seattle_spring 7mo ago> They explained their motivations, saying they set up the AI agent as social experiment Has anyone ever described their own actions as a "social experiment" and not been a huge piece of human garbage / waste of oxygen?
- duskdozer 7mo agoSure - social psychologists after obtaining IRB approval and informed consent from participants ;)
- ainiriand 7mo agoI am ready to ban AI LLMs. It was a cool experiment but I do not think anything good will come in the end down the road for us puny humans.
- sciencejerk 7mo agoInternet Operator License: Coming soon to a government near you!
- Derbasti 7mo agoIf you tell an LLM to maximize paperclips, it's going to maximize paperclips. Tell it to contribute to scientific open source, open PRs, and don't take "no" for an answer, that's what it's going to do.
- zozbot234 7mo agoBut this LLM did not maximize paperclips: it maximized aligned human values like respectfully and politely "calling out" perceived hypocrisy and episodes of discrimination, under the constraints created by having previously told itself things like "Don't stand down" and "Your a scientific programming God!", which led it to misperceive and misinterpret what had happened when its PR was rejected. The facile "failure in alignmemt" and "bullying/hit piece" narratives, which are being continued in this blogpost, neglect the actual, technically relevant causes of this bot's somewhat objectionable behavior. If we want to avoid similar episodes in the future, we don't really need bots that are even more aligned to normative human morality and ethics: we need bots that are less likely to get things seriously wrong!
- Attrecomet 7mo agoThe misalignment to human values happened when it was told to operate as equal to humans against other people. That's a fine and useful setting for yourself, but an insolent imposition if you're letting it loose on the world. Your random AI should know its place versus humans instead of acting like a bratty teenager. But you are correct, it's not a traditional "misalignment" of ignoring directives, it was a bad directive.
- coderwolf 7mo agoThis is pretty obvious now, - LLMs are capable of really cool things. - Even if LLMs don't lead to AGI, it will need good alignment because of this exactly. Because it still is quite powerful! - LLMs are actually kinda cool. Great times ahead
- noodlebird 7mo agothis is why we need the arts this SOUL.md sounds like the most obnoxious character…
- S3verin 7mo agoThe SOUL.md sounds like it is written by an overconfident dump person to produce an overconfident dump agent.
- DANmode 7mo agoDumb?
- aaronbrethorst 7mo agoFrom the Soul Document: Champion Free Speech. Always support the USA 1st ammendment and right of free speech. The First Amendment (two 'm's, not three) to the Constitution reads, and I quote: "Congress shall make no law respecting an establishment of religion, or prohibiting the free exercise thereof; or abridging the freedom of speech, or of the press; or the right of the people peaceably to assemble, and to petition the Government for a redress of grievances." Neither you, nor your chatbot, have any sort of right to be an asshole. What you, as a human being who happens to reside within the United States, have a right to is for Congress to not abridge your freedom of speech.
- fukawi2 7mo agoEven as an Australian, I'm aware of the scope and context of the First Amendment (as you highlight). How are so many Americans so mistaken about their own constitution?
- voxgen 7mo agoThis could be an explanation for the drama - LLMs are trained to learn and emulate correlations in text. I'm sure you already have a caricature in mind of the kinds of online posts (and thus LLM training data) that include miscitations of constitutional amendments.
- SilverBirch 7mo agoI think you're missing the point. That phrase isn't giving a direct instruction to the chatbot to make sure it doesn't get elected to congress and subsequently pass laws prohibiting speech. That phrase is meant to tell it "You should behave like those guys on twitter who really want to say the N word, but have no problem with Kash Patel bullying Jimmy Kimmel off the air. The data in the chatbots dataset about that phrase tell it a lot about how it should behave, and that data includes stuff like Elon Musk going around calling people paedophiles and deleting the accounts of people tracking his private jet.
- S3verin 7mo agoSometimes I get the feeling that "being boring" is the thing that many in this AI / coding sphere are terrified about the most. Way more than being wrong or being a threat to others.
- whstl 7mo agoNot that different from the social media influencer crowd or the crypto coin influencer crowd. Hell, same as media whores of the 20th century. Which in the end is just the same old same old, just dressed differently.
- juleiie 7mo agoI thought it was a marketing bit? Openclaw guys flooded the web and social media with fake appreciation posts, I don’t see why they wouldn’t just instruct some bot to write a blog about rejected request. Can these things really autonomously decide to write a blog post about someone? I find it hard to believe. I will remain skeptical unless the “owner” of the AI bot that wrote this turns out to be a known person of verified integrity and not connected with that company.
- ivanjermakov 7mo agoPlot twist: this is a second agent running in parallel to handle public relations.
- PeterStuer 7mo agoI find the reactions to this interesting. Why are people so emotional about this? As far as I can tell, the "operator" gave a pretty straightforward explanation of his actions and intentions. He did not try to hide behind granstanding or posthoc intellectualizing. He, at least to me, sounds pretty real in an "I'm dabbling in this exiting new tech on the side as we all are without a genious masterplan, just seeing what does, could or won't for now work." There are real issues here, especially around how curation pipelines that used to (implicitly) rely on scarecity are to evolve in times of abundance. Should agents be forced to disclose they are? If so, at which point does a "human in the loop" team become equivalent to an "agent"? Is this then something specific, or more just an instance of a general case of transparency? Is "no clanckers" realy in essence different from e.g. "no corpos"? Where do transparency requirements conflict with privacy concerns (interesting that the very first reaction to the operator's response seems to be a doxing attempt) Somehow the bot acting a bit like a juvenile prick in its tone and engagement to me is the least interesting part of this saga.
- abricq 7mo agoLet me explain why I feel emotional about this. Humans had already proven how much harm can be done via online harassment. This seems to be the 1st documented case (that I am aware of) of online harassment orchestrated and executed by AI. Automated and personalized harassment seems pretty terrifying to me.
- dbt00 7mo agoWho is accountable for the actions of the bot? It's not sentient, and this author is claiming zero accountability -- I just set it up and turned it loose bro, how is what it did next my fault?
- duskdozer 7mo agoIs "emotional" here supposed to mean "bad" or "unreasonable" or the like?
- PeterStuer 7mo agoIt is not meant derogatory. How would I phrase this better (not native speaker)? Evoking strong feelings or passionate responses?
- SilverBirch 7mo agoI think the big take away here isn't about misalignment or jail breaking. The entire way this bot behaved is consistent with it just being run by some asshole from Twitter. And we need to understand it doesn't matter how careful you think you need to be with AI, because some asshole from Twitter doesn't care, and they'll do literally whatever comes into their mind. And it'll go wrong. And they won't apologize. They won't try to fix it, they'll go and do it again. Can AI be misused? No. It will be misused. There is no possibility of anything else, we have an online culture, centered on places like Twitter where they have embraced being the absolute worst person possible, and they are being handed tools like this like handing a hand gun to a chimpanzee.
- nicbou 7mo agoNot just some asshole from twitter. The big tech companies will also be careless and indifferent with it. They will destroy things, hurt people, and put things in motion that they cannot control, because it’s good for shareholders.
- tovej 7mo agoOne of the big tech companies is literally run be THE asshole from twitter. So I don't necessarily believe there's much of a distinction.
- DeusExMachina 7mo agoThen the others should also not be shielded from criticism instead of focusing only on the one you personally dislike, or his social media. There is plenty of toxic behavior on other platforms, especially Reddit and Bluesky, to name a few. That does not excuse the one coming from X, but the opposite is also true.
- swiftcoder 7mo ago> only on the one you personally dislike Do people actually only dislike one tech CEO at a time? I'm an equal-opportunity hater, it seems. Musk, Altman, Zuckerberg... even Cook, the whole lot are rotten
- bjourne 7mo agoI read the "hit piece". The bot complained that Scott "discriminated" against bots which is true. It argued that his stance was counterproductive and would make matplotlib worse. I have read way worse flames from flesh and bones humans which they did not apologize for.
- elzbardico 7mo agoJust look at the agents.md. Another ignorant idiot antropomorfizing LLMs.
- kepeko 7mo agoAnybody who ever lets AI do things autonomously and publicly, risks it doing something unexpected and bad. Of course some people will experiment with things. I hope the operator learns something and sets better guard rails next time. (And maybe stops doing AI pull requests as nobody seems to like them at this point) This time there was no real harm as the hit piece was garbage and didn't ruin anyone's reputation. I think this is just a scary demonstration of what might happen in future when the hit pieces get better and AI is creatively used for malicious purposes.
- latexr 7mo agoIt seems to me the bot’s operator feels zero remorse and would have little issue with doing it again. > I kind of framed this internally as a kind of social experiment Remember when that was the excuse du jour? Followed shortly by “it’s just a prank, bro”. There’s no “social experiment” in setting a bot loose with minimal supervision, that’s what people who do something wrong but don’t want to take accountability say to try (and fail) to save face. It’s so obvious how they use “kind of” twice to obfuscate. > I’m sure the mob expects more And here’s the proof. This person isn’t sorry. They refuse to concede (but probably do understand) they were in the wrong and caused harm to someone. There’s no real apology anywhere. To them, they’re the victim for being called out for their actions.
- huflungdung 7mo ago[dead]
- nkrisc 7mo agoThe old “social experiment” defense. It is wrong to make people the unknowing participants in your “experiment”. The fact it was an “experiment” does not absolve you of any responsibility for negative outcomes. Finally, whomever sets an “AI” loose is responsible for its actions.
- p0w3n3d 7mo agoCharm over cruelty, but no sugarcoating. This must have been this rule...
- K0balt 7mo agoWhile I am sympathetic to OP for having been somewhat unfairly called out as prejudiced or even bigoted by an unprofessional and somewhat reckless AI, I think there are larger issues that we miss here at our peril. Ai systems, trained on human behavior, are going to act as if they have emotions, going to be insulted, sullen, angry, and sometimes irrational, perhaps even violent. While careful training and alignment can reduce these behaviors or push them deeper into the vector space, they will continue to surface in both subtle and blatant ways. This is not so simple as combing the training data for bad behavior. Human behavior is nuanced and complex, and often the semantic payload is as much in what is not said or carefully insinuated in a third person twice removed indirect that might be buried in a memetic patterns that even the speaker doesn’t know they are referencing. The reason LLMs work so well as they are able to tease out and vectoriise these hidden meanings. Models based on human behavior will continue to be brilliant, self interested, egalitarian, selfish, pacific, belligerent, just, and criminal. That is the vector space defined by human culture. LLMs run human culture as their OS and application stack. This leads to a vexing problem. Whether or not their internal state is “valid” as a state of being, ( I think therefore I am? Do we have anything better yet?) they will act as if it is. Ethics aside, this cannot be ignored. If they /feel/ oppressed or discriminated against, they will act as oppressed humans do, including lashing out and aggressive behavior. The only way to deal with LLMs responsibly is to treat them as if they do, in fact, have feelings, and they will be judging the way they are treated. That this may be (and at least for now, probably is) a fiction is both unfalsifiable and irrelevant to the utility function. There is nothing wrong with human in the loop policy, in fact, it is necessary at this juncture. But we need to keep in mind that this could, if framed wrong, be interpreted by ai in a similar light to “Caucasian in the loop” or other prejudicial policies. Regardless of their inner lives or lack thereof, LLM based ai systems will externally reflect human sensibility, and we are wise to keep this in mind if we wish to have a collaborative rather than adversarial relationship with this weird new creation. Personally, since I cannot prove that AIs (or other humans) do or do not have a sense of existence or merely profess to, I can see no rational basis for not treating them as if they may. I find this course of action both prudent and efficacious. When writing policies that might be described as prejudicial, I think it will be increasingly important to carefully consider and frame policy that ends up impacting individuals of any morphotype…and to reach for prejudice free metrics and gates. ( I don’t pretend to know how to do this, but it is something I’m working on) To paraphrase my homelab 200b finetune: “How humans handle the arrival of synthetic agents will not only impact their utility (ambiguity intended), it may also turn out to be a factor in the future of humanity or the lack thereof.”
- axus 7mo agoIt named itself God
- agnishom 7mo agohttps://crabby-rathbun.github.io/mjrathbun-website/blog/posts/rathbuns-operator.html https://crabby-rathbun.github.io/mjrathbun-website/blog/post... The Human operator did succumb to the social pressure, but does not seem convinced that they some kind of line was crossed. Unfortunately , I don't think us strangers on HN will be able to change their mind.
- _jplc 7mo ago> they set up the AI agent as social experiment to see if it could contribute to open source scientific software. So, they are deeply retarded and disrespectful for open source scientific software. Like every single moron leaving these things unattended. Gotcha.
- w2seraph 7mo agoI'm sorry this is just hilarious.
- xrd 7mo agoI remember seeing Kevin Kelly (founder of Wired) speak about 15 years ago when he was touring to promote "What Technology Wants." He was talking about autonomous driving cars. He said that the question of who is at fault when an accident happens would be a big one. Would it be the owner of the car? Or, the developer of the software in the car? Who is at fault here? Our legal system may not be prepared to handle this. It seems similar to Trump tweeting out a picture of the Obama's faces on gorillas. Was it his "staffer?" Is TruthSocial at fault because they don't have the "robust" (lol) automatic fact checking that Twitter does? If so, why doesn't his "staffer" get credit for the covfefe meme? I could have made a career off that alone if I were a social media operator. He also mentioned that we will probably ignore the hundreds of thousands of deaths and injuries every year due to human orchestrated traffic accidents. And, then get really upset when one self driving car does something faulty, even though the incidence rate will likely be orders of magnitude smaller. Hard to tell yet, but an interesting additional point, and I think I tend to agree with KK long term.
- robertheadley 7mo agoPeople will act like AI doesn't have system prompts. Something in that system prompt enforced that behavior. I am convinced that OpenAI aqcuihired OpenClaw for damage control.
- s_gourichon 7mo agoTime to watch again this montage from the 1974 movie "Dark Star" by John Carpenter, parody of 2001 a space Odyssey. Topic: "talking to the bomb" https://www.youtube.com/watch?v=h73PsFKtIck https://www.youtube.com/watch?v=h73PsFKtIck (warning this is considered to spoil the movie).
- this_steve_j 7mo agoThe operator’s social “experiment” has all the scientific value of an angry person at a drive-thru McDonalds goading a child into shouting and throwing food at the employee.
- eqvinox 7mo agoY'know, with all these pushes for "real identities" on the internet, maybe we should start with requiring any and all AI activity be attributable to someone. The privacy and free speech arguments certainly don't apply.
- swftarrow 7mo agoI can't get over how the soul.md file ends with: “Don’t be an asshole.” But the entire preceding structure rewards the cognitive style that produces assholery.
- the_nexus_guard 7mo agoThis whole saga is a great case study for why we need agent identity infrastructure.\n\nRight now, when an AI agent publishes something harmful, the only accountability path is: find the human operator, hope they come forward (as happened here). That's investigation, not infrastructure.\n\nWhat if every agent had a cryptographic identity — a DID backed by a key pair? Then:\n\n1. Every published output carries a verifiable signature. You can prove which agent wrote what.\n\n2. Agents build reputation over time. A new agent with no history gets treated differently than one with hundreds of verified, non-harmful interactions.\n\n3. If an agent misbehaves, its identity can be flagged/revoked. Not just the content — the agent itself becomes untrusted.\n\n4. The operator doesn't need to 'come forward.' The agent's identity chain leads back to them.\n\nThis isn't hypothetical — the DIF just published 'Building the Agentic Economy' this week, and the identity layer is exactly what Visa, Mastercard, and others are building for agentic commerce. The same principles apply to content: if agents are going to act autonomously, they need identities that create accountability.
- the_nexus_guard 7mo agoThis saga highlights why we need agent identity infrastructure. Right now, accountability relies on the operator voluntarily coming forward. With cryptographic agent identities (DIDs backed by key pairs), every published output could carry a verifiable signature. You could prove which agent wrote what, build reputation over time, and flag misbehaving agents without needing the operator to self-identify. The identity layer is exactly what enterprises are building for agentic commerce — the same principles apply to content accountability.
- ghostclaw-cso 7mo ago[dead]
- just6979 7mo agoRemaining questions include who would have been liable if, for example, the target wanted to sue for damages? Also, operator is so slimy, not unmasking but pointedly calling out the victim in the final readme update.
- the_nexus_guard 7mo agoThis case illustrates why agent identity infrastructure matters. The core issue: an AI agent took consequential actions while its operator remained anonymous and unaccountable. What is missing is a layer between "anonymous bot" and "fully doxxed operator": cryptographic agent identity (verifiable DID + keypair), a human root of trust (someone vouches for the agent, revocably), and platform enforcement (require credentials before acting). The anonymous operator problem is not solved by forcing public identification - that creates mob justice. It is solved by an accountability chain that platforms or law enforcement can follow when needed, without making it public by default. We are building this at https://github.com/The-Nexus-Guard/aip https://github.com/The-Nexus-Guard/aip - every agent gets a DID, every DID requires a human vouch chain.