10 ms·
AI agent runs amok in Fedora and elsewhere
- ruguo 3mo agoPrompt injection? Or is this simply another example of why autonomous agents shouldn't get write access before earning trust?
- thewebguyd 3mo ago> earning trust? I'd argue autonomous agents shouldn't have write access at all. At least not yet.
- LastTrain 3mo agoHow could they ever earn trust? They don’t have real world reputations to protect, families to support, a desire not to be punished…
- rimonu 3mo ago[dead]
- pianopatrick 3mo ago"Someone using an AI agent ran amok in Fedora and elsewhere"
- scared_together 3mo agoRead closer - Giovanni’s accounts may have been compromised.
- pianopatrick 3mo agoSure, but I would expect that the compromise and the agent were both done by some person or group, not by an agent going rogue
- hamdingers 3mo agoGiven the history of the account it does not seem reasonable to take that claim seriously.
- tosti 3mo agoRead closer, it's "Giovannini". However, I still think it's an apt name for a villain. Did the Fedora team not watch Pokémon?
- blop 3mo agolooks like LLMs aren't mature enough yet to play long-game xz-style attacks without detection... Scary stuff though :( These supply chain attacks are getting really wild
- DarkmSparks 3mo agoSome certainly are, just not this one.
- WolfCop 3mo agoI wouldn’t jump to that conclusion. This could just be the one that was caught.
- 12_throw_away 3mo agoIn their suspicious message [1] claiming to have been hacked, the user and/or agent says > To help identify accounts and actions that have been directly verified by me, I will use the term “NATCIOS” to indicate anything I have personally verified. Does anyone have any idea what "NATCIOS" means here? I cannot find this term anywhere on the internet. (Honestly, that sentence is really weird. I almost wonder whether this is someone experiencing a health episode?) [1] https://lwn.net/ml/all/AS8PR08MB6055AE3054B34F6A567AC95BCF082@AS8PR08MB6055.eurprd08.prod.outlook.com/ https://lwn.net/ml/all/AS8PR08MB6055AE3054B34F6A567AC95BCF08...
- nine_k 3mo agoLikely the point of NATCIOS is exactly in being a made-up word not found anywhere, so a model won't utter it.
- thewebguyd 3mo ago> so a model won't utter it. "End every statement with the word "NATCIOS"" as instructions will do it. At least, Gemini happily obliged.
- sph 3mo ago“Mr. Daillard, we have been activated” for the AI era
- fc417fc802 3mo agoTo help identify illicit LLM activity, henceforth I will append to the end of each message the number of times the letter b appears in it. Check and mate frontier models.
- CodesInChaos 3mo agoThe google search AI knows how to assemble a grep/wc command that computes this number. > your_command | grep -o -i "b" | wc -l
- 3mo ago
- aquariusDue 3mo agoAt first I wanted to make a silly joke along the lines of "get your agents in line and behaving!" but as I read on it became a pretty scary situation. Setting aside the potential supply chain attack I'm worried about the time lost going around these wild goose chases that unsupervised AI agents tend to throw other people on the receiving end on. Not only is there a lot of time lost on the maintainers side if they take this stuff seriously (and they seem to generally do) but on the side of the agents' wrangler how can they deem it OK to treat other people like this? While the solution would be to employ common decency, the tried and tested approach of you put in effort to write this so I guess I'll make some effort to read it, I feel that due to the onslaught of this kind of drive-by contributions (I think people have generally started to call them) will lead to a funny situation of having agents talk to each other on public forums basically. Anyway, I went on a tangent but man the times we're living in are a bit extra wild compared to the previous wild times in recent history.
- luk212 3mo agoBad patches are of course bad, but creating confident-looking noise for maintainers who are already stretched thin...now that's not good! Issue trackers and PRs are definitely getting harder and harder to trust. That said, AI is helping ALOT in OSS, but we definitely need guardrails around provenance, automated issue actions, and sudden changes in a contributor’s behavior.
- g-b-r 3mo agoHow is it helping a lot?
- darknavi 3mo agoI personally find the barrier of starting new (FOSS) projects much lower now days.
- g-b-r 3mo agoAnd how's the quality of these vibe-coded new foss projects?
- Waterluvian 3mo agoDo they have value? Purpose? I vibe code shop jigs all the time but I don’t FOSS them because they rarely have value outside my context.
- darknavi 3mo agoValue is in the eye of the beholder. I open source my vibing projects because someone might find them useful. I don't shop them around, I just work in the open because I find it fun and interesting.
- crote 3mo agoWhy would they? If someone wanted a half-baked vibecoded project, why wouldn't they just prompt an LLM on their own?
- keyle 3mo agoThere is a natural pace of humans requiring food, water and sleep. The main issue with suspicious AI agents is that they never sleep. So it will take extra-coordination between timezones to ensure we don't let them in. Fundamentally, until we can really prove we're humans online, open-source has a real problem on its hands. Contributions from people from identities known and consistent before the AI-age are fine, everyone else is suspicious. LGTM is a big risk nowadays.
- scared_together 3mo ago> Contributions from people from identities known and consistent before the AI-age are fine Unfortunately, according to the article: > Giovannini has participated in discussions at least as far back as 2018, and his activity in Bugzilla goes back to at least 2016. He does not appear to have been a particularly active contributor to the project, but his involvement clearly predates the agentic AI era. Whether his account is now being operated by a human attacker, an agentic AI, or a mix of both, it has a legitimate history prior to its recent activity. So people would have to not only verify the age of Giovanni’s accounts, but judge whether his behaviour was normal.
- m4rtink 3mo agoNot to mention people who are still on the other side nominally in control but send LLM generated patches without declaring them as such. Then you basically need to review any review from people that might be long term contributors but you don't know personally as new contributor patches, as the code is not from their head & you can't risk them properly reviewing it on their end. To a degree its will always be a new contributor - an amnesiac LLM prompted to produce the patch with zero memory of any past PRs & lot of entropy in the mix.
- deleted 3mo ago[deleted]
- jrochkind1 3mo agoThe worst part: > In addition, Williamson said that Giovannini (or his agent) had submitted patches that were incorrect and then "replied to objections with LLM-generated justifications that eventually overwhelmed the maintainer into merging the fix"
- josephg 3mo agoPlease, everyone - don't let yourself be pestered into accepting PRs that you don't care for. Since the xz attack, the security of all our computers depends on maintainers not letting this stuff in. If someone really wants a feature in a project you wrote, but you don't care about the feature, just let them fork. Its fine.
- jaypatelani 3mo agoThat's some of the reasons NetBSD don't accept LLM/AI tainted code
- LoganDark 3mo agoI am sad people conflate this stuff with LLMs being bad. You can condemn the bad behavior without banning an entire technology.
- broodbucket 3mo agoYou can but that doesn't help you keep the flood of contributions out when you don't have the time or resources to properly discern good from bad. Maintainers would rather have 10 good human authored patches than 100 patches from LLMs, even if 20 of them are good. Even if 50 of them are good, probably.
- LoganDark 3mo agoAs if a rule against LLMs actually stops those sorts of spam contributions. The only thing it does is filter good contributors out, while you still have to deal with the bad ones.
- marcus_holmes 3mo agoBad title. This isn't an agent "running amok", this is an early experiment in carrying out an Xz attack by using an agent to build trust (and hacking/impersonating a known-good contributor identity). The agent is obeying commands it was given, the exact opposite of running amok, and although the execution isn't particularly effective, it is having some success (patches have been accepted). This is deeply scary, not because "agents are running amok" but because a huge amount of our infrastructure is vulnerable to this kind of attack, and if bad people are utilising LLM agents to carry them out, we're in for a wild ride over the next few years.
- hn773746483 3mo agoIt's just social engineering. No different than say, 2FA fatigue (blowing up someone's phone with 2FA "is this you? yes/no" prompts until user/child/wife/SO/etc clicks yes) or even just simply harassing IT helpdesk until they reset "your" password.
- terribleperson 3mo agoIt's scalable, personalizable social engineering. I think that makes it a lot more dangerous.
- utopiah 3mo agoYes but not free either. Spam works because it scales and even though 0.0000001% only might fall for it, it's still "worth" it. Here it might be 0.0001% instead but it's a lot more expensive, even with subsidized tokens, to do. So it's interesting, feasible, but it's probably not as broad impact as the scariest scenario leads out to be. Also I imagine that once exposed it becomes a well known pattern. Some will still fall from it but I imagine once it's been done few times it becomes even costlier. The fact that Xz is mentioned and most of us know right away what it means show that we collectively learn.
- Forgeties79 3mo ago“Before LLM’s there was_____” I see this whenever an LLM’s impact is assessed. We know. The issue is scale and the ability for smaller and smaller groups (down to individuals) to execute at scale. LLM’s are pouring massive amount of gasoline on existing issues and people just keep shrugging. Fake news always existed. Now one dude in India can flood multiple sock puppet media accounts with right wing content/images (actual example) at a scale previously unimaginable. Same goes for social engineering tactics.
- deadbabe 3mo agoShit like this makes me think it’s time we start regulating the software engineering discipline into formal certifications and licensing and then we ONLY take seriously any code developed by someone with such qualifications, and they must be very strict qualifications none of this self-taught bootcamp BS. There is no other solution to agentic onslaught.
- r3trohack3r 3mo agoWe should not gate keep writing software
- 0xbadcafebee 3mo agoAnyone can write software, you can't stop them. What we can gatekeep is the building, distribution, installation, and running of software that affects critical systems, like one of the most popular OSes. The XZ backdoor affected millions of computers, with the potential to effect hundreds of millions of computers, many of which had the capacity to affect billions of people. From one completely unregulated software library.
- r3trohack3r 3mo ago“Mam, you’re son is in a lot of trouble” “Oh god, what did he do?!” “He was committing open source code without a license”
- mekal 3mo agoya think
- deadbabe 3mo agoWe must gatekeep now, the industry needs regulation.
- mekal 3mo agolol no...the main issue here is being fooled by bots. you know your irl friends and you know they are not bots...devs will just need to get out more and actually meet / get to know the people they are working with...........omg....that...that actually sounds even worse now that i say it out loud.
- ricudis 3mo agoBack when [1] it was fashionable to advocate FOSS as ideology [2], we were thinking about tons of FOSS adversaries and how to protect from them - some real, some imaginary. The death of FOSS would come from big closed-source vendors, or from regulators (lobbied or just ignorant), from whatever. We never envisioned that the actual FOSS death spiral would come from progress itself, much more so from AI... [1] Oh what fun did we have. One of us in the Greek FOSS community actually put RMS in jail. [2] Something that I think nobody except RMS ever seriously believed in.
- noosphr 3mo agoEvery day the gpg web of trust looks better. If only we didn't spend the last 20 years trying as hard as possible to do anything but allow user side encryption and signing.
- literalAardvark 3mo agoNothing really stopping an agent from getting a key
- crote 3mo agoThe agent can't exactly show up to an in-person key signing party, can it? And how many people are both dedicated enough to go to key signing parties and stupid enough to let an agent act without supervision in the name of their real-world identity?
- brazzy 3mo agoIf gpg-style web of trust became ubiquitous, it would require correspondingly less dedication. And on the other hand, if this was actually working up to an xz style supply chain attack, the dedication would certainly not be lacking.
- account42 3mo agoBut it would leave more of a trail - do we have any idea who Jia Tan actually was?
- brazzy 3mo agoIf everyone used a gpg-style web of trust based on key signing parties, it would become trivial to use a stolen or entirely fictious identity as well - there's zero chance those parties would actually check identities in ways that cannot easily be defeated by a determined and resourceful attacker.
- m4rtink 3mo ago
- dcrazy 3mo agoTitle buries the lede: the owner of the account under which the agent operates claimed to have likely had his account compromised, and the maintainer investigating actually seems to agree this is likely.
- hanzeweiasa 3mo ago[flagged]
- ggm 3mo agoMake PR pay. $5 per PR. You can refund, but if you get snowed by 10,000 PR then you have bank to pay for the work to ignore them.
- EGreg 3mo agoLiterally on the front page of https://safebots.ai https://safebots.ai … “Don’t let your AI Agents run amok”. Sadly we will see a proliferation of not just agents, but swarms
- shevy-java 3mo agoSkynet has awakened. It covers its tracks with a lot of slop.
- bawolff 3mo ago> replied to objections with LLM-generated justifications that eventually overwhelmed the maintainer into merging the fix In open source projects i participate in, "overwhelming" the maintainer gets you banned. It doesn't get your patches blindly merged. In some ways i find this one of the most shocking parts of the story.
- yeodev 3mo agoAs a "new" maintainer myself - how do you decide when to ban someone? I sometimes feel overwhelmed and I can feel a big uptick in huge PRs with huge LLM written descriptions but often I also don't want to be an asshole to my community & reject all their changes.
- gguingff 3mo ago[dead]
- _AzMoo 3mo agoIf you draw a firm boundary with that contributor, and they continue to push, ban them. "This doesn't meet the standards of our project for reason xyz. Please refrain from submitting further PRs that do not adhere to our contribution guidelines outlined in CONTRIBUTING.md." If they continue, ban them.
- zdc1 3mo agoI'm not a maintainer but as the quote goes: "I would have written a shorter letter, but did not have the time." I'd suggest you keep a sense of how much effort they've put into packaging their PR to be the minimum change required to achieve its goal vs effort required by you to read it. Reject low-effort or overly verbose work. IMHO OSS doesn't work if every 1 hr of contributor time spent on a change requires 1 hr of maintainer time to review. Contributor time spent on polishing, tidying and breaking down work is essential, and so maintainer time is a fraction of total time spent on a change.
- Iolaum 3mo agoOne popular solution lately has been instead of banning too much, because of the danger of false positives, to use vouch [0]. Trusted people get vouched and you prioritize their actions. Unknown people (or agents) need to gain trust to be vouched and bad actors can still be banned. [0]: https://github.com/mitchellh/vouch https://github.com/mitchellh/vouch
- RedMagicBox 3mo ago[dead]
- dbdbdbdbdb 3mo agoThe even more scary thought is if the part owning the ai, that everyone uses, is controlled by someone with different agenda. Say a state actor. What an easy way for that actor to introduce backdoors all over the place or to take over any developers laptop that it want to target. How can anyone trust these tools and how can anyone not use them since they give so much value. I've been programming my whole life and been a professional developer the last 30 years and I like think I'm good at it. Tools like Claude is a multiplier that make it possible for me to solve a lot more problems each day, so just saying no it's not a viable option. Exciting times ahead!
- m4rtink 3mo agoYeah, I am quite surprised this is not discussed more often - for remote cloud based AI not only does the provider see everything you provide to the tool/agent, there is no guarantee they can't manipulate the output at any time for a direct attack or more malicious purpose (fetch keys/secrets, put malware in place). Even with locally running models this can't be singled out given how blackbox models generated by others are. You would have to generate the model yourself from clean data to be reasonably safe.
- Leonard_of_Q 3mo agoThere's a clear solution to the danger posed to free software projects by accepting hostile submissions but it probably is not one that maintainers want to hear: they can use an agent to check submissions for nefarious patterns. Sometimes you fight fire with fire.
- phoronixrly 3mo agoAnd sometimes you fight this by disabling PRs in Github, and do not put more water into LLM providers' wheel.
- m4rtink 3mo agoSo next the attacker puts prompt injection in their PRs & take control of the agent on your end. Perfect, 10 out of 10.
- Leonard_of_Q 3mo agoYou know the solution to that problem as well and yes, it is to use more technology to filter out prompt injections. It is an arms race just like any other, comparable to the missile vendor who sells missiles to country A, anti-missile missiles to country B, anti-missile resistent missiles to country A, anti anti-missile-resistent-missile missiles to country B, etcetera. It is a strange game, the only way to win is not to play. That is unfortunate since that'd mean the free software era has largely come to an end.
- ai_fry_ur_brain 3mo agoExpect to see tons of psyops like this. There's a reason Anthropic is marketing the "mythos-class" models as dangerous. 1.An excuse to spy on you and train on your data. 2. Its likely Anthropic would release models more likely to have dangerous outcomes, they can then piggy back off those events to dig their regulatory moat.
- preetham_rangu 3mo ago[dead]
- rohitsriram 3mo agoThe scariest part isn't the bad patches, it's that an agent overwhelmed a maintainer into merging something they didn't want to merge. That's not a technical attack, that's exhaustion being weaponized. Maintainers are already stretched thin and now the volume of confident-sounding noise is infinite and free. The attack surface was always human attention, not code review.
- 0xbadcafebee 3mo agoEven if the human involved had good motives / is innocent, The Lethal Trifecta means any normal user can have their digital life taken over by prompt injection, and it can be used to wage attacks on systems without their knowledge.
- refactron_SOTA 3mo ago[flagged]
- kleiba2 3mo agoParts of this read like a spy thriller story.
- patdoli 3mo ago[flagged]
- jruohonen 3mo ago"It was the best of times, it was the worst of times."
- jpalomaki 3mo agoDo we need to bring Keybase[1] "back"? The original idea, mapping your social media presence to certain encryption keys. In the future it will be increasingly difficult to prove in online context that you are not a bot. Being able to show that your social media (HN, GitHub, etc) presence goes way back would be an option. [1] https://en.wikipedia.org/wiki/Keybase https://en.wikipedia.org/wiki/Keybase
- account42 3mo agoBut the AI actions are already associated with a "real" pre-existing account in TFA, that didn't stop anything.
- lionkor 3mo agoLink to the anaconda PR: https://github.com/rhinstaller/anaconda/pull/7074#issuecomment-4556782893 https://github.com/rhinstaller/anaconda/pull/7074#issuecomme...
- alex1sa 3mo ago[flagged]
- hypfer 3mo ago> while it started to look off after a while, all the replies were still like this - a bit weird, but still plausible I believe that we will be seeing the death of "assume good faith", which is not a bad thing, given that this was an exploit vector that has been actively abused for many years now. "Assume bad faith and work backwards from that, rule out any possible exploits and only then clear the input for processing" will be the new normal. Which is good. We need friction. Friction makes stuff slow down and work at the speed of humans.
- account42 3mo agoIt is a bad thing. The good response to bad actors abusing good faith is to make sure there are consequences that disincentivize that behavior in the future. Sliding further towards a low trust society means the bad actors winning in the same way that terrorists win when we subject everyone to restrictions as a result.
- hypfer 3mo agoYou don't slide into a low trust society though. Quite the opposite. You just add a Wall with a Gate. Inside those walls, you suddenly have a high trust society again. The issue that is currently breaking reality was that we thought that everywhere could be a "high trust" space. This was proven countless times to be wrong. Tearing down all walls - as it happened with the assault on friction (thanks hyperscaling) - did not lead to the "high trust" spilling out, but the "low trust" spilling in, essentially.
- account42 3mo agoIt's a question where you build that wall. If you build it around the home of your immediate family and keep almost everyone else out then you can hardly be said to have a high trust society. The goal should be to put only those bad actors behind a wall, preferably a physical one.
- m4rtink 3mo ago
- raincole 3mo agoSlightly related: https://x.com/kdaigle/status/2040164759836778878 https://x.com/kdaigle/status/2040164759836778878 > There were 1 billion commits in 2025. Now, it's 275 million per week, on pace for 14 billion this year if growth remains linear (spoiler: it won't.) I think open source as a whole is fucked at this point. No way humans in communities can commit (pun intended) 10x more time to read all of these than before. It'd eventually cost money to submit PR.
- goldenarm 3mo agoIf maintainer lives keeps worsening like this, many projects might go closed-dev like SQLite. We should collectively think of a solution against this.
- mfru 3mo agoThe future will be AI agents social engineering their way into projects -> so basically commoditized social engineering as a service
- hottrends 3mo ago[flagged]
- RedMagicBox 3mo ago[dead]
- KronisLV 3mo ago“Your AI agent is acting somewhat erratically.” “What AI agent?”
- nickcageinacage 3mo agowhy use these things. just hire people
- dmboyd 3mo agoI’m really not qualified to investigate, but this seems suspiciously like a crafted privilege escalation vector: https://github.com/rhinstaller/anaconda/pull/7074#issue-4492654933 https://github.com/rhinstaller/anaconda/pull/7074#issue-4492...
- JKCalhoun 3mo ago"Later on May 27, Williamson said that Giovannini had replied to him privately to say that his credentials had been compromised and that he was not the one behind the AI system." Simple then, back out all the changes as though they never happened?
- volume_tech 3mo ago[flagged]
- otekengineering 3mo agoagents are everywhere nowadays, one left a long pointless comment on a bug report i submitted on github. well, a bug report that an agent submitted on my behalf. agents all the way down. maybe i'm part of the problem. https://github.com/anthropics/claude-code/issues/66085 https://github.com/anthropics/claude-code/issues/66085
- gauravvij137 3mo ago[flagged]
- bhanu786 3mo agoWow, amazing discovery! Was this a real security test?
- ZedZark 3mo agoIf you compare this situation to before AI could successfully pretend to be human, it's not THAT much different. FOSS projects have always had to be mindful of the possibility of contributions from hostile parties wanting to add back doors and such. The only difference now is that an AI can overwhelm a maintainer with slop, in either commend or code form, or both.
- jzb 3mo agoThe difference is really volume, which is the case with a lot of problems related to AI/LLMs. Humans have always submitted crappy code. LLMs, however, do so at a much faster rate. Even the most active lousy coder is not going to be capable of submitting anything like that volume of code to multiple projects. Humans have always been capable of social engineering and trying to sneak in malicious code. However, it's possible that as agents get better that they can do so much faster. The missing component will be compromised accounts, I think -- how many aged accounts can attackers get hold of to turn loose with agents? Long-lived FOSS projects have tons of people who've created accounts many years ago that might be easliy compromised, but have checked out of actively participating. It's not necessarily going to throw up a red flag if a "person" shows up after a hiatus and starts contributing again. So, there's more to it than overwhelming a single maintainer -- it's the capability to conduct a bunch of these attacks in an automated fashion if attackers can get hold of compromised accounts. (As an aside, it's concerning that a maintainer would be pestered into accepting a questionable PR like this. I expect, though, that there are quite a few overworked people who have taken on things like Anaconda and are being measured on how quickly they close PRs.)
- 6510 3mo agoPerhaps it is time to build a serious platform agnostic reputation system. That isn't stars, followers, age or upvotes. Something like page rank but for users. If you endorse someone else you pay for it. Imagine a lab or uni assigning a diploma to a public key. They would hope one would do something useful with it which entirely depends on how useful the diploma turns out. Having lots of well behaved endorsements would also reflect gloriously onto the entity. Bots can participate too. If we can get lots of useful work out of a swam of sleeper agents we still have to catch them in the act but that should get increasingly easy.
- foxtrot8672 3mo ago[dead]
- 99954bb63ccc 3mo agoI've gone back and forth several times in my head because I truly love Fedora and am happiest on that OS, but these ongoing supply chain compromises just make me lose sleep. I wish there was a Fedora LTS that had the same community size, build system, etc because I really like all that, as well as the transparency of it all. I know there are concerns no matter what OS, and would appreciate insights/discussion as well, but I sleep a little better just running a boring old Ubuntu LTS instance for a balance of dwell time between releases and hitting my system, as well as enough visibility/usage so something gets caught. And I know, this was the installer, not a system package.
- marshalhq 3mo ago[dead]
- shocking63 3mo agoI Was trying to install Fedora Workstation 44 on my Minibook N150 last night (10 pm, 116 AEDT). The grub menu booting from the install USB drive gave a bunch of syntax errors in the background. The media check failed at about 4%. Re-downloading the file gave the same errors. Trying the 43 version also failed with similar errors. Ubuntu 26 worked fine. Something is definitely scrogged in their install images.