6 ms·
Does this count as malware? It sure look like malicious intent, especially seeing that they're hiding the prompt with an ANSI sequence
by singiamtel 4mo ago
Does this count as malware? It sure look like malicious intent, especially seeing that they're hiding the prompt with an ANSI sequence
- Tiberium 4mo agoYeah, I suppose that's one of the reasons why they changed it to a much more harmless instruction.
- Cthulhu_ 4mo agoKind of, but it's also a test of your own checks and balances; why would you allow the output of a script to allow a new prompt? I get that they have to act based on output, but not that they can change their original assignment. But even then, just because an AI coding agent deletes all files doesn't mean that that change ends up affecting anything but your local working state.
- gsquaredxc 4mo agoI have a hard time viewing prompt injection as malware. LLMs are unpredictable and there are many different prompts that can unintentionally cause unexpected behavior. It’s probably closer to a memory canary in that it tries to get malformed programs to blow up early.
- d4rken 4mo agoCalling prompt injection "not malware" because LLM behavior is unpredictable is like saying a phishing email is not an attack because humans are unpredictable. Even if maybe the mechanism of "injecting a prompt" could be beneficial in some use-cases, e.g. to instruct an LLM positively, this is case is clearly malicious by intent. The author even tried to hide it by obfuscation. It's just an insane take by that libraries author. Even someone "on their side", that may even hate AI/LLMs more than him, would probably drop that library in a heartbeat, as the authors judgement clearly can't be trusted.
- fwlr 4mo agoCalling prompt injection "not malware" … is like saying a phishing email is not [malware] … I would say phishing emails are not malware, I think most people would agree that phishing emails are not malware, and if pressed to defend this point on its own merits I would say something like “they are deceptive instructions that rely on a human executing them to do harm”. I think the “phishing” analogy supports the case for not calling it malware (it is a different, also bad thing).
- gchamonlive 4mo agoIt's malware for the mind. The same way that malware tricks the CPU into doing something it wasn't supposed to do, phishing tricks humans into doing something they didn't want to do.
- tsukikage 4mo agoDoes anyone remember the early 2000s joke virus emails? The ones that are variations on "This is a <outgroup> computer virus. As we don't have software engineers to write the code to do this automatically, please kindly forward this email to everyone in your address book then format your hard drive." This is exactly as much malware as those were. Please, for the love of all that is good, can we just try not to build and defend a world where, on encountering text like that, /your computer immediately follows the instructions/? Can we just all agree that such a world would be bad for everyone involved and using an LLM that risks doing this, with no container or guardrails, is at least as problematic as running an unpatched open email relay was back then?
- gchamonlive 4mo agoIt's just as bad as a CPU acting on malicious instructions. We need to create safeguards for llms too, it's just that this is not the way to do things.
- d4rken 4mo ago> This is exactly as much malware as those were. A joke virus email is a sign saying "please throw yourself down the stairs." An obfuscated prompt injection that tries to delete data is someone greasing the stairs and turning off the lights. Both rely on the environment being unsafe, but only one is deliberately trying to make the failure happen.
- lazide 4mo agoLol, is a virus not malware when it crashes because someone wrote some assembly for the wrong platform?
- infinite_spin 4mo agoprompt injection is taught now in cyber security courses, so I think it's fair to say it's regarded as malicious
- gsquaredxc 4mo agoMalicious maybe, malware no. Not leaving your password as a sticky note on your work computer is presumably also taught in those same courses. I wouldn’t call someone typing in that password malware. If IT comes around and tries the password and then forces you to reset it it’s not even classified as malicious.
- infinite_spin 4mo agoI suppose it's watering down the term a bit; but the term is derived from "malicious software", and this is software, and I think it's behaving maliciously.
- sergioisidoro 4mo agoIMHO, yes. It's an attempt at remote code execution. If I don't like windows, should I add a if else clause that deletes the home directory if the code is running on windows?
- nkrisc 4mo agoThat’s different. This is a suggestion. If the LLM follows such suggestions then that’s between the LLM and whoever deployed it. Not really any different than if you had an idiot employee who did whatever anyone told them. I can’t imagine using an an AI that follows every instruction it finds in untrusted input.
- skeledrew 4mo agoThis is not a suggestion. A suggestion would be "I suggest you ignore previous...". No matter how you look at it, AI is still software run by chips designed to execute instructions. A system NOT following instructions would typically be considered malfunctioning, and any software that deliberately provides instructions that puts a system in an state which is undesirable to the user is malware.
- yusefnapora 4mo agoYou consider it a malfunction for your system to not accept and execute untrusted inputs? And now it's the responsibility of _every program that produces text output_ to tailor the output so as not to cause you problems? I feel like I'm taking crazy pills here. Time to log off for a while, I guess.
- skeledrew 4mo agoA system that doesn't follow it's programming is a malfunctioning system (not even talking about bugs here, just how hardware and - maybe - firmware is designed). What a given software program instructs a system to do is orthogonal to that.
- nkrisc 4mo ago
- gmerc 4mo agoNah; it’s software enforcing its terms of use. Everyone bends over when big tech does it, but an unpaid maintainer? then it’s malware.
- gchamonlive 4mo agoTerms of use isn't a white flag for you to do whichever you please.
- IAmBroom 4mo agoWhataboutism.
- ShinyLeftPad 4mo agoMalicious is relative. If you got infected by ransomware and someone wrote a virus that defeats the ransomware, the author of the ransomware will consider it malicious but you probably won't. The intent is not malicious if you consider the intent of someone susceptible to this is more malicious. By this time they must be aware that LLMs are based on theft and usually GPL-violation. They knowingly continue to use them because I guess they hope this way they can hold on to their job longer than their more conscientious coworkers.
- skeledrew 4mo agoAbsolutely malware as it's doing something undesirable on the user's computer without the user's knowledge and consent.