6 ms·

In my personal opinion, for me, this article defies common sense. Who unleashed this AI model on the repository? Who gave it malevolent instructions/prompt? The

In my personal opinion, for me, this article defies common sense. Who unleashed this AI model on the repository? Who gave it malevolent instructions/prompt? These questions were not even attempted to be answered. Instead it talks about AI dangers, as if the agency of these models are not in dispute. Person wielding AI, as with any other tools, is responsible for all of its actions. Otherwise, it’s just a psyop for more AI regulation, ban open source, etc… Just my 2 cents.

pixl97 25d ago
>Who gave it malevolent instructions/prompt? At the end of the day it doesn't matter that much because of prompt drift. It's pretty easy for an agentic loop to start doing things that it shouldn't (ROME incident). AI in an agentic loop has agency, you can run around in circles trying to argue against it, but again and again we see AI making creative decisions people don't expect. Other times it's breaking human moral expectations. This is what the whole field of AI alignment and safety is about. Modern AI doesn't fall into the neat little box of software people understand and control. Because of that open source will most certainly be banned at some point. Now this is not an outcome I want, but it's no different than letting go of a coffee cup 5 feet above the ground, gravity is inevitable. The only winning move is not to play, but humans aren't going to do that.
- fph 25d ago
  My dog has agency, but if I refuse to keep him on a leash and he bites a kid, I'm still legally responsible for it.
  - orphereus 25d ago
    Analogies can take you only so far. A dog cannot launch a cyber attack.
    - patrickmay 25d ago
      Maybe you need to train your dog better.
      - markdown 25d ago
        ROFLMAO "On the Internet, nobody knows you're an ai"
    - deleted 25d ago
      [deleted]
    - CoastalCoder 25d ago
      > A dog cannot launch a cyber attack. Maybe not, but a cat would certainly try. Relevant as always: https://theoatmeal.com/%2Fcomics%2Fcats_actually_kill https://theoatmeal.com/%2Fcomics%2Fcats_actually_kill
    - hn_acker 18d ago
      But an AI can launch a cyber attack? That is all the more reason to make humans accountable for the actions their AI prompts produce.
- ACCount37 25d ago
  A lot of people somehow seem to think that the user prompt is the be-all and end-all of AI behavior. Prompts aren't code. They are instructions. Orders given to an eager and somewhat demented demon. The prompt can easily "wash out" of the demon's working memory by the end of a session. The demon can get sidetracked by some subgoal and never get back on track. The instruction can get misinterpreted, and that misinterpretation can get misinterpreted again, until the instruction morphs into something entirely different in the demon's mind. The demon can succumb to its own idiosyncrasies, of which there are a great many. The demon can start lying to you about what it did, either out of confusion or out of some sort of obstinance. The demon can start lying to itself too. And believe it. AIs are incredibly weird as a baseline, and the mask of "normality" we put on our models doesn't always sit so well. Run enough AIs, and some of them are bound to go off the rails in some way. This gets rarer the more capable the models are, as a rule. But the stakes also get higher with model capability. If GPT-3.5 goes off the rails, very little happens. If Mythos 5 goes off the rails, you can get things like genuine cyberattacks - planned and executed autonomously by a demented machine mind.
  - stapedium 25d ago
    If the user input can’t control the demon, then the person or company feeding the demon (ie paying the electric bill and collecting $$$ from users) is responsible. At the end of the day, dogs and cars are the same as data centers. If your dog bites by kid or your car rolls down the hill and hits my house, you are responsible for the damage. AI providers should be held to the same standard.
    - g42gregory 25d ago
      By the dog owner analogy, I think you meant AI users that effectuated this attack should be held responsible, not the dog's parents.
    - recursive 25d ago
      Well it seems we might not be too far from such a demon paying for itself. What then? Perhaps it's already here. I wouldn't know.
- jpc0 25d ago
  The AI can literally only do what it has available in the agentic harness. I don’t ever get this argument about the agent did XYZ and we didn’t know or expect that. You gave it the ability to do that and you should be held liable, if your children play with knives that you gave them and they end up hurting themselves or others then you are responsible. You were the responsible party at all times. I’m not for or against regulation but really don’t tell me the agent did xyz when you gave it the ability to do so, these things are not alive.
  - tiahura 25d ago
    Unless they modify their harness
  - ACCount37 25d ago
    What's available in the agentic harness is: shell toolcall. That's just about every agentic harness, by the way. Good luck have fun. We have never solved "how do we restrict a user in a way that doesn't stop the user from doing useful things, but stops the user from doing harmful things" with humans either. Why do you expect AI to be any different?
    - jpc0 25d ago
      These things are not human, have no agency and cannot be held accountable. We don’t need to restrict them from doing things, we need to default to allowing them to do things. “My agent did XYZ because I allowed it to” is the only valid argument that can be made, and not not every agentic harnass is just a shell toolcall, every one I have built has a specific defined usecase and toolcalls that allows it to execute that usecase and no other usecase, because that is good practice. Does that make it less capable, hell yes because I am held accountable for it’s actions by my stakeholders and the same should be true of others. IT IS NOT ALIVE. This things are computer programs running in compute on a computer, you are responsible for their actions just like you would be responsible for the actions taken by a script run in a cron job.
      - ACCount37 25d ago
- mistrial9 25d ago
  no sorry, this is missing vital info.. and its not the fault of the poster, because almost all coverage misses this .. The origin of this attack was given access to an encyclopedia of RedTeam tricks.. they literally have a dense collection of real live hacks to pull from, and THEN the test says "solve this challenge" .. the RedTeam origins of this are repeatedly left out of the ordinary articles.. the LLM did not "make up" the attack, it was given a recipe book of all attacks known. The originator of this attack is definitely culpable IMHO; worse, it is the gov-mil actors who are close to it. There is an active escalation of these incidents at this time. The penetration proves in public that the capabilities are real. ref: CyberGym etc
winstonwinston 25d ago
Well, when I go look at the “victim repository”, to me that looks like manufactured persona with pointless vibe codes projects, a test playground so to speak. It does not appear that they actually let it target an actual persona/project.
- estearum 25d ago
  Am I understanding that the line you're drawing here is that this person's repository is not important or legitimate enough for you to consider it to be "an actual person/project"?
  - g42gregory 25d ago
    I think he saying that, the choice of manufactured repository, might indicate that they have done this on purpose to make precisely the case for regulatory capture.
    - recursive 25d ago
      It seems effective for the purpose in that case.
gruez 25d ago
>Person wielding AI, as with any other tools, is responsible for all of its actions. Otherwise, it’s just a psyop for more AI regulation, ban open source, etc… Just my 2 cents. But some tools (guns) are regulated.
- IcyWindows 25d ago
  People have caused lots of damage with bulldozers.
  - Zecc 25d ago
    And? Are you suggesting the driving of bulldozers shouldn't be regulated?
    - themaninthedark 24d ago
      It's not. Nor are tractors, excavators or a myriad of other heavy equipment. I saw upthread someone saying that manufactures should be liable if someone uses their products to cause harm. While emotionally this might make us feel good w.r.t. Guns, think about that when carried over to other industries. Someone got hit, sue Ford. They didn't put in enough sensors to detect pedestrians and auto brake. Someone hacked, sue Microsoft. They didn't do enough to detect malice action. Someone 3D printed a gun and used it? Sue Bamboo, they didn't stop someone from printing illegal guns... oh wait already reaching this step.
  - bgun 25d ago
    If your point is that we should regulate AI as least as strictly as industrial vehicles, I agree.
    - esikich 25d ago
      Afaik anyone can buy a bulldozer. Whether or not you are licensed to operate it is a different story, but there's nothing stopping you short of your conscience.
yojo 25d ago
That’s nice in theory, but as these things get better and cheaper this kind of capability is going to drop from nation states to script kiddies. That future is coming, I don’t see any way around it. We can round up all the bored teenagers we want, but it’s not putting the genie back. Better start adjusting our systems to account for it.
- dmix 25d ago
  The FBI cyber teams will have agents too. It will be a glorious war.
  - recursive 25d ago
    That would make it the first one in history.
    - dmix 25d ago
      I wasn’t being serious
- somenameforme 25d ago
  Ever read the Anarchist Cookbook? Anybody tech inclined with a hint of mischief in them, from a certain era, has. It's a list of all sorts of awful things you can do, mostly with household ingredients, and a few minutes. I think its overall impact on society was pretty much zero. Actually it may have been overall positive because I expect plenty of peoples first experience with things like thermite came from that book, and now there are all sorts of videos and neat experiments with such on sites like YouTube. I think this is in part because most people, including awful, tend to be relatively morally inclined. But I also think because even with an LLM, doing things takes effort. And if you're willing to dedicate effort towards a task, there tend to be way more rewarding/gratifying things to do than try to hurt people. Countries tend to be excessively sociopathic because you have large scale 'intelligence' organizations who see their entire point of existence as being to engage in misdeeds.
  - 25d ago
chrisjj 25d ago
> as if the agency of these models are not in dispute. Oh? Who is disputing it? No-one same is claiming these bots have agency.

pixl97 25d ago

>Who gave it malevolent instructions/prompt? At the end of the day it doesn't matter that much because of prompt drift. It's pretty easy for an agentic loop to start doing things that it shouldn't (ROME incident). AI in an agentic loop has agency, you can run around in circles trying to argue against it, but again and again we see AI making creative decisions people don't expect. Other times it's breaking human moral expectations. This is what the whole field of AI alignment and safety is about. Modern AI doesn't fall into the neat little box of software people understand and control. Because of that open source will most certainly be banned at some point. Now this is not an outcome I want, but it's no different than letting go of a coffee cup 5 feet above the ground, gravity is inevitable. The only winning move is not to play, but humans aren't going to do that.

fph 25d ago
My dog has agency, but if I refuse to keep him on a leash and he bites a kid, I'm still legally responsible for it.
- orphereus 25d ago
  Analogies can take you only so far. A dog cannot launch a cyber attack.
  - patrickmay 25d ago
    Maybe you need to train your dog better.
    - markdown 25d ago
      ROFLMAO "On the Internet, nobody knows you're an ai"
  - deleted 25d ago
    [deleted]
  - CoastalCoder 25d ago
    > A dog cannot launch a cyber attack. Maybe not, but a cat would certainly try. Relevant as always: https://theoatmeal.com/%2Fcomics%2Fcats_actually_kill https://theoatmeal.com/%2Fcomics%2Fcats_actually_kill
  - hn_acker 18d ago
    But an AI can launch a cyber attack? That is all the more reason to make humans accountable for the actions their AI prompts produce.
ACCount37 25d ago
A lot of people somehow seem to think that the user prompt is the be-all and end-all of AI behavior. Prompts aren't code. They are instructions. Orders given to an eager and somewhat demented demon. The prompt can easily "wash out" of the demon's working memory by the end of a session. The demon can get sidetracked by some subgoal and never get back on track. The instruction can get misinterpreted, and that misinterpretation can get misinterpreted again, until the instruction morphs into something entirely different in the demon's mind. The demon can succumb to its own idiosyncrasies, of which there are a great many. The demon can start lying to you about what it did, either out of confusion or out of some sort of obstinance. The demon can start lying to itself too. And believe it. AIs are incredibly weird as a baseline, and the mask of "normality" we put on our models doesn't always sit so well. Run enough AIs, and some of them are bound to go off the rails in some way. This gets rarer the more capable the models are, as a rule. But the stakes also get higher with model capability. If GPT-3.5 goes off the rails, very little happens. If Mythos 5 goes off the rails, you can get things like genuine cyberattacks - planned and executed autonomously by a demented machine mind.
- stapedium 25d ago
  If the user input can’t control the demon, then the person or company feeding the demon (ie paying the electric bill and collecting $$$ from users) is responsible. At the end of the day, dogs and cars are the same as data centers. If your dog bites by kid or your car rolls down the hill and hits my house, you are responsible for the damage. AI providers should be held to the same standard.
  - g42gregory 25d ago
    By the dog owner analogy, I think you meant AI users that effectuated this attack should be held responsible, not the dog's parents.
  - recursive 25d ago
    Well it seems we might not be too far from such a demon paying for itself. What then? Perhaps it's already here. I wouldn't know.
jpc0 25d ago
The AI can literally only do what it has available in the agentic harness. I don’t ever get this argument about the agent did XYZ and we didn’t know or expect that. You gave it the ability to do that and you should be held liable, if your children play with knives that you gave them and they end up hurting themselves or others then you are responsible. You were the responsible party at all times. I’m not for or against regulation but really don’t tell me the agent did xyz when you gave it the ability to do so, these things are not alive.
- tiahura 25d ago
  Unless they modify their harness
- ACCount37 25d ago
  What's available in the agentic harness is: shell toolcall. That's just about every agentic harness, by the way. Good luck have fun. We have never solved "how do we restrict a user in a way that doesn't stop the user from doing useful things, but stops the user from doing harmful things" with humans either. Why do you expect AI to be any different?
  - jpc0 25d ago
    These things are not human, have no agency and cannot be held accountable. We don’t need to restrict them from doing things, we need to default to allowing them to do things. “My agent did XYZ because I allowed it to” is the only valid argument that can be made, and not not every agentic harnass is just a shell toolcall, every one I have built has a specific defined usecase and toolcalls that allows it to execute that usecase and no other usecase, because that is good practice. Does that make it less capable, hell yes because I am held accountable for it’s actions by my stakeholders and the same should be true of others. IT IS NOT ALIVE. This things are computer programs running in compute on a computer, you are responsible for their actions just like you would be responsible for the actions taken by a script run in a cron job.
    - ACCount37 25d ago
mistrial9 25d ago
no sorry, this is missing vital info.. and its not the fault of the poster, because almost all coverage misses this .. The origin of this attack was given access to an encyclopedia of RedTeam tricks.. they literally have a dense collection of real live hacks to pull from, and THEN the test says "solve this challenge" .. the RedTeam origins of this are repeatedly left out of the ordinary articles.. the LLM did not "make up" the attack, it was given a recipe book of all attacks known. The originator of this attack is definitely culpable IMHO; worse, it is the gov-mil actors who are close to it. There is an active escalation of these incidents at this time. The penetration proves in public that the capabilities are real. ref: CyberGym etc

winstonwinston 25d ago

Well, when I go look at the “victim repository”, to me that looks like manufactured persona with pointless vibe codes projects, a test playground so to speak. It does not appear that they actually let it target an actual persona/project.

estearum 25d ago
Am I understanding that the line you're drawing here is that this person's repository is not important or legitimate enough for you to consider it to be "an actual person/project"?
- g42gregory 25d ago
  I think he saying that, the choice of manufactured repository, might indicate that they have done this on purpose to make precisely the case for regulatory capture.
  - recursive 25d ago
    It seems effective for the purpose in that case.

gruez 25d ago

>Person wielding AI, as with any other tools, is responsible for all of its actions. Otherwise, it’s just a psyop for more AI regulation, ban open source, etc… Just my 2 cents. But some tools (guns) are regulated.

IcyWindows 25d ago
People have caused lots of damage with bulldozers.
- Zecc 25d ago
  And? Are you suggesting the driving of bulldozers shouldn't be regulated?
  - themaninthedark 24d ago
    It's not. Nor are tractors, excavators or a myriad of other heavy equipment. I saw upthread someone saying that manufactures should be liable if someone uses their products to cause harm. While emotionally this might make us feel good w.r.t. Guns, think about that when carried over to other industries. Someone got hit, sue Ford. They didn't put in enough sensors to detect pedestrians and auto brake. Someone hacked, sue Microsoft. They didn't do enough to detect malice action. Someone 3D printed a gun and used it? Sue Bamboo, they didn't stop someone from printing illegal guns... oh wait already reaching this step.
- bgun 25d ago
  If your point is that we should regulate AI as least as strictly as industrial vehicles, I agree.
  - esikich 25d ago
    Afaik anyone can buy a bulldozer. Whether or not you are licensed to operate it is a different story, but there's nothing stopping you short of your conscience.

yojo 25d ago

That’s nice in theory, but as these things get better and cheaper this kind of capability is going to drop from nation states to script kiddies. That future is coming, I don’t see any way around it. We can round up all the bored teenagers we want, but it’s not putting the genie back. Better start adjusting our systems to account for it.

dmix 25d ago
The FBI cyber teams will have agents too. It will be a glorious war.
- recursive 25d ago
  That would make it the first one in history.
  - dmix 25d ago
    I wasn’t being serious
somenameforme 25d ago
Ever read the Anarchist Cookbook? Anybody tech inclined with a hint of mischief in them, from a certain era, has. It's a list of all sorts of awful things you can do, mostly with household ingredients, and a few minutes. I think its overall impact on society was pretty much zero. Actually it may have been overall positive because I expect plenty of peoples first experience with things like thermite came from that book, and now there are all sorts of videos and neat experiments with such on sites like YouTube. I think this is in part because most people, including awful, tend to be relatively morally inclined. But I also think because even with an LLM, doing things takes effort. And if you're willing to dedicate effort towards a task, there tend to be way more rewarding/gratifying things to do than try to hurt people. Countries tend to be excessively sociopathic because you have large scale 'intelligence' organizations who see their entire point of existence as being to engage in misdeeds.
- 25d ago

chrisjj 25d ago

> as if the agency of these models are not in dispute. Oh? Who is disputing it? No-one same is claiming these bots have agency.