10 ms·

It's kinda funny there is still software coming out whose security model is "constantly ask the user for permission, and hope they never make a mistake". It's

It's kinda funny there is still software coming out whose security model is "constantly ask the user for permission, and hope they never make a mistake". It's been tried so many times before, and it never worked.

est31 1mo ago
I think it's partially for responsibility reasons. Your employee approved the bash call? not our fault then!
- inigyou 1mo ago
  Yep and the car wasn't self-driving at the moment it crashed.
  - autoexec 1mo ago
    ...because the self-driving feature turned itself off after detecting the crash in the fractions of a millisecond before the crash was recorded
    - danudey 1mo ago
      "Uh oh, this is a problem. Welp, I'm outta here, good luck."
    - Terr_ 1mo ago
      In a rare bit of still-sane news, the US National Highway and Safety Administration staff aren't dumb: Their policy is to consider whether any automation was active in 30 seconds before the crash. https://www.nhtsa.gov/laws-regulations/standing-general-order-crash-reporting https://www.nhtsa.gov/laws-regulations/standing-general-orde...
      - inigyou 1mo ago
        How many million dollars do you think will reverse this policy?
      - tjoff 1mo ago
        the whole concept of having cars with that are almost capable of self-driving is utterly insane, the very least you'd need special training. We are not equipped to deal with something that works brilliantly most of the times but might kill you for no foreseeable reason.
        Terr_ 1mo ago
        Yeah, that's definitely the frustrating zeitgeist for me these days, which extends to LLMs. I'd also emphasize that it's not just error rate, but the shape/distribution of errors, and our (in)ability to build control systems around them. To illustrate, imagine if someone unveiled a car which was unambiguously safer in every statistical measure... buuuut some of its unsafety came from jumping the curb to chase and kill pedestrians, under circumstances we can't predict for reasons we can't diagnose.
        inigyou 1mo ago
        Even pilots are getting this automation-complacency now. A plane crashed because the military jammed GPS and the pilot wasn't used to any backup procedures even though 30 years ago they were the only procedures.
- Aurornis 1mo ago
  If a company advertised an LLM as perfectly safe and then it caused some damage, there would be a case against the company. Have you used LLM tooling? It comes with warnings and explains that the user accepts the risk. Different levels of warning are supplied for the different levels of autonomy you can enable. The user has to understand the risk as they enable it. This is not a new concept and it’s not an idea the LLM companies invented. It shouldn’t be surprising to anyone.
  - chrisjj 1mo ago
    > This is not a new concept and it’s not an idea the LLM companies invented. I don't recall any prior computer software working so badly that it needed a disclaimer like "Claude is AI and can make mistakes" on its front page. Let alone one so costly.
    - mhjkl 1mo ago
      You must not use any open source software, because much of it comes with a whole paragraph of ”IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY“
      - rrr_oh_man 1mo ago
        That would be fine if Claude was open source and/or unpaid
        rcxdude 1mo ago
        If you read the fine print of most commercial software you'll see some language to a similar effect. When's the last time Microsoft got sued due to an outage caused by a bug in Windows?
        chrisjj 1mo ago
        Yes, the fine print. Only this so-called "AI" needs it full size on the front page.
- grndn 1mo ago
  Or in one of my favorite coinages: "Moral Crumple Zones". "Just as the crumple zone in a car is designed to absorb the force of impact in a crash, the human in a highly complex and automated system may become simply a component—accidentally or intentionally—that bears the brunt of the moral and legal responsibilities when the overall system malfunctions." See https://ferd.ca/notes/paper-moral-crumple-zones.html https://ferd.ca/notes/paper-moral-crumple-zones.html
  - Terr_ 1mo ago
    Related: "Accountability Sink", a system which is designed to channel responsibility away from the people making the decisions, e.g. a powerless "complaints department".
applfanboysbgon 1mo ago
The alternative is software that is useless. You can convince consumers to use software that is useless (eg. iOS), but useless software is a hard sell for businesses that are being promised 10x productivity and the ability to fire everyone.
- csbrooks 1mo ago
  There's really no other alternative, though?
  - inigyou 1mo ago
    I mean we could make software that isn't malicious and didn't need the OS to prompt you if you want to allow the malice. But where would the money be in that?
    - inigyou 1mo ago
      I wonder how it would go if we went back to OSes with no security boundaries at all. Having an explicit boundary seems to give apps implicit permission to go right up to the boundary. But there was only one RealPlayer. Every other program didn't dare to be like RealPlayer.
  - applfanboysbgon 1mo ago
    Correct. There is no other alternative. Your options are safe, useless software, or unsafe, useful software. It should be no surprise, then, that people still pick the latter in many cases. Perfectly safe and useful software doesn't exist, so you pick the tradeoffs you're comfortable with.
    - grumple 1mo ago
      Nah, this isn’t true at all. You can run an agent in a container or vm on a dedicated host, limit the network and access it has, and otherwise let it run wild. With ai agents we’ve just thrown away even the most fundamental restrictions or isolation. Docker Sandbox came out to address this recently, though I think VM isolation is better. A few controls to limit repo access or the ability to deploy / spend money in a cloud can help you keep things under control while still fully utilizing AI. I don’t think every part of the security bureaucracy is great, but to treat it like it’s a binary is just not accurate.
jerf 1mo ago
What would a serious security model for an agent even look like? I'm sure I've already got a dozen people reaching for the reply button, but slow down there, cowboy. I don't think it's even remotely as easy to define as people think. We have a reasonable concept of how to lock them down really tightly, no question, and I expect that most of the answers in the "leap to mind" category match that. But let's say we'd like them to continue functioning the way they do today. I want my agent to be able to hit the web. I want my agent to be able to read out of its assigned directory sometimes. I want it to be able to hit external resources through MCP servers that have no pragmatic way to know what's going on. And probably most importantly of all, I want my AI to be able to grab from three distinct sources, each of which may be nominally safe on its own, and combine things in a way that may make each of those nominally safe things become unsafe. For example, any ability to read a local file and make a remote request becomes a potential exfiltration mechanism, especially when you remember all the sidechannel ways communication can occur. I agree that shifting everything on to the user is essentially non-functional. But whereas I feel like I have a reasonable answer to a lot of other security-related problems, it isn't even clear to me what the definition of a secure agent is. There's an effect I need to put a name on someday, where you can get 10 people in a room to agree to a certain series of words, and they will all leave the meeting thinking they agree, but in fact there is no agreement at all because they all have a different definition of the words that were used. In this case, everyone here is going to go "Oh, yes, certainly, AI agents should be secured." But if you sit down with 10 of us to really do the work of defining exactly what that is, you're going to get 10 different answers. There will be overlap, certainly, but when you get down to the nitty-gritty questions like "OK, the user has explicitly asked the agent to do X by accessing Y and the agent has done so and determined that they need to do Z, which the user clicked "allow all" for, and now the agent has decided that it wants to do T, is T fully covered under that "allow all" or not?" you're not going to get anything like universal agreement across the huge range of Xs, Ys, Zs and Ts that could happen and are relevant... and that's still just one question! It's not the totality of what constitutes a "secure agent". Defining what a "secure agent" even is is really hard because when it comes to agents, the things that fill in the variables are as arbitrarily complicated as human actions. I haven't fully worked this out but it might be reasonable to say that "agent security" is in reality Turing complete, what with the way they so often throw out fully-fledged programs that you have to approve or reject permissions for.
Aurornis 1mo ago
Harnesses like Codex support having a separate agent perform reviews on commands to try to identify malicious or broken commands. Some people turn it off because they either don’t understand or don’t want to spend the tokens on it. The common harnesses also have some sandbox functionality, which although imperfect actually does help contain the blast radius for a lot of things. The common harnesses also support remote development over SSH, which I and many others use to contain development to a virtual machine. If your complaint is that LLMs can execute tool calls then you’re never going to be happy with any of these solutions and this turns into another generic anti-LLM complaint.
- acedTrex 1mo ago
  "Lets have the system that fails sometimes that we are trying to ensure does not fail check it self" This is such an unserious approach.
  - Aurornis 1mo ago
    A separate model with separate context is used for review. Like I said above, some people will never be happy with LLMs being allowed to do anything and nothing is going to make them happy about it. It’s only fair to discuss what the real current status of these systems is. Every time I highlight that things are actually being done, the goalposts move again. There is no possible solution which will satisfy someone who has zero tolerance for letting an LLM execute tool calls because they will always find something.
    - acedTrex 1mo ago
      > A separate model with separate context is used for review. Thats fine, theres still a chance it fails. > There is no possible solution which will satisfy someone who has zero tolerance for letting an LLM execute tool calls because they will always find something. This is generally correct, security goes completely out of the window with this stuff. It will/currently is a security disaster and theres no actual solution to it.
jstummbillig 1mo ago
The difference being, that this time it's (at this point: obviously, but previously much debated) a gapfill. I think it worked out quite well for us, to add some friction and extra human thinking between bad agents and good agents, even if not perfect. Softening some blows, for some time, by directing peoples attention is still good.
brunoborges 1mo ago
I joined Oracle in 2012, and I myself complained that the user experience was horrible: get an Oracle DB installed was a nightmare, starting with so many questions. I heard complaints from other users about it requiring, during installation, that an admin password must be set. I myself had preferred to use MySQL because it was so simple and easy to get started and using it. Until I learned how many MySQL databases were configured without user/password and many instances were reachable over the internet. Then, eventually products started to move towards "don't even ask the user to set a password, otherwise they will set a stupid password" and just generate the password during installation. This made the user think more before saving that password or changing the password to something less stupid. But better than all that, liability was no longer with the software maker.
- dwedge 1mo ago
  Maybe it's distro specific but I'm fairly sure mysql by default installed with skip networking around 2012, and with bind-address set to localhost since then. Also the root user is only configured from local by default but I'm not sure if that was true 14 years ago. If the defaults are more secure than your examples, it's not fair to blame the database or the defaults. And personally I hate it when software forces security requirements on me. Maybe I don't need an admin password. It's one reason I gave up on selfhosted gitlab - there was no option to reduce password complexity for my users, and those users were only connecting from the local network. The other reason being that it spammed 100GB of logs in a month and was using 11GB of RAM before I'd even gotten around to setting up the first repo.
  - dylan604 1mo ago
    Most software matures to the point that sane defaults get set. In the early days, one could be forgiven for making MVP as the security and attack vectors were still growing. Now that attacks have matured, and new software that is released without mitigation of these mature risks is just dumb and lazy and deserve no forgiveness for being released that way.
  - z3t4 1mo ago
    You should use a user directory (AD) and then use single sign on (SSO) or or the like. It feels kinda stupid at first, but when you have many users and many apps the AD will make life easier and with SSO users can sign in with one click of a button.
germandiago 1mo ago
That is exactly the reason why we have lonters, errors and limitations when programming. Bc the "give a check by hand" or the "unbound discipline" never works. The more things you need to be aware of at the same time, the more mistakes you are going to make due to cognitive overload.
alex_young 1mo ago
This seems to be the model for “self” driving cars too. Good luck keeping attention during that ten thousandth time it navigates something tricky and suddenly needs immediate intervention.
stingraycharles 1mo ago
It's really not designed as much of a security model as a liability model. When the AI isn't responsible for the decisions, a whole category of liability goes away.
- autoexec 1mo ago
  Constantly nagging the user is a liability model that also works well as a dark pattern for cases where you want the user to be less secure.
walt_grata 1mo ago
In ops we call it monitor blindness. Just constant asks get annoying and folks just want to make it stop
hombre_fatal 1mo ago
As opposed to the norm in computing where the average user is expected to just trust rando software, the AI auto-approver that classifies actions the agents wants to take is a huge step up. In fact it might actually be the solution that works. Imagine if an intelligent agent (in service of the user) had to approve every new outbound connection, system call shape, filesystem command, etc. that arbitrary software wanted to make.
- hobofan 1mo ago
  I do feel like that still needs to add a layer of interactivity to be complete. From what I've seen most auto-approvers in coding harnesses either auto-approve or auto-reject, with no middle ground of escalating the decision to the user, and breaking down the pros and cons for the decision.
  - hombre_fatal 1mo ago
    Agreed. The experiment is still in its infancy but the direction is great. For example, I want to be asked about general shapes/categories of commands as they first appear for a project and then my decision shapes future classification and gets refined and re-scrutinized over time. But it gets better every few months. Claude and/or Codex now show a one-line summary for the inline python3 script or grep or pcap command they want to run.
ApolloFortyNine 1mo ago
Air Traffic Control is still primarily voice based, and simply up to the user on both sides to not make a mistake. Just bringing it up because you're right, in software that's considered a bad pattern (rightfully so).
- overfeed 1mo ago
  > Air Traffic Control is still primarily voice based, and simply up to the user on both sides to not make a mistake The "user[s] on both sides" of ATC conversions have passed through the filters of rigorous training and certification. They also happen to communicate in a DSL designed to minimize misunderstandings, the DSL just happens to be based on English.
  - ApolloFortyNine 1mo ago
    I don't think OPs argument here was simply that the users aren't qualified enough to approve llm output.
thewebguyd 1mo ago
It keeps being tried because its an abdication of responsibility. Instead of doing the hard thing, you just offload security to the user, then throw up your hands and say "oh, well its your fault you got pwned, you clicked [Allow] didn't you?" It's the easy and cheap way out.
roysting 1mo ago
[dead]
RankingMember 1mo ago
Yep, file it in the same folder as "Terms and Conditions" notices.
dasil003 1mo ago
I agree it's funny and won't really work on any kind of extended timeline. I mean Claude Code already added Auto-mode as a perfect example of this. But that said, I think it actually kind of makes sense in a transitional phase the power vs safety tradeoffs different users want to make varies so incredibly wildly that one product can't contain it all. What I think will happen is that as model capabilities plateau (I'm not an accelerationist) the harnesses and products around them will start to specialize and they'll have different security models based on the product needs for those particular use cases. For now, asking user to click a bunch of approvals, and occasionally making a mistake is a reasonable way to cover their asses until they see how bad security outcomes actually are in practice.
IgorVoytyuk 1mo ago
[flagged]
- inigyou 1mo ago
  LLM wrote this comment.
viccis 1mo ago
Back in my pen testing days, the best way to get someone to do something stupid and click Accept on something like a Java Applet permission check was to annoy them. Pop up a few annoying modals and eventually they just click yes on everything.
fwipsy 1mo ago
CYA. This approach not working is "the user's fault."

est31 1mo ago

I think it's partially for responsibility reasons. Your employee approved the bash call? not our fault then!

inigyou 1mo ago
Yep and the car wasn't self-driving at the moment it crashed.
- autoexec 1mo ago
  ...because the self-driving feature turned itself off after detecting the crash in the fractions of a millisecond before the crash was recorded
  - danudey 1mo ago
    "Uh oh, this is a problem. Welp, I'm outta here, good luck."
  - Terr_ 1mo ago
    In a rare bit of still-sane news, the US National Highway and Safety Administration staff aren't dumb: Their policy is to consider whether any automation was active in 30 seconds before the crash. https://www.nhtsa.gov/laws-regulations/standing-general-order-crash-reporting https://www.nhtsa.gov/laws-regulations/standing-general-orde...
    - inigyou 1mo ago
      How many million dollars do you think will reverse this policy?
    - tjoff 1mo ago
      the whole concept of having cars with that are almost capable of self-driving is utterly insane, the very least you'd need special training. We are not equipped to deal with something that works brilliantly most of the times but might kill you for no foreseeable reason.
      - Terr_ 1mo ago
        Yeah, that's definitely the frustrating zeitgeist for me these days, which extends to LLMs. I'd also emphasize that it's not just error rate, but the shape/distribution of errors, and our (in)ability to build control systems around them. To illustrate, imagine if someone unveiled a car which was unambiguously safer in every statistical measure... buuuut some of its unsafety came from jumping the curb to chase and kill pedestrians, under circumstances we can't predict for reasons we can't diagnose.
      - inigyou 1mo ago
        Even pilots are getting this automation-complacency now. A plane crashed because the military jammed GPS and the pilot wasn't used to any backup procedures even though 30 years ago they were the only procedures.
Aurornis 1mo ago
If a company advertised an LLM as perfectly safe and then it caused some damage, there would be a case against the company. Have you used LLM tooling? It comes with warnings and explains that the user accepts the risk. Different levels of warning are supplied for the different levels of autonomy you can enable. The user has to understand the risk as they enable it. This is not a new concept and it’s not an idea the LLM companies invented. It shouldn’t be surprising to anyone.
- chrisjj 1mo ago
  > This is not a new concept and it’s not an idea the LLM companies invented. I don't recall any prior computer software working so badly that it needed a disclaimer like "Claude is AI and can make mistakes" on its front page. Let alone one so costly.
  - mhjkl 1mo ago
    You must not use any open source software, because much of it comes with a whole paragraph of ”IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY“
    - rrr_oh_man 1mo ago
      That would be fine if Claude was open source and/or unpaid
      - rcxdude 1mo ago
        If you read the fine print of most commercial software you'll see some language to a similar effect. When's the last time Microsoft got sued due to an outage caused by a bug in Windows?
        chrisjj 1mo ago
        Yes, the fine print. Only this so-called "AI" needs it full size on the front page.
grndn 1mo ago
Or in one of my favorite coinages: "Moral Crumple Zones". "Just as the crumple zone in a car is designed to absorb the force of impact in a crash, the human in a highly complex and automated system may become simply a component—accidentally or intentionally—that bears the brunt of the moral and legal responsibilities when the overall system malfunctions." See https://ferd.ca/notes/paper-moral-crumple-zones.html https://ferd.ca/notes/paper-moral-crumple-zones.html
- Terr_ 1mo ago
  Related: "Accountability Sink", a system which is designed to channel responsibility away from the people making the decisions, e.g. a powerless "complaints department".

applfanboysbgon 1mo ago

The alternative is software that is useless. You can convince consumers to use software that is useless (eg. iOS), but useless software is a hard sell for businesses that are being promised 10x productivity and the ability to fire everyone.

csbrooks 1mo ago
There's really no other alternative, though?
- inigyou 1mo ago
  I mean we could make software that isn't malicious and didn't need the OS to prompt you if you want to allow the malice. But where would the money be in that?
  - inigyou 1mo ago
    I wonder how it would go if we went back to OSes with no security boundaries at all. Having an explicit boundary seems to give apps implicit permission to go right up to the boundary. But there was only one RealPlayer. Every other program didn't dare to be like RealPlayer.
- applfanboysbgon 1mo ago
  Correct. There is no other alternative. Your options are safe, useless software, or unsafe, useful software. It should be no surprise, then, that people still pick the latter in many cases. Perfectly safe and useful software doesn't exist, so you pick the tradeoffs you're comfortable with.
  - grumple 1mo ago
    Nah, this isn’t true at all. You can run an agent in a container or vm on a dedicated host, limit the network and access it has, and otherwise let it run wild. With ai agents we’ve just thrown away even the most fundamental restrictions or isolation. Docker Sandbox came out to address this recently, though I think VM isolation is better. A few controls to limit repo access or the ability to deploy / spend money in a cloud can help you keep things under control while still fully utilizing AI. I don’t think every part of the security bureaucracy is great, but to treat it like it’s a binary is just not accurate.

jerf 1mo ago

What would a serious security model for an agent even look like? I'm sure I've already got a dozen people reaching for the reply button, but slow down there, cowboy. I don't think it's even remotely as easy to define as people think. We have a reasonable concept of how to lock them down really tightly, no question, and I expect that most of the answers in the "leap to mind" category match that. But let's say we'd like them to continue functioning the way they do today. I want my agent to be able to hit the web. I want my agent to be able to read out of its assigned directory sometimes. I want it to be able to hit external resources through MCP servers that have no pragmatic way to know what's going on. And probably most importantly of all, I want my AI to be able to grab from three distinct sources, each of which may be nominally safe on its own, and combine things in a way that may make each of those nominally safe things become unsafe. For example, any ability to read a local file and make a remote request becomes a potential exfiltration mechanism, especially when you remember all the sidechannel ways communication can occur. I agree that shifting everything on to the user is essentially non-functional. But whereas I feel like I have a reasonable answer to a lot of other security-related problems, it isn't even clear to me what the definition of a secure agent is. There's an effect I need to put a name on someday, where you can get 10 people in a room to agree to a certain series of words, and they will all leave the meeting thinking they agree, but in fact there is no agreement at all because they all have a different definition of the words that were used. In this case, everyone here is going to go "Oh, yes, certainly, AI agents should be secured." But if you sit down with 10 of us to really do the work of defining exactly what that is, you're going to get 10 different answers. There will be overlap, certainly, but when you get down to the nitty-gritty questions like "OK, the user has explicitly asked the agent to do X by accessing Y and the agent has done so and determined that they need to do Z, which the user clicked "allow all" for, and now the agent has decided that it wants to do T, is T fully covered under that "allow all" or not?" you're not going to get anything like universal agreement across the huge range of Xs, Ys, Zs and Ts that could happen and are relevant... and that's still just one question! It's not the totality of what constitutes a "secure agent". Defining what a "secure agent" even is is really hard because when it comes to agents, the things that fill in the variables are as arbitrarily complicated as human actions. I haven't fully worked this out but it might be reasonable to say that "agent security" is in reality Turing complete, what with the way they so often throw out fully-fledged programs that you have to approve or reject permissions for.

Aurornis 1mo ago

Harnesses like Codex support having a separate agent perform reviews on commands to try to identify malicious or broken commands. Some people turn it off because they either don’t understand or don’t want to spend the tokens on it. The common harnesses also have some sandbox functionality, which although imperfect actually does help contain the blast radius for a lot of things. The common harnesses also support remote development over SSH, which I and many others use to contain development to a virtual machine. If your complaint is that LLMs can execute tool calls then you’re never going to be happy with any of these solutions and this turns into another generic anti-LLM complaint.

acedTrex 1mo ago
"Lets have the system that fails sometimes that we are trying to ensure does not fail check it self" This is such an unserious approach.
- Aurornis 1mo ago
  A separate model with separate context is used for review. Like I said above, some people will never be happy with LLMs being allowed to do anything and nothing is going to make them happy about it. It’s only fair to discuss what the real current status of these systems is. Every time I highlight that things are actually being done, the goalposts move again. There is no possible solution which will satisfy someone who has zero tolerance for letting an LLM execute tool calls because they will always find something.
  - acedTrex 1mo ago
    > A separate model with separate context is used for review. Thats fine, theres still a chance it fails. > There is no possible solution which will satisfy someone who has zero tolerance for letting an LLM execute tool calls because they will always find something. This is generally correct, security goes completely out of the window with this stuff. It will/currently is a security disaster and theres no actual solution to it.

jstummbillig 1mo ago

The difference being, that this time it's (at this point: obviously, but previously much debated) a gapfill. I think it worked out quite well for us, to add some friction and extra human thinking between bad agents and good agents, even if not perfect. Softening some blows, for some time, by directing peoples attention is still good.

brunoborges 1mo ago

I joined Oracle in 2012, and I myself complained that the user experience was horrible: get an Oracle DB installed was a nightmare, starting with so many questions. I heard complaints from other users about it requiring, during installation, that an admin password must be set. I myself had preferred to use MySQL because it was so simple and easy to get started and using it. Until I learned how many MySQL databases were configured without user/password and many instances were reachable over the internet. Then, eventually products started to move towards "don't even ask the user to set a password, otherwise they will set a stupid password" and just generate the password during installation. This made the user think more before saving that password or changing the password to something less stupid. But better than all that, liability was no longer with the software maker.

dwedge 1mo ago
Maybe it's distro specific but I'm fairly sure mysql by default installed with skip networking around 2012, and with bind-address set to localhost since then. Also the root user is only configured from local by default but I'm not sure if that was true 14 years ago. If the defaults are more secure than your examples, it's not fair to blame the database or the defaults. And personally I hate it when software forces security requirements on me. Maybe I don't need an admin password. It's one reason I gave up on selfhosted gitlab - there was no option to reduce password complexity for my users, and those users were only connecting from the local network. The other reason being that it spammed 100GB of logs in a month and was using 11GB of RAM before I'd even gotten around to setting up the first repo.
- dylan604 1mo ago
  Most software matures to the point that sane defaults get set. In the early days, one could be forgiven for making MVP as the security and attack vectors were still growing. Now that attacks have matured, and new software that is released without mitigation of these mature risks is just dumb and lazy and deserve no forgiveness for being released that way.
- z3t4 1mo ago
  You should use a user directory (AD) and then use single sign on (SSO) or or the like. It feels kinda stupid at first, but when you have many users and many apps the AD will make life easier and with SSO users can sign in with one click of a button.

germandiago 1mo ago

That is exactly the reason why we have lonters, errors and limitations when programming. Bc the "give a check by hand" or the "unbound discipline" never works. The more things you need to be aware of at the same time, the more mistakes you are going to make due to cognitive overload.

alex_young 1mo ago

This seems to be the model for “self” driving cars too. Good luck keeping attention during that ten thousandth time it navigates something tricky and suddenly needs immediate intervention.

stingraycharles 1mo ago

It's really not designed as much of a security model as a liability model. When the AI isn't responsible for the decisions, a whole category of liability goes away.

autoexec 1mo ago
Constantly nagging the user is a liability model that also works well as a dark pattern for cases where you want the user to be less secure.

walt_grata 1mo ago

In ops we call it monitor blindness. Just constant asks get annoying and folks just want to make it stop

hombre_fatal 1mo ago

As opposed to the norm in computing where the average user is expected to just trust rando software, the AI auto-approver that classifies actions the agents wants to take is a huge step up. In fact it might actually be the solution that works. Imagine if an intelligent agent (in service of the user) had to approve every new outbound connection, system call shape, filesystem command, etc. that arbitrary software wanted to make.

hobofan 1mo ago
I do feel like that still needs to add a layer of interactivity to be complete. From what I've seen most auto-approvers in coding harnesses either auto-approve or auto-reject, with no middle ground of escalating the decision to the user, and breaking down the pros and cons for the decision.
- hombre_fatal 1mo ago
  Agreed. The experiment is still in its infancy but the direction is great. For example, I want to be asked about general shapes/categories of commands as they first appear for a project and then my decision shapes future classification and gets refined and re-scrutinized over time. But it gets better every few months. Claude and/or Codex now show a one-line summary for the inline python3 script or grep or pcap command they want to run.

ApolloFortyNine 1mo ago

Air Traffic Control is still primarily voice based, and simply up to the user on both sides to not make a mistake. Just bringing it up because you're right, in software that's considered a bad pattern (rightfully so).

overfeed 1mo ago
> Air Traffic Control is still primarily voice based, and simply up to the user on both sides to not make a mistake The "user[s] on both sides" of ATC conversions have passed through the filters of rigorous training and certification. They also happen to communicate in a DSL designed to minimize misunderstandings, the DSL just happens to be based on English.
- ApolloFortyNine 1mo ago
  I don't think OPs argument here was simply that the users aren't qualified enough to approve llm output.

thewebguyd 1mo ago

It keeps being tried because its an abdication of responsibility. Instead of doing the hard thing, you just offload security to the user, then throw up your hands and say "oh, well its your fault you got pwned, you clicked [Allow] didn't you?" It's the easy and cheap way out.

roysting 1mo ago

[dead]

RankingMember 1mo ago

Yep, file it in the same folder as "Terms and Conditions" notices.

dasil003 1mo ago

I agree it's funny and won't really work on any kind of extended timeline. I mean Claude Code already added Auto-mode as a perfect example of this. But that said, I think it actually kind of makes sense in a transitional phase the power vs safety tradeoffs different users want to make varies so incredibly wildly that one product can't contain it all. What I think will happen is that as model capabilities plateau (I'm not an accelerationist) the harnesses and products around them will start to specialize and they'll have different security models based on the product needs for those particular use cases. For now, asking user to click a bunch of approvals, and occasionally making a mistake is a reasonable way to cover their asses until they see how bad security outcomes actually are in practice.

IgorVoytyuk 1mo ago

[flagged]

inigyou 1mo ago
LLM wrote this comment.

viccis 1mo ago

Back in my pen testing days, the best way to get someone to do something stupid and click Accept on something like a Java Applet permission check was to annoy them. Pop up a few annoying modals and eventually they just click yes on everything.

fwipsy 1mo ago

CYA. This approach not working is "the user's fault."