7 ms·

Imagine a future where state actors have hundreds of AI agents fixing bugs, gaining reputation while they slowly introduce backdoors. I really hope open source

Imagine a future where state actors have hundreds of AI agents fixing bugs, gaining reputation while they slowly introduce backdoors. I really hope open source models succeed.

pas 2y ago
... if we want security it needs trust anyway. it doesn't matter if it's amazing Code GPT or Chad NSA, the PR needs to be reviewed by someone we trust. it's the trust that's the problem. web of trust purists were right just ahead of the time.
- cqqxo4zV46cp 2y ago
  That just…doesn’t make any sense. Everyone starts from zero and works their way up.
- bee_rider 2y ago
  It would actually be sort of interesting if multiple adversarial intelligence agencies could review and sign commits. We might not trust any particular intelligence agency, but I bet the NSA and China would both be interested in not letting much through, if they knew the other guy was looking.
  - ecshafer 2y ago
    That is an interesting solution. If China, US, Russia, EU, etc all sign off and say "yep this is secure" we should trust it. Since if they think they found an exploit, they might assume the other people found an exploit. This is a little bit like the idea of a fair cut for a cake. If you have two people that want the last slice of cake, you have one cut and the other choose the first slice, since the chooser will choose the biggest slice, so the slicer knowing they will get the smaller will make it as equal as possible. In this case the NSA makes the cut (the code), and Russia / China chooses if its allowed in.
    - Muromec 2y ago
      NSA makes the cut and China picks the public key to use. In all seriousness, those people will quickly find some middle ground and will just share keys with each other
      - bee_rider 2y ago
        Maybe also throw EFF into the mix.
- pvg 2y ago
  Chad NSA It's called the ANS is Chad.
- grepfru_it 2y ago
  this is why microsoft bought github and has been onboarding major open source projects. they will be the trusted 3rd party (whether we like it our not is a different story)
legobmw99 2y ago
Why would open source models make this scenario you are painting better?
- undersuit 2y ago
  Because in the closed source model the frustrated developer that looked into this SSH slowness submits a ticket for the owner of the malicious code to dismiss.
  - legobmw99 2y ago
    This seems completely unrelated to the grandparent comment’s mention of open source LLMs
    - deleted 2y ago
      [deleted]
    - undersuit 2y ago
      You're right, I read the comment as: > Imagine a future where state actors have hundreds of AI agents fixing bugs, gaining reputation while they slowly introduce backdoors. I really hope open source () succeed. I guess we can only hope verifiable and open source models can counteract the state actors.
  - cqqxo4zV46cp 2y ago
    It’s insane to consider the actual discovery of this to be anything other than a lightning strike. What’s more interesting here is that we can say with near certainty that there are other backdoors like this out there.
    - Randalthorro 2y ago
      Time to start looking at similar cases for sure.
  - matrix_overload 2y ago
jnwatson 2y ago
Presumably the state actors are looking for other state actor's bugs, and would try to fix them, or least fix them to only work for them. That's quite a game of cat and mouse.
exitzer0 2y ago
I work for a large closed-source software company and I can tell you with 100% that it is full of domestic and foreign agents. Being open source means that more eyes can and will look at something. That only increases the chance of malicious actions being found out ... just like this supply-chain attack.
- deleted 2y ago
  [deleted]
- naruhodo 2y ago
  Reminds me of the scene in Fight Club where the unreliable narrator is discussing car defects to a fellow airline passenger. Quoting from flawed memory: Passenger: Which company? Narrator: A large one.
4death4 2y ago
Imagine a world where a single OSS maintainer can do the work of 100 of today’s engineers thanks to AI. In the world you describe, it seems likely that contributors would decrease as individual productivity increases.
- Muromec 2y ago
  And be burned out 100x faster
- sureglymop 2y ago
  Wouldn't everything produced by an AI explicitly have to be checked/reviewed by a human? If not, then the attack vector just shifts to the AI model and that's where the backdoor is placed. Sure, one may be 50 times more efficient at maintaining such packages but the problem of verifiably secure systems actually gets worse not better.
starspangled 2y ago
Why AI agents?
deleted 2y ago
[deleted]

Imagine a future where state actors have hundreds of AI agents fixing bugs, gaining reputation while they slowly introduce backdoors. I really hope open source models succeed.

pas 2y ago

... if we want security it needs trust anyway. it doesn't matter if it's amazing Code GPT or Chad NSA, the PR needs to be reviewed by someone we trust. it's the trust that's the problem. web of trust purists were right just ahead of the time.

cqqxo4zV46cp 2y ago
That just…doesn’t make any sense. Everyone starts from zero and works their way up.
bee_rider 2y ago
It would actually be sort of interesting if multiple adversarial intelligence agencies could review and sign commits. We might not trust any particular intelligence agency, but I bet the NSA and China would both be interested in not letting much through, if they knew the other guy was looking.
- ecshafer 2y ago
  That is an interesting solution. If China, US, Russia, EU, etc all sign off and say "yep this is secure" we should trust it. Since if they think they found an exploit, they might assume the other people found an exploit. This is a little bit like the idea of a fair cut for a cake. If you have two people that want the last slice of cake, you have one cut and the other choose the first slice, since the chooser will choose the biggest slice, so the slicer knowing they will get the smaller will make it as equal as possible. In this case the NSA makes the cut (the code), and Russia / China chooses if its allowed in.
  - Muromec 2y ago
    NSA makes the cut and China picks the public key to use. In all seriousness, those people will quickly find some middle ground and will just share keys with each other
    - bee_rider 2y ago
      Maybe also throw EFF into the mix.
pvg 2y ago
Chad NSA It's called the ANS is Chad.
grepfru_it 2y ago
this is why microsoft bought github and has been onboarding major open source projects. they will be the trusted 3rd party (whether we like it our not is a different story)

legobmw99 2y ago

Why would open source models make this scenario you are painting better?

undersuit 2y ago
Because in the closed source model the frustrated developer that looked into this SSH slowness submits a ticket for the owner of the malicious code to dismiss.
- legobmw99 2y ago
  This seems completely unrelated to the grandparent comment’s mention of open source LLMs
  - deleted 2y ago
    [deleted]
  - undersuit 2y ago
    You're right, I read the comment as: > Imagine a future where state actors have hundreds of AI agents fixing bugs, gaining reputation while they slowly introduce backdoors. I really hope open source () succeed. I guess we can only hope verifiable and open source models can counteract the state actors.
- cqqxo4zV46cp 2y ago
  It’s insane to consider the actual discovery of this to be anything other than a lightning strike. What’s more interesting here is that we can say with near certainty that there are other backdoors like this out there.
  - Randalthorro 2y ago
    Time to start looking at similar cases for sure.
- matrix_overload 2y ago

jnwatson 2y ago

Presumably the state actors are looking for other state actor's bugs, and would try to fix them, or least fix them to only work for them. That's quite a game of cat and mouse.

exitzer0 2y ago

I work for a large closed-source software company and I can tell you with 100% that it is full of domestic and foreign agents. Being open source means that more eyes can and will look at something. That only increases the chance of malicious actions being found out ... just like this supply-chain attack.

deleted 2y ago
[deleted]
naruhodo 2y ago
Reminds me of the scene in Fight Club where the unreliable narrator is discussing car defects to a fellow airline passenger. Quoting from flawed memory: Passenger: Which company? Narrator: A large one.

4death4 2y ago

Imagine a world where a single OSS maintainer can do the work of 100 of today’s engineers thanks to AI. In the world you describe, it seems likely that contributors would decrease as individual productivity increases.

Muromec 2y ago
And be burned out 100x faster
sureglymop 2y ago
Wouldn't everything produced by an AI explicitly have to be checked/reviewed by a human? If not, then the attack vector just shifts to the AI model and that's where the backdoor is placed. Sure, one may be 50 times more efficient at maintaining such packages but the problem of verifiably secure systems actually gets worse not better.

starspangled 2y ago

Why AI agents?

deleted 2y ago

[deleted]