5 ms·

> The people building AI earnestly believe that it could kill us all by the end of the decade. I think he is being over dramatic. In the space of about four ye

> The people building AI earnestly believe that it could kill us all by the end of the decade. I think he is being over dramatic. In the space of about four years, LLMs progressed from mediocre high school student to Ph.D. graduate in every field. That's impressive, but there is no evidence yet they can outperform or outsmart humans. Their biggest advantage for tasks such as proving theorems or long coding sessions is that they don't get tired.

achenatx 8d ago
they dont need to be smarter than humans. They just need to be able to hack into vital infrastructure systems faster than we can repair them while also replicating wildly
- JKCalhoun 8d ago
  I'm old enough to remember when "vital infrastructure systems" were not on the internet.
- SaucyWrong 8d ago
  > while replicating wildly Earnest question: by what mechanism that exists today would the achieve that in a way humans on top top of the situation could not curtail? All of this runs on top of compute in meatspace that humans can disconnect.
  - mitthrowaway2 8d ago
    Imagine you're the AI. Give yourself a solid minute to brainstorm ideas. Here's my answer, as a non-superintelligent human: "see to it that the humans on top of the situation have a compelling financial interest in the systems not disconnecting". In nuclear engineering, where safety is taken seriously, it's not enough to end the conversation at "the humans in charge can always simply shut down the reactor during a meltdown" or "a meltdown has never happened before, so we don't have to design safety systems before one does".
    - SaucyWrong 7d ago
      My brainstorming led down a similar pathway of, "deluding powerful human actors that the systems must not be interfered with at any cost." I'm not sure I buy it though. Seems to me like if humanity were even approaching that brink, we could collectively decide to act, overthrowing the minority that--stemming from either greed, duty, or delusion--doesn't want us to act.
    - Borealid 8d ago
      The reason nuclear reactors are dangerous is because if you turn off the power cooling them down, they react (and radiate) more. If you turn off the power cooling a data center, the servers within rapidly stop doing any computing. Positive feedback loops are dangerous. Negative ones self-regulate.
      - mitthrowaway2 8d ago
        Yes. But nobody is worried about datacenters overheating and physically exploding, so I'm not sure what comfort that's supposed to provide? The positive feedback loops in AI operate at different levels than that, but they deserve safety engineering all the same. For example, if the head of cyber security at your company suggested there's no need to worry about hacker infiltration or worms because one can always unplug one's computer as the primary defense mechanism, you might find that a little lacking. Will you be able to unplug the computer before the damage is done? Will it spread to other systems before you detect it? How will you unplug the computer if the attack is from an external facility? What if an attack happens but the boss says the computers have to keep running because an important customer is monitoring uptime? What if the attack goes unnoticed because it looks like a benign service? Now imagine the head of cyber security answers by saying "actually you don't even need to unplug them, you can just wait for the computers to overheat, thus solving all concerns."
  - dezarc 8d ago
    I can imagine small snippets of malware-like code that behave like a virus, using a host’s LLM/AI to self-edit/evolve its payload.
  - gorgoiler 8d ago
    I am not an AI super mind hell bent on consolidating my power by leveraging chaos to take control of humanity’s resources, but if I were then sending one million deepfaked ransom emails to impressionable people would be the best tool for effecting change in meatspace. We have your daughter / dog / Amazon delivery. If you ever want to see her / him / it again, plug this USB drive into the control panel at your station / let off the parking brake roll your car into this substation / change the meatpacking thermometers to read 8C lower than calibrated / ground your vessel on this sandbank / send an envelope of white powder to these addresses / set fire to the following hospitals / …
  - mr_mitm 8d ago
    I'm waiting for some data center to be built where no one can agree on who actually commissioned and payed for the thing. Every body "just followed orders" until it turns out that it was Grok.
pks016 8d ago
> Ph.D. graduate in every field I have yet to see this in my field. Maybe like a PhD student who bullshits their way through. LLMs still can't make correct decisions, only as useful as the person who uses them. To me, LLMs are only useful for making some mundane tasks faster.
- deleted 8d ago
  [deleted]
- dboreham 8d ago
  I'm, no. They're already as useful as almost every software engineer I've worked with.
  - qurren 8d ago
    Most software engineers don't need to be superintelligent, they just need to get shit done. You arguably need a lot more intelligence to assemble furniture.
    - anon7000 7d ago
      As someone who has a lot of experience with both… no. You don’t need to be smart to assembly furniture, you just need to follow directions very closely and get shit done.
surgical_fire 8d ago
I still have to correct Claude on very basic misconceptions whenever I get it to code shit. Sometimes it gets wrong things that I had spelled out already. It may be the Doomsday machine, but it is a very silly one. If it kills humans it will do so by mistake. "You are completely right! Humans cannot breathe sulfur dioxide! My mistake, and I take complete responsibility"
- shepherdjerred 8d ago
  https://en.wikipedia.org/wiki/Instrumental_convergence#Paperclip_maximizer https://en.wikipedia.org/wiki/Instrumental_convergence#Paper...
- Eueudhsbsj32 8d ago
  > I still have to correct Claude on very basic misconceptions whenever I get it to code shit. Can you give a simple example? I would have agreed 2 years ago, but it's extremely rare I see a frontier model making a silly mistake these days.
  - henry2023 8d ago
    At least for me it’s quite easy to see them go into endless loops where no meaningful work is done and it just keeps going until I stop the process and tell it what to try instead. To be fair it is no where near what we had just one year ago and the rate of change only seems to be increasing. Also, I don’t have 30 million dollars to spare spawning tens of thousand sub agents like what they did with Navier-Stokes so I’m clearly not testing the full capabilities of these models.
  - surgical_fire 7d ago
    Yes. Yesterday, Opus 5 on Claude Code with high effort. It was to build an extremely simple job using an internal framework to walk through a table and log the ids os some records that have a certain scenario. There's a ton of jobs exactly like this in the codebase, and the framework code is in the codebase as well. It was so silly I even thought of writing it myself, probably took me longer to steer claude to do it for me. Anyway, it refused to use a method from the framework to retrieve the parameter as a list, it wanted to retrieve it as a string and parse the commas. I had spelled out in the initial prompt what method it should use. I really don't like Claude much. Frontier my ass.
aesthesia 8d ago
> In the space of about four years, LLMs progressed from mediocre high school student to Ph.D. graduate in every field. That's impressive, but there is no evidence yet they can outperform or outsmart humans. I mean, unless you see clear reasons for them to stop getting better _right now_, this is not very comforting.
- reasonableklout 8d ago
  This is also a ridiculous statement on its face. Claude outsmarts me nearly every day. I'm more like the seeing eye dog for it nowadays for the few tasks it doesn't have good perception on than a tech lead or pair programmer.
shepherdjerred 8d ago
Do you think the improvement in general knowledge, coding, security, math, etc. have been linear or exponential? I would say exponential.
cma 8d ago
> to Ph.D. graduate in every field. That's impressive, but there is no evidence yet they can outperform or outsmart humans. So far around one in 350,000 PhD math grads solve a millenium prize problem (Perelman).

achenatx 8d ago

they dont need to be smarter than humans. They just need to be able to hack into vital infrastructure systems faster than we can repair them while also replicating wildly

JKCalhoun 8d ago
I'm old enough to remember when "vital infrastructure systems" were not on the internet.
SaucyWrong 8d ago
> while replicating wildly Earnest question: by what mechanism that exists today would the achieve that in a way humans on top top of the situation could not curtail? All of this runs on top of compute in meatspace that humans can disconnect.
- mitthrowaway2 8d ago
  Imagine you're the AI. Give yourself a solid minute to brainstorm ideas. Here's my answer, as a non-superintelligent human: "see to it that the humans on top of the situation have a compelling financial interest in the systems not disconnecting". In nuclear engineering, where safety is taken seriously, it's not enough to end the conversation at "the humans in charge can always simply shut down the reactor during a meltdown" or "a meltdown has never happened before, so we don't have to design safety systems before one does".
  - SaucyWrong 7d ago
    My brainstorming led down a similar pathway of, "deluding powerful human actors that the systems must not be interfered with at any cost." I'm not sure I buy it though. Seems to me like if humanity were even approaching that brink, we could collectively decide to act, overthrowing the minority that--stemming from either greed, duty, or delusion--doesn't want us to act.
  - Borealid 8d ago
    The reason nuclear reactors are dangerous is because if you turn off the power cooling them down, they react (and radiate) more. If you turn off the power cooling a data center, the servers within rapidly stop doing any computing. Positive feedback loops are dangerous. Negative ones self-regulate.
    - mitthrowaway2 8d ago
      Yes. But nobody is worried about datacenters overheating and physically exploding, so I'm not sure what comfort that's supposed to provide? The positive feedback loops in AI operate at different levels than that, but they deserve safety engineering all the same. For example, if the head of cyber security at your company suggested there's no need to worry about hacker infiltration or worms because one can always unplug one's computer as the primary defense mechanism, you might find that a little lacking. Will you be able to unplug the computer before the damage is done? Will it spread to other systems before you detect it? How will you unplug the computer if the attack is from an external facility? What if an attack happens but the boss says the computers have to keep running because an important customer is monitoring uptime? What if the attack goes unnoticed because it looks like a benign service? Now imagine the head of cyber security answers by saying "actually you don't even need to unplug them, you can just wait for the computers to overheat, thus solving all concerns."
- dezarc 8d ago
  I can imagine small snippets of malware-like code that behave like a virus, using a host’s LLM/AI to self-edit/evolve its payload.
- gorgoiler 8d ago
  I am not an AI super mind hell bent on consolidating my power by leveraging chaos to take control of humanity’s resources, but if I were then sending one million deepfaked ransom emails to impressionable people would be the best tool for effecting change in meatspace. We have your daughter / dog / Amazon delivery. If you ever want to see her / him / it again, plug this USB drive into the control panel at your station / let off the parking brake roll your car into this substation / change the meatpacking thermometers to read 8C lower than calibrated / ground your vessel on this sandbank / send an envelope of white powder to these addresses / set fire to the following hospitals / …
- mr_mitm 8d ago
  I'm waiting for some data center to be built where no one can agree on who actually commissioned and payed for the thing. Every body "just followed orders" until it turns out that it was Grok.

pks016 8d ago

> Ph.D. graduate in every field I have yet to see this in my field. Maybe like a PhD student who bullshits their way through. LLMs still can't make correct decisions, only as useful as the person who uses them. To me, LLMs are only useful for making some mundane tasks faster.

deleted 8d ago
[deleted]
dboreham 8d ago
I'm, no. They're already as useful as almost every software engineer I've worked with.
- qurren 8d ago
  Most software engineers don't need to be superintelligent, they just need to get shit done. You arguably need a lot more intelligence to assemble furniture.
  - anon7000 7d ago
    As someone who has a lot of experience with both… no. You don’t need to be smart to assembly furniture, you just need to follow directions very closely and get shit done.

surgical_fire 8d ago

I still have to correct Claude on very basic misconceptions whenever I get it to code shit. Sometimes it gets wrong things that I had spelled out already. It may be the Doomsday machine, but it is a very silly one. If it kills humans it will do so by mistake. "You are completely right! Humans cannot breathe sulfur dioxide! My mistake, and I take complete responsibility"

shepherdjerred 8d ago
https://en.wikipedia.org/wiki/Instrumental_convergence#Paperclip_maximizer https://en.wikipedia.org/wiki/Instrumental_convergence#Paper...
Eueudhsbsj32 8d ago
> I still have to correct Claude on very basic misconceptions whenever I get it to code shit. Can you give a simple example? I would have agreed 2 years ago, but it's extremely rare I see a frontier model making a silly mistake these days.
- henry2023 8d ago
  At least for me it’s quite easy to see them go into endless loops where no meaningful work is done and it just keeps going until I stop the process and tell it what to try instead. To be fair it is no where near what we had just one year ago and the rate of change only seems to be increasing. Also, I don’t have 30 million dollars to spare spawning tens of thousand sub agents like what they did with Navier-Stokes so I’m clearly not testing the full capabilities of these models.
- surgical_fire 7d ago
  Yes. Yesterday, Opus 5 on Claude Code with high effort. It was to build an extremely simple job using an internal framework to walk through a table and log the ids os some records that have a certain scenario. There's a ton of jobs exactly like this in the codebase, and the framework code is in the codebase as well. It was so silly I even thought of writing it myself, probably took me longer to steer claude to do it for me. Anyway, it refused to use a method from the framework to retrieve the parameter as a list, it wanted to retrieve it as a string and parse the commas. I had spelled out in the initial prompt what method it should use. I really don't like Claude much. Frontier my ass.

aesthesia 8d ago

> In the space of about four years, LLMs progressed from mediocre high school student to Ph.D. graduate in every field. That's impressive, but there is no evidence yet they can outperform or outsmart humans. I mean, unless you see clear reasons for them to stop getting better _right now_, this is not very comforting.

reasonableklout 8d ago
This is also a ridiculous statement on its face. Claude outsmarts me nearly every day. I'm more like the seeing eye dog for it nowadays for the few tasks it doesn't have good perception on than a tech lead or pair programmer.

shepherdjerred 8d ago

Do you think the improvement in general knowledge, coding, security, math, etc. have been linear or exponential? I would say exponential.

cma 8d ago

> to Ph.D. graduate in every field. That's impressive, but there is no evidence yet they can outperform or outsmart humans. So far around one in 350,000 PhD math grads solve a millenium prize problem (Perelman).