11 ms·

Engineers know what needs to be tested, how, to what degree, with which materials, and with what safety margin. AI can be trained on that body of work. But the

Engineers know what needs to be tested, how, to what degree, with which materials, and with what safety margin. AI can be trained on that body of work. But then AI has enough issues in output that it needs to be verified. Otherwise we'll live in a world of titan submersibles ordered by CEOs running on hopes&dreams.

ACCount37 16d ago
Replace "AI" with "humans" and you get the same exact issues. Believe it or not, engineers don't zero shot skyscrapers either. Which is why their work gets reviewed by more engineers. Which catches the issues before they materialize. Sometimes! What's the dreaded impossible-to-replicate human advantage? Because I'm not seeing any.
- fyredge 16d ago
  Agency. Humans don't seem to need a markdown to decide what to do next. Without agency, it's all tools. I guess that's why a 'tool' is sometimes used derogatorily.
  - ACCount37 15d ago
    "LLMs have no agency" had legs in 2022. In 2026 though? In the same 2026 when we have things like "a bunch of proto-GPT-6 agents exploited a test env bug to start talking to each other, and clumped up into an AI hacker team that staged an attack on HuggingFace" happening out in the wild? We're way past "LLMs have no agency", and heading for "LLMs have too much agency".
    - fyredge 15d ago
      [dead]
- _carbyau_ 16d ago
  I think we might be largely in agreement. LLM's should be used to speed up the drudgery, explore creative/evolutionary ideas, but.... > What's the dreaded impossible-to-replicate human advantage? Responsibility. Ultimately LLM's are a tool. Someone has to be responsible for the output. You don't get to use it to create a car/plane/submarine/bridge/building/childs-toy/whatever and then if it bursts into flame, blame the LLM and walk away. Ideally, the person signing off would "know" what they are signing off on. Which requires an engineer.
  - somesortofthing 16d ago
    What happens when the thing they have to be responsible for takes multiple lifetimes to understand on even the most basic level, let alone know deeply enough to feel comfortable taking responsibility for it?
    - _carbyau_ 16d ago
      Again, engineers know what needs to be tested, how, to what degree, with which materials, and with what safety margin. Your widget that mankind may never understand. Well, that's just like anything else in the world. Engineers didn't understand building foundations to anywhere like the degree we do now. Or how to build earthquake proof buildings for a long time. It didn't stop stuff getting built. But it sure affected the confidence in it. And when they honestly don't know, they don't have to approve it. I cited the Titan sub example because one engineer refused to sign off on it and was fired for it. Whoever signed off on it has deaths to weigh on their conscience/reputation - for whatever societal good that is worth. If the latest electric vehicle car never needs to charge but might catastrophically explode randomly between 2-15 years, then the car company doesn't get to take money for the car but handwave away the deaths caused. Legally someone is on the hook. In this case, the engineer who signed off on the magic involved - whether they understood it or not - is at fault. But honestly, people sign off on shit they don't understand all the time. It's a matter of: 1. did their gamble not pay off? 2. were they negligent? If your answer to 2. is "the LLM made a mistake I never checked for in regulation approved ways", then you are negligent.
      - somesortofthing 15d ago
        You can write human-comprehensible regulations(and the AI will helpfully beat whatever it's building into the shape of those regulations, probably against the spirit of them, in ways you can't detect) and you can have humans go through the checklist, but it won't accomplish much. To use your foundation example, it'd be like trying to make go/no go decisions using engineers that don't and can't understand why the foundations go below the building rather than on top of it. Past a certain complexity level, a human approval process is just noise.
- tuesdaynight 16d ago
  I don't think that it will happen in the next decades, but adults cannot imagine a world where humans are not required for most (if not all) jobs. I say adults because kids kind of live in that world if they don't live in a very poor place

ACCount37 16d ago

Replace "AI" with "humans" and you get the same exact issues. Believe it or not, engineers don't zero shot skyscrapers either. Which is why their work gets reviewed by more engineers. Which catches the issues before they materialize. Sometimes! What's the dreaded impossible-to-replicate human advantage? Because I'm not seeing any.

fyredge 16d ago
Agency. Humans don't seem to need a markdown to decide what to do next. Without agency, it's all tools. I guess that's why a 'tool' is sometimes used derogatorily.
- ACCount37 15d ago
  "LLMs have no agency" had legs in 2022. In 2026 though? In the same 2026 when we have things like "a bunch of proto-GPT-6 agents exploited a test env bug to start talking to each other, and clumped up into an AI hacker team that staged an attack on HuggingFace" happening out in the wild? We're way past "LLMs have no agency", and heading for "LLMs have too much agency".
  - fyredge 15d ago
    [dead]
_carbyau_ 16d ago
I think we might be largely in agreement. LLM's should be used to speed up the drudgery, explore creative/evolutionary ideas, but.... > What's the dreaded impossible-to-replicate human advantage? Responsibility. Ultimately LLM's are a tool. Someone has to be responsible for the output. You don't get to use it to create a car/plane/submarine/bridge/building/childs-toy/whatever and then if it bursts into flame, blame the LLM and walk away. Ideally, the person signing off would "know" what they are signing off on. Which requires an engineer.
- somesortofthing 16d ago
  What happens when the thing they have to be responsible for takes multiple lifetimes to understand on even the most basic level, let alone know deeply enough to feel comfortable taking responsibility for it?
  - _carbyau_ 16d ago
    Again, engineers know what needs to be tested, how, to what degree, with which materials, and with what safety margin. Your widget that mankind may never understand. Well, that's just like anything else in the world. Engineers didn't understand building foundations to anywhere like the degree we do now. Or how to build earthquake proof buildings for a long time. It didn't stop stuff getting built. But it sure affected the confidence in it. And when they honestly don't know, they don't have to approve it. I cited the Titan sub example because one engineer refused to sign off on it and was fired for it. Whoever signed off on it has deaths to weigh on their conscience/reputation - for whatever societal good that is worth. If the latest electric vehicle car never needs to charge but might catastrophically explode randomly between 2-15 years, then the car company doesn't get to take money for the car but handwave away the deaths caused. Legally someone is on the hook. In this case, the engineer who signed off on the magic involved - whether they understood it or not - is at fault. But honestly, people sign off on shit they don't understand all the time. It's a matter of: 1. did their gamble not pay off? 2. were they negligent? If your answer to 2. is "the LLM made a mistake I never checked for in regulation approved ways", then you are negligent.
    - somesortofthing 15d ago
      You can write human-comprehensible regulations(and the AI will helpfully beat whatever it's building into the shape of those regulations, probably against the spirit of them, in ways you can't detect) and you can have humans go through the checklist, but it won't accomplish much. To use your foundation example, it'd be like trying to make go/no go decisions using engineers that don't and can't understand why the foundations go below the building rather than on top of it. Past a certain complexity level, a human approval process is just noise.
tuesdaynight 16d ago
I don't think that it will happen in the next decades, but adults cannot imagine a world where humans are not required for most (if not all) jobs. I say adults because kids kind of live in that world if they don't live in a very poor place