10 ms·

> Here’s the thing: most of our habits at work still assume that producing something means you have to understand it. This has only ever been true in a relativ

> Here’s the thing: most of our habits at work still assume that producing something means you have to understand it. This has only ever been true in a relative sense. Until recently, one was expected to understand the software-level code they wrote. But not the machine code, or the computer internals at a deep level. You trust that those layers work. At some point in the past you couldn’t just trust that. At some point in the near future, you will just trust that the software code works. Instead you’ll just debate the spec (the prompt). Your thinking can move one level up the system hierarchy. Everyone’s can. We’ve done it before, we’ll do it again, and this won’t even be the last time it happens. Let go of the anxiety and let it happen.

coldtea 6d ago
>This has only ever been true in a relative sense. Until recently, one was expected to understand the software-level code they wrote. But not the machine code, or the computer internals at a deep level. Just because something wasn't required at all levels or 100% it doesn't mean it wasn't required or it wasn't important though. I trusted that the machine code worked because it was a mostly mechanical transformation, with little to zero non-determinism involved, and the design decisions for the high level code were still mine and understood. Not the same with an LLM, even when see just as an added level of abstraction. Given some LLM produced code, I also trust the compiler will produce working machine code matching it. Doesn't mean I trust the LLM code in the same way.
meander_water 6d ago
You trust machine code because it was produced by a deterministic system that was written and tested thoroughly by other engineers. AI systems are not the same. You can't guarantee deterministic output. Also, code is still the best language we have to convey our intentions as engineers into function. Natural language is too imprecise and ambiguous. Understanding the code allows us to understand the intention behind the code and identify bugs, plan and architect better. Inspecting prompts gives you a blinkered view of the system meaning you're more likely to make incorrect assumptions leading to serious bugs.
- Terr_ 6d ago
  Right, it's a chain of trust between the layers. The link I choose next to me helps determine the quality of successive links, whether I inspect them or not. When I buy a wooden shack from a carpenter, I'd never expect them to core-sample every plank to ensure it's good wood. That's because a good carpenter habitually buys wood from a good supplier, and will have the experience to detect other suspicious indicators. Similarly, my coworker might not have verified the machine code, but I can be pretty confident they didn't vibe-install a compiler by Ken Thompson that puts backdoors into the binary.
- faangguyindia 6d ago
  Are your teammates deterministic output machines?
  - atomicnumber3 6d ago
    In a sense yes. I trust that my coworkers will be accountable for their outcomes and accomplish them with the high quality bar I know they intrinsically hold themselves to. And that they understand our shared goals and if they don't, they will work to become aligned. And if I can't say that for a coworker, well, that's performance feedback.
    - faangguyindia 6d ago
      Highly likely, your trust is just based on your feelings, and they don't do much for the product quality.
    - scrollaway 6d ago
      Your coworkers are if anything less reliably deterministic than an AI. And the human version of a prompt injection is called social engineering and it’s far more of an issue than actual prompt injection and has been for an extremely long time.
      - atomicnumber3 6d ago
optimiz3 6d ago
I'd argue the trust was always relative. Witness what happens when you are hired to untangle the buggy mess that was left by the previous "senior" engineer. Sure there was trust when they were in charge, but based on the code they wrote, they didn't actually understand anything they were doing.
rkagerer 6d ago
You trust that those layers work Unlike AI, those layers were traditionally compiled using deterministic tools. And generally came with a suite of other tools to trace and debug them. I don't trust AI in the same way at all.
glouwbug 6d ago
Then put your money where the mouth is and start committing prompts and having your LLM “compile” them with each production release.
SpicyLemonZest 6d ago
I don't think that's the problem the source article is describing. I've seen a number of instances now where people produce work product they don't understand at any level. "Hey Claude, design this feature and send it out for review, thx!" will often seem to work if you don't know any better.
vrighter 6d ago
because we could trust the layer below. Now we can't
dc396 6d ago
Mostly agreed. In a previous life, I used to write code, then got Peter Principled upwards to the point where I "trusted" the engineers under me to do their job. Did I understand all the code my team wrote? Of course not. Even if I wanted to review all the code written by 20+ developers to the point where I could stay I understood it, there simply wasn't enough hours in the day to do so. I figure what is happening is (human) software engineering as a discipline, at least on large scale projects, is rapidly and irreversibly mutating to software project management. But instead of managing teams of people, you're managing teams of agents. Where I disagree with your comment is in the assertion that everyone "can move one level up the system hierarchy." Some folks can't, so they'll stagnate. Welcome to the Peter Principle World.
- bluefirebrand 6d ago
  > Even if I wanted to review all the code written by 20+ developers to the point where I could stay I understood it, there simply wasn't enough hours in the day to do so. The difference of course being that you had 20+ developers who did understand it. With AI generated code no one has that understanding. The AI sure doesn't > Some folks can't, so they'll stagnate. And we have no answer for those people. Just "fuck you the world changed and you're being left behind"? Where exactly does that lead?
- imtringued 5d ago
  This is illogical. I see this everywhere but it is wrong. If you're developing an agentiv system to build software for you in an automated way using a fleet of agents, you're still an engineer of a technical system even if that system is operating on somewhat fuzzy logic. Management involves complete delegation and instead the task is to find a capable person for the task and then mostly let them just give feedback so you know whether you hired the right person or whether you need someone else. When you're working with a fleet of agents, you're not assembling a team of heterogeneous workers with different skills, you're hiring the same guy over and over again, hoping that he can do any job and yy same guy I also mean that the guy does not learn anything new over time. Meaning traditional management skills count for nothing here.

coldtea 6d ago

>This has only ever been true in a relative sense. Until recently, one was expected to understand the software-level code they wrote. But not the machine code, or the computer internals at a deep level. Just because something wasn't required at all levels or 100% it doesn't mean it wasn't required or it wasn't important though. I trusted that the machine code worked because it was a mostly mechanical transformation, with little to zero non-determinism involved, and the design decisions for the high level code were still mine and understood. Not the same with an LLM, even when see just as an added level of abstraction. Given some LLM produced code, I also trust the compiler will produce working machine code matching it. Doesn't mean I trust the LLM code in the same way.

meander_water 6d ago

You trust machine code because it was produced by a deterministic system that was written and tested thoroughly by other engineers. AI systems are not the same. You can't guarantee deterministic output. Also, code is still the best language we have to convey our intentions as engineers into function. Natural language is too imprecise and ambiguous. Understanding the code allows us to understand the intention behind the code and identify bugs, plan and architect better. Inspecting prompts gives you a blinkered view of the system meaning you're more likely to make incorrect assumptions leading to serious bugs.

Terr_ 6d ago
Right, it's a chain of trust between the layers. The link I choose next to me helps determine the quality of successive links, whether I inspect them or not. When I buy a wooden shack from a carpenter, I'd never expect them to core-sample every plank to ensure it's good wood. That's because a good carpenter habitually buys wood from a good supplier, and will have the experience to detect other suspicious indicators. Similarly, my coworker might not have verified the machine code, but I can be pretty confident they didn't vibe-install a compiler by Ken Thompson that puts backdoors into the binary.
faangguyindia 6d ago
Are your teammates deterministic output machines?
- atomicnumber3 6d ago
  In a sense yes. I trust that my coworkers will be accountable for their outcomes and accomplish them with the high quality bar I know they intrinsically hold themselves to. And that they understand our shared goals and if they don't, they will work to become aligned. And if I can't say that for a coworker, well, that's performance feedback.
  - faangguyindia 6d ago
    Highly likely, your trust is just based on your feelings, and they don't do much for the product quality.
  - scrollaway 6d ago
    Your coworkers are if anything less reliably deterministic than an AI. And the human version of a prompt injection is called social engineering and it’s far more of an issue than actual prompt injection and has been for an extremely long time.
    - atomicnumber3 6d ago

optimiz3 6d ago

I'd argue the trust was always relative. Witness what happens when you are hired to untangle the buggy mess that was left by the previous "senior" engineer. Sure there was trust when they were in charge, but based on the code they wrote, they didn't actually understand anything they were doing.

rkagerer 6d ago

You trust that those layers work Unlike AI, those layers were traditionally compiled using deterministic tools. And generally came with a suite of other tools to trace and debug them. I don't trust AI in the same way at all.

glouwbug 6d ago

Then put your money where the mouth is and start committing prompts and having your LLM “compile” them with each production release.

SpicyLemonZest 6d ago

I don't think that's the problem the source article is describing. I've seen a number of instances now where people produce work product they don't understand at any level. "Hey Claude, design this feature and send it out for review, thx!" will often seem to work if you don't know any better.

vrighter 6d ago

because we could trust the layer below. Now we can't

dc396 6d ago

Mostly agreed. In a previous life, I used to write code, then got Peter Principled upwards to the point where I "trusted" the engineers under me to do their job. Did I understand all the code my team wrote? Of course not. Even if I wanted to review all the code written by 20+ developers to the point where I could stay I understood it, there simply wasn't enough hours in the day to do so. I figure what is happening is (human) software engineering as a discipline, at least on large scale projects, is rapidly and irreversibly mutating to software project management. But instead of managing teams of people, you're managing teams of agents. Where I disagree with your comment is in the assertion that everyone "can move one level up the system hierarchy." Some folks can't, so they'll stagnate. Welcome to the Peter Principle World.

bluefirebrand 6d ago
> Even if I wanted to review all the code written by 20+ developers to the point where I could stay I understood it, there simply wasn't enough hours in the day to do so. The difference of course being that you had 20+ developers who did understand it. With AI generated code no one has that understanding. The AI sure doesn't > Some folks can't, so they'll stagnate. And we have no answer for those people. Just "fuck you the world changed and you're being left behind"? Where exactly does that lead?
imtringued 5d ago
This is illogical. I see this everywhere but it is wrong. If you're developing an agentiv system to build software for you in an automated way using a fleet of agents, you're still an engineer of a technical system even if that system is operating on somewhat fuzzy logic. Management involves complete delegation and instead the task is to find a capable person for the task and then mostly let them just give feedback so you know whether you hired the right person or whether you need someone else. When you're working with a fleet of agents, you're not assembling a team of heterogeneous workers with different skills, you're hiring the same guy over and over again, hoping that he can do any job and yy same guy I also mean that the guy does not learn anything new over time. Meaning traditional management skills count for nothing here.