6 ms·

I will just point out the benefit is not as obvious as you think. Developers have consistently overestimated LLM productivity gains, which still seems true for

I will just point out the benefit is not as obvious as you think. Developers have consistently overestimated LLM productivity gains, which still seems true for agentic AI: https://metr.org/blog/2026-05-11-ai-usage-survey/ https://metr.org/blog/2026-05-11-ai-usage-survey/ It is particularly striking how similar the results are to LLMs before agents. Along with the total absence of long-term data, I think the benefit can be (weakly) denied. Maybe not in the employmemt marketplace, but certainly for myself.

pydry 2mo ago
The productivity depends upon the requirements. If slop is fine (and sometimes it is), the benefits are undeniable. If the dev was the kind that would have produced slop anyway - again, undeniable boost. If the quality needs to be high I think it actually can slow you down, though.
- someoneiam 2mo ago
  Agreed, but also to expand, if the dev is mediocre and ai-assisted coding is a skill, the productivity gains are 10x more mediocre code. Since code is a liability, this is not desirable. Hence, mediocre devs being more productive is an underestimated problem of the age of ai-assisted coding. I see this every day. The result is a whole bunch of dysfunctional systems unnecessarily dislodging perfectly acceptable processes.
  - ermantrout 2mo ago
    "Code is a liability" is the part that usually gets hand-waved, and there's a number on it now. Veracode's 2025 GenAI report ran 80 tasks across 100+ models: when a task could be solved securely or insecurely, the models picked the insecure path 45% of the time, at roughly 2.7x the vulnerability density of human-written code (Java was worst, at a 72% failure rate). So the mediocre-dev case may be worse than "10x more mediocre code." It's more code that also skews insecure by default, and that cost shows up downstream in review and incidents, not at the PR. Throughput goes up, and so does the liability per line.
qsort 2mo ago
> I will just point out the benefit is not as obvious as you think. Developers have consistently overestimated LLM I think there are two different claims here: - developers overestimate productivity gains, which is a solid finding in many of these studies. Skepticism of extremely large productivity gains is warranted and I flatly disbelieve "10x uplift" claims. - LLMs give no productivity uplift at all, which is much harder to defend. A repeat of the famous METR RCT study did find evidence of improved productivity, and this seems to align with the experience of many experts I trust.
- Diogenesian 2mo ago
  Specifically my claim is "the relatively minor productivity uplift I would personally get out of agentic development is offset by the high cost, along with unresolved questions about long-term code maintainability, so I am not convinced that it is actually beneficial." IMO the bigger problem is that ~1.5x individual dev productivity uplift seems to translate into 1.05x uplift across the team. People have been waaaaayyyyy too overconfident about this stuff.
  - breadzeppelin__ 2mo ago
    I am both a career developer and experienced team manager. from first hand experience the 1.5x im getting from AI is not flowing down to my team / org because why would i output 50% more when the pay environment and leadership are already underwhelming. That additional 50% productivity goes completely to side projects built on my second computer between 9-5 tasks
    - Diogenesian 2mo ago
      Actually, the METR report speculates that some of the overreported productivity uplift comes from grabbing unnecessary low-hanging fruit, things like "oh I'll make a web dashboard to keep track of this stuff // wow that would have taken all day without Claude!" But in the olden days they would have just used a notepad. Yet psychologically they built a real thing and saved a lot of time.
deaton 2mo ago
I think the real disaster is that once you let the LLM work on a project for a bit, you start to lose understanding of what exactly is even happening under the hood in the project. You can take steps to mitigate this, but agents don't exactly encourage the behavior required to maintain a good understanding of what's going on.
- westurner 2mo ago
  Like becoming a manager? When a person becomes a manager, they do or do not have enough time and expertise to review all of the code that they trust the team to produce. Managers usually get into automated testing; unit tests, integration tests, acceptance tests, and maybe also BDD syntax Managers and developers are responsible for setting a test coverage threshold for merge approval. If there is 100% branch coverage test coverage for a codebase, what would coverage-guided fuzzing or property testing find? If there is 100% branch coverage test coverage for a codebase, what is the value of spending resources on formal verification? How does the value of LLM-produced 100% branch coverage compare to no-LLM 100% branch coverage?
  - tablarasa 2mo ago
    > How does the value of LLM-produced 100% branch coverage compare to no-LLM 100% branch coverage? This is such a salient question. Sometimes (definitely not always) the test suites produced by LLMs are so trivial it's scary. Coverage can be an illusion for sure.
    - coldtea 2mo ago
      I have been seeing such trivial shit getting submitted and deleting them from the test suites, but it's disheartening to see that the devs didn't have the better taste to see those LLM "tests" are so trivial they border on noop before submitting them...
      - westurner 2mo ago
        What new or existing metric would better assess test coverage than line coverage and branch coverage? I wrote a tool called tert - I guess it's called an agent harness now - to run various test runners and log test output and coverage output to disk. FWIU stripping spaces from JSON does save tokens. It seems like feeding coverage lines-missing maps into the prompt results in better output, better LLM-authored tests. "Refactor these tests for maintainability and coverage. Use fixtures, mocks, and parametrization"

pydry 2mo ago

The productivity depends upon the requirements. If slop is fine (and sometimes it is), the benefits are undeniable. If the dev was the kind that would have produced slop anyway - again, undeniable boost. If the quality needs to be high I think it actually can slow you down, though.

someoneiam 2mo ago
Agreed, but also to expand, if the dev is mediocre and ai-assisted coding is a skill, the productivity gains are 10x more mediocre code. Since code is a liability, this is not desirable. Hence, mediocre devs being more productive is an underestimated problem of the age of ai-assisted coding. I see this every day. The result is a whole bunch of dysfunctional systems unnecessarily dislodging perfectly acceptable processes.
- ermantrout 2mo ago
  "Code is a liability" is the part that usually gets hand-waved, and there's a number on it now. Veracode's 2025 GenAI report ran 80 tasks across 100+ models: when a task could be solved securely or insecurely, the models picked the insecure path 45% of the time, at roughly 2.7x the vulnerability density of human-written code (Java was worst, at a 72% failure rate). So the mediocre-dev case may be worse than "10x more mediocre code." It's more code that also skews insecure by default, and that cost shows up downstream in review and incidents, not at the PR. Throughput goes up, and so does the liability per line.

qsort 2mo ago

> I will just point out the benefit is not as obvious as you think. Developers have consistently overestimated LLM I think there are two different claims here: - developers overestimate productivity gains, which is a solid finding in many of these studies. Skepticism of extremely large productivity gains is warranted and I flatly disbelieve "10x uplift" claims. - LLMs give no productivity uplift at all, which is much harder to defend. A repeat of the famous METR RCT study did find evidence of improved productivity, and this seems to align with the experience of many experts I trust.

Diogenesian 2mo ago
Specifically my claim is "the relatively minor productivity uplift I would personally get out of agentic development is offset by the high cost, along with unresolved questions about long-term code maintainability, so I am not convinced that it is actually beneficial." IMO the bigger problem is that ~1.5x individual dev productivity uplift seems to translate into 1.05x uplift across the team. People have been waaaaayyyyy too overconfident about this stuff.
- breadzeppelin__ 2mo ago
  I am both a career developer and experienced team manager. from first hand experience the 1.5x im getting from AI is not flowing down to my team / org because why would i output 50% more when the pay environment and leadership are already underwhelming. That additional 50% productivity goes completely to side projects built on my second computer between 9-5 tasks
  - Diogenesian 2mo ago
    Actually, the METR report speculates that some of the overreported productivity uplift comes from grabbing unnecessary low-hanging fruit, things like "oh I'll make a web dashboard to keep track of this stuff // wow that would have taken all day without Claude!" But in the olden days they would have just used a notepad. Yet psychologically they built a real thing and saved a lot of time.

deaton 2mo ago

I think the real disaster is that once you let the LLM work on a project for a bit, you start to lose understanding of what exactly is even happening under the hood in the project. You can take steps to mitigate this, but agents don't exactly encourage the behavior required to maintain a good understanding of what's going on.

westurner 2mo ago
Like becoming a manager? When a person becomes a manager, they do or do not have enough time and expertise to review all of the code that they trust the team to produce. Managers usually get into automated testing; unit tests, integration tests, acceptance tests, and maybe also BDD syntax Managers and developers are responsible for setting a test coverage threshold for merge approval. If there is 100% branch coverage test coverage for a codebase, what would coverage-guided fuzzing or property testing find? If there is 100% branch coverage test coverage for a codebase, what is the value of spending resources on formal verification? How does the value of LLM-produced 100% branch coverage compare to no-LLM 100% branch coverage?
- tablarasa 2mo ago
  > How does the value of LLM-produced 100% branch coverage compare to no-LLM 100% branch coverage? This is such a salient question. Sometimes (definitely not always) the test suites produced by LLMs are so trivial it's scary. Coverage can be an illusion for sure.
  - coldtea 2mo ago
    I have been seeing such trivial shit getting submitted and deleting them from the test suites, but it's disheartening to see that the devs didn't have the better taste to see those LLM "tests" are so trivial they border on noop before submitting them...
    - westurner 2mo ago
      What new or existing metric would better assess test coverage than line coverage and branch coverage? I wrote a tool called tert - I guess it's called an agent harness now - to run various test runners and log test output and coverage output to disk. FWIU stripping spaces from JSON does save tokens. It seems like feeding coverage lines-missing maps into the prompt results in better output, better LLM-authored tests. "Refactor these tests for maintainability and coverage. Use fixtures, mocks, and parametrization"