6 ms·

That study is garbo and I suspect you didn't even read the abstract. Am I right?

by bird0861 10mo ago

gravypod 10mo ago
I've heard this mentioned a few times. Here is a summarized version of the abstract: > ... We conduct a randomized controlled trial (RCT) > ... AI tools ... affect the productivity of experienced > open-source developers. 16 developers with moderate AI > experience complete 246 tasks in mature projects on which they > have an average of 5 years of prior experience. Each task is > randomly assigned to allow or disallow usage of early-2025 AI > tools. ... developers primarily use Cursor Pro ... and > Claude 3.5/3.7 Sonnet. Before starting tasks, developers forecast that allowing > AI will reduce completion time by 24%. After completing the > study, developers estimate that allowing AI reduced completion time by 20%. > Surprisingly, we find that allowing AI actually increases > completion time by 19%—AI tooling slowed developers down. This > slowdown also contradicts predictions from experts in economics > (39% shorter) and ML (38% shorter). To understand this result, > we collect and evaluate evidence for 21 properties of our setting > that a priori could contribute to the observed slowdown effect—for > example, the size and quality standards of projects, or prior > developer experience with AI tooling. Although the influence of > experimental artifacts cannot be entirely ruled out, the robustness > of the slowdown effect across our analyses suggests it is unlikely > to primarily be a function of our experimental design. So what we can gather: 1. 16 people were randomly given tasks to do 2. They knew the codebase they worked on pretty well 3. They said AI would help them work 24% faster (before starting tasks) 4. They said AI made them ~20% faster (after completion of tasks) 5. ML Experts claim that they think programmers will be ~38% faster 6. Economists say ~39% faster. 7. We measured that people were actually 19% slower This seems to be done on Cursor, with big models, on codebases people know. There are definitely problems with industry-wide statements like this but I feel like the biggest area AI tools help me is if I'm working on something I know nothing about. For example: I am really bad at web development so CSS / HTML is easier to edit through prompts. I don't have trouble believing that I would be slower trying to make an edit to code that I already know how to make. Maybe they would see the speedups by allowing the engineer to select when to use the AI assistance and when not to.
- saturatedfat 10mo ago
  it doesnt control for skill using models/experience using models. this looks VERY different at hour 1000 and hour 5000 than hour 100.
  - brumar 10mo ago
    Lazy from me to not check if I remember well or not, but the dev that got productivity gains was a regular user of cursor.

That study is garbo and I suspect you didn't even read the abstract. Am I right?

I've heard this mentioned a few times. Here is a summarized version of the abstract: > ... We conduct a randomized controlled trial (RCT) > ... AI tools ... affect the productivity of experienced > open-source developers. 16 developers with moderate AI > experience complete 246 tasks in mature projects on which they > have an average of 5 years of prior experience. Each task is > randomly assigned to allow or disallow usage of early-2025 AI > tools. ... developers primarily use Cursor Pro ... and > Claude 3.5/3.7 Sonnet. Before starting tasks, developers forecast that allowing > AI will reduce completion time by 24%. After completing the > study, developers estimate that allowing AI reduced completion time by 20%. > Surprisingly, we find that allowing AI actually increases > completion time by 19%—AI tooling slowed developers down. This > slowdown also contradicts predictions from experts in economics > (39% shorter) and ML (38% shorter). To understand this result, > we collect and evaluate evidence for 21 properties of our setting > that a priori could contribute to the observed slowdown effect—for > example, the size and quality standards of projects, or prior > developer experience with AI tooling. Although the influence of > experimental artifacts cannot be entirely ruled out, the robustness > of the slowdown effect across our analyses suggests it is unlikely > to primarily be a function of our experimental design. So what we can gather: 1. 16 people were randomly given tasks to do 2. They knew the codebase they worked on pretty well 3. They said AI would help them work 24% faster (before starting tasks) 4. They said AI made them ~20% faster (after completion of tasks) 5. ML Experts claim that they think programmers will be ~38% faster 6. Economists say ~39% faster. 7. We measured that people were actually 19% slower This seems to be done on Cursor, with big models, on codebases people know. There are definitely problems with industry-wide statements like this but I feel like the biggest area AI tools help me is if I'm working on something I know nothing about. For example: I am really bad at web development so CSS / HTML is easier to edit through prompts. I don't have trouble believing that I would be slower trying to make an edit to code that I already know how to make. Maybe they would see the speedups by allowing the engineer to select when to use the AI assistance and when not to.

saturatedfat 10mo ago
it doesnt control for skill using models/experience using models. this looks VERY different at hour 1000 and hour 5000 than hour 100.
- brumar 10mo ago
  Lazy from me to not check if I remember well or not, but the dev that got productivity gains was a regular user of cursor.