21 ms·

AI slows down open source developers. Peter Naur can teach us why

363 points by jwhiles 1y ago

mkagenius 1y ago
AI tends to slow us down because we don't really know what it's good at. Can it write a proper Nginx config? I don't know—let's try. And then we end up wasting 30 minutes on it. Fully autonomous coding tools like v0, a0, or Aider work well as long as the context is small. But once the context grows—usually due to mistakes made in earlier steps—they just can’t keep up. There's no real benefit of "try again" loop yet. For now, I think simple VSCode extensions are the most useful. You get focused assistance on small files or snippets you’re working on, and that’s usually all you need.
- ethan_smith 1y ago
  The context switching cost between coding and AI interaction is substantial and rarely measured in these studies. Each prompt/review cycle breaks flow state, which is particularly damaging for complex programming tasks where deep concentration yields the greatest productivity.
  - bluefirebrand 1y ago
    This has been my experience too Ever since my company made switching to Cursor mandatory, I have not been able to hit any kind of flow. I know my own productivity has plummeted and I suspect many others are as well, but no one is saying anything I have spoken up once or twice and only been smacked down for my troubles, so I am not surprised everyone else is clammed up
doc_manhat 1y ago
I directionally disagree with this: ``` It's common for engineers to end up working on projects which they don't have an accurate mental model of. Projects built by people who have long since left the company for pastures new. It's equally common for developers to work in environments where little value is placed on understanding systems, but a lot of value is placed on quickly delivering changes that mostly work. In this context, I think that AI tools have more of an advantage. They can ingest the unfamiliar codebase faster than any human can, and can often generate changes that will essentially work. ``` Reason: you cannot evaluate the work accurately if you have no mental model. If there's a bug given the systems unwritten assumptions you may not catch it. Having said that it also depends on how important it is to be writing bug free code in the given domain I guess. I like AI particularly for green field stuff and one off scripts as it let's you go faster here. Basically you build up the mental model as you're coding with the AI. Not sure about whether this breaks down at a certain codebase size though.
- horsawlarway 1y ago
  Just anecdotally - I think your reason for disagreeing is a valid statement, but not a valid counterpoint to the argument being made. So > Reason: you cannot evaluate the work accurately if you have no mental model. If there's a bug given the systems unwritten assumptions you may not catch it. This is completely correct. It's a very fair statement. The problem is that a developer coming into a large legacy project is in this spot regardless of the existence of AI. I've found that asking AI tools to generate a changeset in this case is actually a pretty solid way of starting to learn the mental model. I want to see where it tries to make changes, what files it wants to touch, what libraries and patterns it uses, etc. It's a poor man's proxy for having a subject matter expert in the code give you pointers. But it doesn't take anyone else's time, and as long as you're not just trying to dump output into a PR can actually be a pretty good resource. The key is not letting it dump out a lot of code, in favor of directional signaling. ex: Prompts like "Which files should I edit to implement a feature which does [detailed description of feature]?" Or "Where is [specific functionality] implemented in this codebase?" Have been real timesavers for me. The actual code generation has probably been a net time loss.
  - Roscius 1y ago
    > I've found that asking AI tools to generate a changeset in this case is actually a pretty solid way of starting to learn the mental model. This. Leveraging the AI to start to develop the mental model is an advantage. But, using the AI is a non-trivial skill set that needs to be learned. Skepticism of what it's saying is important. AI can be really useful just like a 747 can be useful, but you don't want someone picked off the street at random flying it.
    - bluefirebrand 1y ago
      > This. Leveraging the AI to start to develop the mental model is an advantage Is there any evidence that AI helps you build the mental model of an unfamiliar codebase more quickly? In my experience trying to use AI for this it often leads me into the weeds
  - 1y ago
gjsman-1000 1y ago
What I thought was fascinating, and should be a warning sign to everyone here: Before beginning the study, the average developer expected about a 20% productivity boost. After ending the study, the average developer (potentially: you) believed they actually were 20% more productive. In reality, they were 0% more productive at best, and 40% less productive at worst. Think about what it would be like to be that developer; off by 60% about your own output. If you can't even gauge your own output without being 40% off on average, 60% off at worst; be cautious about strong opinions on anything in life. Especially politically. Edit 1: Also consider, quite terrifyingly, if said developers were in an online group, together, like... here. The one developer who said she thought it made everyone slower (the truth in this particular case), would be unanimously considered an idiot, downvoted to the full -4, even with the benefit of hindsight. Edit 2: I suppose this goes to show, that even on Hacker News, where there are relatively high-IQ and self-aware individuals present... 95% of the crowd can still possibly be wildly delusional. Stick to your gut, regardless of the crowd, and regardless of who is in it.
- pphysch 1y ago
  Given how deadlines/timelines tend to (not) work in SWE, this is not surprising.
  - gjsman-1000 1y ago
    Perhaps; but this is a developer's own output with an AI tool, compared against their own historical output when they didn't use it. Apparently, the average developer (read: quite possibly most people here) can't even hit the broadside of a barn in estimating their own productivity.
    - sureglymop 1y ago
      That doesn't surprise me at all. Isn't software engineering in essence about being constantly confronted with new problems to solve and having to come up with a sufficient one on the fly? It seems very hard to estimate this, even if you know yourself well.
xyst 1y ago
Not surprising. Use of LLM has only been helpful in initial exploration of unknown code bases or languages for me. Using it beyond that is just more work. First parse the broken response, remove any useless junk, have it reprocess with updated query. It’s a nice tool to have (just as search engines gave us easy access to multiple sources/forums), but its limitations are well known. Trying to use it 100% as intended is a massive waste of time and resources (energy use…)
nico 1y ago
> They are experienced open source developers, working on their own projects I just started working on a 3-month old codebase written by someone else, in a framework and architecture I had never used before Within a couple hours, with the help of Claude Code, I had already created a really nice system to replicate data from staging to local development. Something I had built before in other projects, and I new that manually it would take me a full day or two, especially without experience in the architecture That immediately sped up my development even more, as now I had better data to test things locally Then a couple hours later, I had already pushed my first PR. All code following the proper coding style and practices of the existing project and the framework. That PR, would have taken me at least a couple of days and up to 2 weeks to fully manually write out and test So sure, AI won’t speed everyone or everything up. But at least in this one case, it gave me a huge boost As I keep going, I expect things to slow down a bit, as the complexity of the project grows. However, it’s also given me the chance to get an amazing jumpstart
- kevmo314 1y ago
  You've missed the point of the article, which in fact agrees with your anecdote. > It's equally common for developers to work in environments where little value is placed on understanding systems, but a lot of value is placed on quickly delivering changes that mostly work. In this context, I think that AI tools have more of an advantage. They can ingest the unfamiliar codebase faster than any human can, and can often generate changes that will essentially work.
  - moogleii 1y ago
    That would be an aside, or a comment, not the point of the article.
  - antonvs 1y ago
    > You've missed the point of the article Sadly clickbait headlines like the OP, "AI slows down open source developers," spread this misinformation, ensuring that a majority of people will have the same misapprehension.
rosspackard 1y ago
One mediocre paper/study (it should not even be called that with all the bias and sample size issues) and now we have to put up with stories re-hashing and dissecting it. I really hope these don't get upvoted more in the future. 16 devs. And they weren't allowed to pick which tasks they used the AI on. Ridiculous. Also using it on "old and >1 million line" codebases and then extrapolating that to software engineering in general. Writers like this then theorize why AI isn't helpful, then those "theories" get repeated until it feels less like a theory and more like a fact and it all proliferates into an echo chamber of AI isn't a useful tool. There have been too many anecdotes and my own personal experience to ignore that it isn't useful. It is a tool and you have to learn it to be successful with it.
- sumeno 1y ago
  > And they weren't allowed to pick which tasks they used the AI on. They were allowed to pick whether or not to use AI on a subset of tasks. They weren't forced to use AI on tasks that don't make sense for AI
  - rosspackard 1y ago
    Half the tasks they were not allowed to use AI.
    - sumeno 1y ago
      Yes, and the other half they had the option to use AI. That's why I said they were allowed to pick whether or not to use AI on a subset of tasks. On the other subset they were not allowed to use AI.
  - throwaway284927 1y ago
    That is not true, usage of AI was decided randomly. From the paper: "To directly measure the impact of AI tools on developer productivity, we conduct a randomized controlled trial by having 16 developers complete 246 tasks (2.0 hours on average) on well-known open-source repositories (23,000 stars on average) they regularly contribute to. Each task is randomly assigned to allow or disallow AI usage, and we measure how long it takes developers to complete tasks in each condition."
uludag 1y ago
Great article and I was having very similar thoughts with regards to this productivity study and the "Programming as Theory Building" paper. I'm starting to be convinced that if you are the original author of a program and still have the program's context in the head, you are the asymptote to which any and all AI systems will approach but never surpass: maybe not in terms of raw coding speed, but in terms of understanding the program, its vision of development, its deficiencies and hacks, its context, its users and what they want, the broader culture the program exists in, etc. I really like how the author then brought up the point that for most daily work we don't have the theory built, even a small fraction of it, and that this may or may not change the equation.
- conartist6 1y ago
  Thanks, <3
cratermoon 1y ago
dissected https://www.fightforthehuman.com/are-developers-slowed-down-by-ai-evaluating-an-rct-and-what-it-tells-us-about-developer-productivity/ https://www.fightforthehuman.com/are-developers-slowed-down-...
omnicognate 1y ago
All these studies that show "AI makes developers x% more/less productive" are predicated on the idea that developer "productivity" can be usefully captured in a single objectively measurable number. Just one problem with that...
- narush 1y ago
  Thanks for the feedback! I strongly agree this is not the only measure of developer productivity -- but it's certainly one of them. I think this measure as speaks very directly to how _many_ developers (myself included) understand the impact of AI tools on their own work currently (e.g. just speeding up implementation speed). (The SPACE [1] framework is a pretty overview of considerations here; I agree with a lot of it, although I'll note that METR [2] has different motivations for studying developer productivity than Microsoft does.) [1] https://dl.acm.org/doi/10.1145/3454122.3454124 https://dl.acm.org/doi/10.1145/3454122.3454124 [2] https://metr.org/about https://metr.org/about
- charcircuit 1y ago
  As long as the true productivity is correlated with that number it should be fine.
yomismoaqui 1y ago
Someone on X said that these agentic AI tools (Claude Code, Amp, Gemini Cli) are to programming like the table saw was to hand-made woodworking. It can make some things faster and better than a human with a saw, but you have to learn how to use them right (or you will loose some fingers). I personally find that agentic AI tools make me be more ambitious in my projects, I can tackle some things I didn't tthougth about doing before. And I also delegate work that I don't like to them because they are going to do it better and quicker than me. So my mind is free to think on the real problems like architecture, the technical debt balance of my code... Problem is that there is the temptation of letting the AI agent do everything and just commit the result without understanding YOUR code (yes, it was generated by an AI but if you sign the commit YOU are responsible for that code). So as with any tool try to take the time to understand how to better use it and see if it works for you.
- bgwalter 1y ago
  "You are using it wrong!" This is insulting to all pre-2023 open source developers, who produced the entire stack that the "AI" robber barons use in their companies. It is even more insulting because no actual software of value has been demonstrably produced using "AI".
  - yomismoaqui 1y ago
    > It is even more insulting because no actual software of value has been demonstrably produced using "AI". Claude Code and Amp (equivalent from Sourcegraph) are created by humans using these same tools to add new features and fix bugs. Having used both tools for some weeks I can tell you that they provide a great value to me, enough that I see paying $100 monthly as a bargain related to that value. Edit: typo
    - jdiff 1y ago
      GP is pointing out the distinct lack of AI driven development in the wild. At this point, agents should be visibly maintaining at least a few popular codebases across this world wide web. The fact that there aren't raises some eyebrows for the claims that are regularly made by proponents. Not just the breathless proponents, either. Even taking claims very conservatively, FOSS maintainer burnout should be a thing of the past, but the only noted interaction with AI seems to be amplifying it.
d00mB0t 1y ago
Blasphemy! How dare you say our Emperor has no clothes! AI is becoming a cult and I'm not here for it.
gr8beehive 1y ago
Mirror neurons got people drinking the same stupid kool aid without realizing it.
narush 1y ago
Hey HN -- study author here! (See previous thread on the paper here [1].) I think this blog post is an interesting take on one specific factor that is likely contributing to slowdown. We discuss this in the paper [2] in the section "Implicit repository context (C.1.5)" -- check it out if you want to see some developer quotes about this factor. > This is why AI coding tools, as they exist today, will generally slow someone down if they know what they are doing, and are working on a project that they understand. I made this point in the other thread discussing the study, but in general, these results being surprising makes it easy to read the paper, find one factor that resonates, and conclude "ah, this one factor probably just explains slowdown." My guess: there is no one factor -- there's a bunch of factors that contribute to this result -- at least 5 seem likely, and at least 9 we can't rule out (see the full factors table on page 11). > If there are no takers then I might try experimenting on myself. This sounds super cool! I'd be very excited to see how you set this up + how it turns out... please do shoot me an email (in the paper) if you do this! > AI slows down open source developers. Peter Naur can teach us why Nit: I appreciate how hard it is to write short titles summarizing the paper (the graph title is the best I was able to do after a lot of trying) -- but I might have written this "Early-2025 AI slows down experienced open-source developers. Peter Naur can give us more context about one specific factor." It's admittedly less of a catchy-title, but I think getting the qualifications right are really important! Thanks again for the sweet write-up! I'll hang around in the comments today as well. [1] https://news.ycombinator.com/item?id=44522772 https://news.ycombinator.com/item?id=44522772 [2] https://metr.org/Early_2025_AI_Experienced_OS_Devs_Study.pdf https://metr.org/Early_2025_AI_Experienced_OS_Devs_Study.pdf
- antonvs 1y ago
  > Early-2025 AI slows down experienced open-source developers. Even that's too general, because it'll depend on what the task is. It's not as if open source developers in general never work on tasks where AI could save time.
  - narush 1y ago
    We call this over-generalization out specifically in the "We do not provide evidence that:" table in the blog post and paper - I agree there are tasks these developers are likely sped up on with early-2025 tools.
neuroelectron 1y ago
Good article and it makes sense. I wish I had sometime in my career worked on a codebase that was possible to be understood without 10 years of experience. Instead most of my development time was spent tracing execution paths through tangles of abstractions in nested objects in 10M LOC legacy codebases. My buddy who introduced me to the job is still doing it today and now uses AI and this has given him the free time to start working on his own side projects. So there's certain types if jobs where AI will certainly speed up your development.
bunderbunder 1y ago
> It's a really fabulous study... Ehhhh... not so much. It had serious design flaws in both the protocol and the analysis. This blog post is a fairly approachable explanation of what's wrong with it: https://www.argmin.net/p/are-developers-finally-out-of-a-job https://www.argmin.net/p/are-developers-finally-out-of-a-job
- narush 1y ago
  Hey, thanks for linking this! I'm a study author, and I greatly appreciate that this author dug into the appendix and provided feedback so that other folks can read it as well. A few notes if it's helpful: 1. This post is primarily worried about ordering considerations -- I think this is a valid concern. We explicitly call this out in the paper [1] as a factor we can't rule out -- see "Bias from issue completion order (C.2.4)". We have no evidence this occurred, but we also don't have evidence it didn't. 2. "I mean, rather than boring us with these robustness checks, METR could just release a CSV with three columns (developer ID, task condition, time)." Seconded :) We're planning on open-sourcing pretty much this data (and some core analysis code) later this week here: https://github.com/METR/Measuring-Early-2025-AI-on-Exp-OSS-Devs https://github.com/METR/Measuring-Early-2025-AI-on-Exp-OSS-D... - star if you want to dig in when it comes out. 3. As I said in my comment on the post, the takeaway at the end of the post is that "What we can glean from this study is that even expert developers aren’t great at predicting how long tasks will take. And despite the new coding tools being incredibly useful, people are certainly far too optimistic about the dramatic gains in productivity they will bring." I think this is a reasonable takeaway from the study overall. As we say in the "We do not provide evidence that:" section of the paper (Page 17), we don't provide evidence across all developers (or even most developers) -- and ofc, this is just a point-in-time measurement that could totally be different by now (from tooling and model improvements in the past month alone). Thanks again for linking, and to the original author for their detailed review. It's greatly appreciated! [1] https://metr.org/Early_2025_AI_Experienced_OS_Devs_Study.pdf https://metr.org/Early_2025_AI_Experienced_OS_Devs_Study.pdf
  - bunderbunder 1y ago
    Thanks for the response, you make some very points. Sorry, I had missed your response on the original post. I don't know if it was there yet, or because for some reason their blog is configured to only show the first two comments by default. :/ Either way, my bad. I think my bias as someone who spends too much time looking at social science papers is that the protocol allows for spillover effects that, to me, imply that the results must be interpreted much more cautiously than a lot of people are doing. (And then on top of that I'm trying to be hyper-cautious and skeptical when I see a paper whose conclusions align with my biases on this topic.) Granted, that sort of thing is my complaint about basically every study on developer productivity when using LLMs that I've seen so far. So I appreciate how difficult this is to study in practice.
tomasz_fm 1y ago
Only one developer in this study had more than 50h of Cursor experience, including time spent using Cursor during the study. That one developer saw a 25% speed improvement. Everyone else was an absolute Cursor beginner with barely any Cursor experience. I don't find it surprising that using tools they're unfamiliar with slows software engineers down. I don't think this study can be used to reach any sort of conclusion on use of AI and development speed.
- Art9681 1y ago
  This is exactly my same take. Any tool an engineer is inexperienced with will slow them down. AI is no different.
  - bluefirebrand 1y ago
    This runs counter to the starry eyed promises of AI letting people with no experience accomplish things
    - TeMPOraL 1y ago
      That promise is true, though, and the two claims are not opposite. The devil is in details, specifically in what you mean by "people" and "accomplish things". If by "people" you mean "general public", and by "accomplish things" you mean solving some immediate problems, that may or may not involve authoring a script or even a small app - then yes, this is already happening, and is a big reason behind the AI hype as it is. If by "people" you mean "experienced software engineers", and by "accomplish things" you mean meaningful contributions to a large software product, measured by high internal code and process quality standards, then no - AI tools may not help with that directly, though chances are greater when you have enough experience with those tools to reliably give them right context and steer away from failure modes. Still, solving one-off problems != incremental improvements to a large system.
      - bluefirebrand 1y ago
        > If by "people" you mean "experienced software engineers", My post is a single sentence and I literally wrote "people with no experience"
whatever1 1y ago
They didn’t use the latest model that was released yesterday night. Follow my paid course to learn how to vibe code/s
methuselah_in 1y ago
Those of current generation students who have access to ai might become slow over time. Because when things are not readily available then they have to struggle and work harder in that process, at that time I thing human a lot of secondary things ! Now when everything is easily available especially knowledge without knowing how to struggle with basics. It will eventually make kids dumb. But can be opposite also. Eventually even I become slow even I keep on using chat gpt or gemini.
piker 1y ago
My main two attempts at using an “agentic” coding workflow were trying to incorporate an Outlook COM interface into my rust code base and to streamline an existing abstract windows API interaction to avoid copying memory a couple of times. Both wasted tremendous amounts of time and were ultimately abandoned leaving me only slightly more educated about windows development. They make great autocompletion engines but I just cannot see them being useful in my project otherwise.
- crinkly 1y ago
  This is typically what I see when I’ve seen it applied. And as always trying to hammer nails in with a banana. Rather than fit two generally disparate things together it’s probably better to just use VSTO and C# (hammer and nails) rather than some unholy combination no one else has tried or suffered through. When it goes wrong there’s more info to get you unstuck.
  - piker 1y ago
    To be fair though, unsafe rust (where the COM lives) is basically just C, so I totally expected it to be tractable in the same way it has been tractable for the last 20ish years? But it isn’t. Why is interacting with the OS’ API in a compiled language the wrong approach in 2025? Why must I use this managed Frankenstein’s monster of dotnet? I didn’t want to ship or expect a whole runtime for what should be a tiny convenience DLL. Insane
- charcircuit 1y ago
  I had the opposite experience. Gemini was able to work with COM and accomplish what I needed despite me never using COM before.
  - tonyedgecombe 1y ago
    I've done a lot of work with COM over the years and that is the last technology I would trust to an AI. It's very easy to write COM code that appears to work but contains subtle bugs.
joshmarlow 1y ago
I've gotten some pretty cool things working with LLMs doing most of the heavy lifting using the following approaches: * spec out project goals and relevant context in a README and spec out all components; have the AI build out each component and compose them. I understand the high-level but don't necessarily know all of the low-level details. This is particularly helpful when I'm not deeply familiar with some of the underlying technologies/libraries. * having an AI write tests for code that I've verified is working. As we all know, testing is tedious - so of course I want to automate it. And we written tests (for well written code) can be pretty easy to review.
antimora 1y ago
I'm one of the regular code reviewers for Burn (a deep learning framework in Rust). I recently had to close a PR because the submitter's bug fix was clearly written entirely by an AI agent. The "fix" simply muted an error instead of addressing the root cause. This is exactly what AI tends to do when it can't identify the actual problem. The code was unnecessarily verbose and even included tests for muting the error. Based on the person's profile, I suspect their motivation was just to get a commit on their record. This is becoming a troubling trend with AI tools.
- meindnoch 1y ago
  >a deep learning framework in Rust [...] This is becoming a troubling trend with AI tools. The serpent is devouring its own tail.
  - LoganDark 1y ago
    Deep learning can be incredibly cool and not just used for AI slop.
  - TeMPOraL 1y ago
    OTOH when they'll start getting good AI contributions, then... it'll be too late for us all.
- tomrod 1y ago
  As a side question: I work in AI, but mostly python and theory work. How can I best jump into Burn? Rust has been intriguing to me for a long time
- lvl155 1y ago
  This is a real problem that’s only going to get worse. With the major model providers basically keeping all the data themselves, I frankly don’t like this trend long term.
- dawnerd 1y ago
  That's what I love about LLMs. You can spot it doesn't know the answer, tell it that it's wrong and it'll go, "You're absolutely right. Let me actually fix it" It scares me how much code is being produced by people without enough experience to spot issues or people that just gave up caring. We're going to be in for wild ride when all the exploits start flowing.
hartator 1y ago
I am not super sure how to quickly writing benchmark scripts that are one-shot used slows anyone down, but okay.
lsy 1y ago
Typically debugging, e.g., a tricky race condition in an unfamiliar code base would require adding logging, refactoring library calls, inspecting existing logs, and even rewriting parts of your program to be more modular or understandable. This is part of the theory-building. When you have an AI that says "here is the race condition and here is the code change to make to fix it", that might be "faster" in the immediate sense, but it means you aren't understanding the program better or making it easier for anyone else to understand. There is also the question of whether this process is sustainable: does an AI-edited program eventually fall so far outside what is "normal" for a program that the AI becomes unable to model correct responses?
- sodapopcan 1y ago
  This is always my thought whenever I hear the "AI let me build a feature in a codebase I didn't know in a language I didn't know" (which is often, there is at one in these comments). Great, but what have you learned? This is fine for small contributions, I guess, but I don't hear a lot of stories of long-term maintenance. Unpopular opinion, though, I know.
  - threetonesun 1y ago
    I guess it's a question of how anyone learns. There's some value in typing code, I suppose, but with tab complete that's been gone for a long time. Letting AI write something and then reading it seems as good as copying and pasting from some other source.
    - sodapopcan 1y ago
      I'm not super qualified to answer as I haven't gone deep into AI at all. But from my limited observations I'd say yes and no. You generally aren't copy/pasting entire features, just snippets that you yourself have to string together in a sensible way. Of course there are lots of people who still do this and what's why I find most people in this industry infuriating to work with. It's all good when it's boilerplate, and that's actually my primary use of "AI"—it's essentially been a snippets replacement (and is quite good at that).
andix 1y ago
What I noticed: AI development constantly breaks my flow. It makes me more tired, and I work for shorter time periods on coding. It's a myth that you can code a whole day long. I usually do intervals of 1-3 hours for coding, with some breaks in between. Procrastination can even happen on work related things, like reading other project members code/changes for an hour. It has a benefit to some extent, but during this time I don't get my work done. Agentic AI works the best for me. Small refactoring tasks on a selected code snippet can be helpful, but isn't a huge time saver. The worst are AI code completions (first version Copilot style), they are much more noise then help.
- rightbyte 1y ago
  It would be interesting to record what one do in a day at the desk. Probably quite depressing to watch. Like, I think 1h would be streaching it for mature codebases.
  - andix 1y ago
    The 1h I'm talking about is not all the time I might spend reading on code. It's the time I might procrastinate on my tasks with reading unrelated code. Like doom scrolling on social media: Let's see what the fancy new guy got done this week. I need to feel better, I'm just going to look at the commits of the guy in the other team that always breaks production. Let's see how close he got to that recently, ...
afro88 1y ago
I said this when the linked paper was shared and got downvotes: it's based on early 2025 data. My point isn't that it should be completely up to date, but that how we need to consider it in that context. This is pre Claude 4, Claude Code. Pre Gemini 2.5 even. These models are such a big step up from what came previously. Just like we put a (2023) on articles here so they are considered in the right context, so too this paper should be. Blanket "AI tools slow sown development" statements with a "look this rigorous paper says so!" is ignoring a key variable: the rate of effectiveness improvement. If said paper evaluated with the current models, the picture would be different. Also in 3 months time. AI tools aren't a static thing that either works or don't indefinitely.
- tonyedgecombe 1y ago
  >This is pre Claude 4, Claude Code. Pre Gemini 2.5 even. The most interesting point from the article wasn't about how well the AI's worked, rather it was the gap between peoples perception and their actual results.
blake1 1y ago
I think a reasonable summary of the study referenced is that: "AI creates the perception of productivity enhancements far beyond the reality." Even within the study, there were some participants who saw mild improvements to productivity, but most had a significant drop in productivity. This thread is now full of people telling their story about huge productivity gains they made with AI, but none of the comments contend with the central insight of this study: that these productivity gains are illusions. AI is a product designed to make you value the product. In matters of personal value, perception is reality, no question. Anyone relying heavily on AI should really be worried that it is mostly a tool for warping their self-perception, one that creates dependency and a false sense of accomplishment. After all, it speaks a highly optimized stream of tokens at you, and you really have to wonder what the optimization goal was.
- BriggyDwiggs42 1y ago
  I’ve noticed that you can definitely use them to help you learn something, but that your understanding tends to be more abstract and LLM-like that way. You definitely want to mix it up when learning too.
  - daxfohl 1y ago
    I've also had bad results with hallucinations there. I was trying to learn more about multi-dimensional qubit algorithms, and spent a whole day learning a bunch of stuff that was fascinating but plain wrong. I only figured out it was wrong at the end of the day when I tried to do a simulation and the results weren't consistent. Early in the chat it substituted a `-1` for an `i`, and everything that followed was garbage. There were also some errors that I spotted real-time and got it to correct itself. But yeah, IDK, it presents itself so confidently and "knows" so much and is so easy to use, that it's hard not to try to use as a reference / teacher. But it's also quite dangerous if you're not confirming things; it can send you down incorrect paths and waste a ton of time. I haven't decided whether the cost is worth the benefit or not. Presumably they'll get better at this over time, so in the long run (probably no more than a year) it'll likely easily exceed the ROI breakeven point, but for now, you do have to remain vigilant.
Kim_Bruning 1y ago
I think different people use these tools differently. I've got mine set up to start in "rubber duck" mode, where I do rubber duck programming, before asking the AI to help me with certain tasks (if at all). Low impact utility scripts? The AI gets let off the leash. Critical core logic? I might do most of the work myself (though having a rubber duck can still be good!)
ringeryless 1y ago
not to mention the annoyance of AI assisted issues being opened, many times incorrectly due to hallucinations. these tickets hammer human teams with nonsense and suck resources away from real issues.
alganet 1y ago
This idea that some developers have some "mental model" and others not is an extraordinary claim, and I don't see extraordinary evidence. It sounds like a good thing, right? "Wow, mental model. I want that, I want to be good and have big brain", which encourages you to believe the bullshit. The truth is, this paper is irrelevant and a waste of time. It only serves the purpose of creating discussion around the subject. It's not science, it's a cupholder for marketing.
- imiric 1y ago
  You couldn't be more wrong. If you've ever programmed, or worked with programmers, that is not an extraordinary claim at all, but a widely accepted fact. A mental model of the software is what allows a programmer to intuitively know why the software is behaving a certain way, or what the most optimal design for a feature would be. In the vast majority of cases these intuitions are correct, and other programmers should pay attention to them. This ability is what separates those with a mental model and those without. On the other hand, LLMs are unable to do this, and are usually not used in ways that help build a mental model. At best, they can summarize the design of a system or answer questions about its behavior, which can be helpful, but a mental model is an abstract model of the software, not a textual summary of its design or behavior. Those neural pathways can only be activated by natural learning and manual programming.
  - alganet 1y ago
    > You couldn't be more wrong. Explanation missing. > If you've ever programmed, or worked with programmers, that is not an extraordinary claim at all. One step ahead of you. I already say this is engineered to encourage belief "I want to be good, big brain, and open source is good, I want to be good big brain". It's marketing. > A mental model of the software is what allows a programmer [yadda yadda] I'm not saying it doesn't exist, I'm saying the paper doesn't provide any relevant information regarding the phenomena. > Those neural pathways can only be activated by natural learning and manual programming. Again, probably true. But the paper doesn't provide any relevant information regarding this phenomena. --- Your answer seems to disagree with me, but displays a disjointed understanding of what I'm really addressing. --- As a lighthearted fun analogy, I present: https://isotropic.org/papers/chicken.pdf https://isotropic.org/papers/chicken.pdf The paper does not prove the existence of chickens. It says chicken a lot, but never addresses the phenomena of chickens existing.
wellpast 1y ago
The fact that the devs thought the AI saved them time is no surprise to me… at least at this point in my career. Developers (people?) in general for some reason just simply cannot see time. It’s why so many people don’t believe in estimation. What I don’t understand is why. Is this like a general human brain limitation (like not being able to visualize four dimensions, or how some folks don’t have an internal monologue)? Or is this more psychodynamic or emotional? It’s been super clear and interesting to me how developers I work with want to believe AI (code generation) is saving them time when it’s clearly obviously not. Is it just the hope that one day it will? Is it fetishization of AI? Why in an industry that so requires clarity of thinking and expression (computer processors don’t like ambiguity), can we be so bad at talking about, thinking about… time? Don’t get me started on the static type enthusiasts who think their strong type system (another seeming fetish) is saving them time.
deleted 1y ago
[deleted]
munificent 1y ago
> The inability of developers to tell if a tool sped them up or slowed them down is fascinating in itself, probably applies to many other forms of human endeavour, and explains things as varied as why so many people think that AI has made them 10 times more productive, why I continue to use Vim, why people drive in London etc. In boating, there's a notion of a "set and drift" which describes how wind and current pushes a boat off course. If a mariner isn't careful, they'll end up far from their destination because of it. This is because when you're sitting in a boat, your perception of motion is relative and local. You feel the breeze on your face, and you see how the boat cuts through the surrounding water. You interpret that as motion towards your destination, but it can equally consist of wind and current where the medium itself is moving. I think a similar effect explains all of these. Our perception of "making progress" is mostly a sense of motion and "stuff happening" in our immediate vicinity. It's not based on a perception of the goal getting closer, which is much harder to measure and develop an intuition for. So people tend to choose strategies that make them feel like they're making progress even if it's not the most effective strategy. I think this is why people often take "shortcuts" when driving that are actually longer. All of the twists and turns keep them busy and make them feel like they're making more progress than zoning out on a boring interstate does.
- thinkingemote 1y ago
  Exactly! Waze the navigation app tends to route users on longer routes but which feels more fast. When driving we perceive our journey as fast or slow not by the actual length but by our memories of what happened. Waze knows human drivers are happier with driving a route that may be longer in time and distance of they feel like they are making progress with the twists and turns. Ai tools makes programming feel easier. That it might be actually less productive is interesting but we humans prefer the easier shortcuts. Our memories of coding with AI tells us that we didn't struggle and therefore we made progress.
  - tjr 1y ago
    That sounds like a navigation tool that I absolutely do not want! Occasionally I do enjoy meandering around, but usually fastest / shortest path would be preferred. And I'm not sure about the other either. In my 20+ year career in aerospace software, the most memorable times were solving interesting problems, not days with no struggle just churning out code.
i_love_retros 1y ago
The AI hype will die off just like block chain and web3. LLMs are a solution in search of a problem. All the VCs are gonna lose a ton of money! OpenAI will be NopenAI, relegated to the dustbin of history. We never asked for this, nobody wants it. Companies using AI and promoting it in their products will be seen as tacky and cheap. Just like developers and artists that use it.
diamond559 1y ago
Measure twice cut once. Not cut 100 times and hope it does it right once.
trey-jones 1y ago
Doing my own post-mortem of a recent project (the first that I've leaned on "AI" tools to any extent), my feeling was the following: 1. It did not make me faster. I don't know that I expected it to. 2. It's very possible that it made me slower. 3. The quality of my work was better. Slower and better are related here, because I used these tools more to either check ideas that I had for soundness, or to get some fresh ideas if I didn't have a good one. In many cases the workflow would be: "I don't like that idea, what else do you have for me?" There were also instances of being led by my tools into a rabbit hole that I eventually just abandoned, so that also contributes to the slowness. This might happen in instances where I'm using "AI" to help cover areas that I'm less of an expert in (and these were great learning experiences). In my areas of expertise, it was much more likely that I would refine my ideas, or the "AI" tool's ideas into something that I was ultimately very pleased with, hence the improved quality. Now, some people might think that speed is the only metric that matters, and certainly it's harder to quantify quality - but it definitely felt worth it to me.
- jpc0 1y ago
  I do this a lot and absolutely think it might even improve it, and this is why I like the current crop of AIs that are more likely to be argumentative and not just capitulate. I will ask the AI for an idea and then start blowing holes in its idea, or will ask it to do the same for my idea. And I might end up not going with it’s idea regardless but it got me thinking about things I wouldn’t have thought about. Effectively its like chatting to a coworker that has a reasonable idea about the domain and can bounce ideas around.
  - trey-jones 1y ago
    I'm on record saying it's "like the smartest coworker I've ever had" (no offense).
remorses 1y ago
Using AI agents productively requires setting up a repository for collaboration, it means writing docs and making the build process easy and fast. As any other tool AI is slow to adopt but has huge gains later on
stevekrouse 1y ago
Such a great essay! Peter Naur's thesis is also the central point in my talk about vibe coding from last month: https://www.youtube.com/watch?v=1WC8dxMC4Xw https://www.youtube.com/watch?v=1WC8dxMC4Xw I'm spending an inordinate amount of time turning that video into an essay, but I feel like I'm being scooped already, so here's my current draft in case anyone wants to get a sneak preview: https://valdottown--89ed76076a6544019f981f7d4397d736.web.val.run/vibe-code https://valdottown--89ed76076a6544019f981f7d4397d736.web.val... Feedback appreciated :)
hakfoo 1y ago
My two cents: My experience with AI is that it's workable for "approximate" things, but it's frustratingly difficult to use as a precision tool. It works great for the trivial demos, where you say "here's an API, build a client" without significant constraints, because that use case is a pretty wide, vague goal. I wasn't going to hold it accountable for matching existing corporate branding, code style, or how to use storage efficiently, so it can work fine. But most of the real work is in the "precision tool" space. You aren't building that many blank-slate API clients, many of the actual tickets are "flip bit 29 of data structure XQ33 when it's a married taxpayer filing singly and huckleberries are in season". The actual change is 3 lines of code, and the effort is in thinking and understanding the problem (and the hundreds of lines of misdocumented code surrounding the problem). I've had Claude decide it wanted to refactor a bunch of unrelated code after asking for a minor, specific change. Or the classic "here's 2000 lines of code that solve the problem in a highly Enterprise way, when the real developer would look at the problem and spit up 150 lines of actual functionality". You can either spend 30 minutes writing the prompt to do the specific precision thing you want and only that, or you can just write the fix directly.
kazinator 1y ago
> Interestingly the developers predict that AI will make them faster, and continue to believe that it did make them faster, even after completing the task slower than they otherwise would! It's like a form of gambling in which you don't have a simple indicator that you're going broke. The addiction is the same though.
sltr 1y ago
A couple of months ago I put forth Naur's program theory as an argument why LLM's can't replace human developers: > LLMs as they currently exist cannot master a theory, design, or mental construct because they don't remember beyond their context window. Only humans can can gain and retain program theory. https://news.ycombinator.com/item?id=44114631 https://news.ycombinator.com/item?id=44114631
cadamsdotcom 1y ago
The study doesn’t provide data that can be extrapolated in any way to any statement about anybody slowing down or speeding up. 1. They used Cursor, which makes you spend your whole day saying “yes” to code changes. Cursor is the windows vista “cancel or allow” of agentic coding. 2. Every user was a Cursor beginner. The study measured developers’ effectiveness at writing code during the ramp up period while they were learning to use a bad tool.

363 points by jwhiles 1y ago

AI tends to slow us down because we don't really know what it's good at. Can it write a proper Nginx config? I don't know—let's try. And then we end up wasting 30 minutes on it. Fully autonomous coding tools like v0, a0, or Aider work well as long as the context is small. But once the context grows—usually due to mistakes made in earlier steps—they just can’t keep up. There's no real benefit of "try again" loop yet. For now, I think simple VSCode extensions are the most useful. You get focused assistance on small files or snippets you’re working on, and that’s usually all you need.

ethan_smith 1y ago
The context switching cost between coding and AI interaction is substantial and rarely measured in these studies. Each prompt/review cycle breaks flow state, which is particularly damaging for complex programming tasks where deep concentration yields the greatest productivity.
- bluefirebrand 1y ago
  This has been my experience too Ever since my company made switching to Cursor mandatory, I have not been able to hit any kind of flow. I know my own productivity has plummeted and I suspect many others are as well, but no one is saying anything I have spoken up once or twice and only been smacked down for my troubles, so I am not surprised everyone else is clammed up

doc_manhat 1y ago

I directionally disagree with this: ``` It's common for engineers to end up working on projects which they don't have an accurate mental model of. Projects built by people who have long since left the company for pastures new. It's equally common for developers to work in environments where little value is placed on understanding systems, but a lot of value is placed on quickly delivering changes that mostly work. In this context, I think that AI tools have more of an advantage. They can ingest the unfamiliar codebase faster than any human can, and can often generate changes that will essentially work. ``` Reason: you cannot evaluate the work accurately if you have no mental model. If there's a bug given the systems unwritten assumptions you may not catch it. Having said that it also depends on how important it is to be writing bug free code in the given domain I guess. I like AI particularly for green field stuff and one off scripts as it let's you go faster here. Basically you build up the mental model as you're coding with the AI. Not sure about whether this breaks down at a certain codebase size though.

horsawlarway 1y ago
Just anecdotally - I think your reason for disagreeing is a valid statement, but not a valid counterpoint to the argument being made. So > Reason: you cannot evaluate the work accurately if you have no mental model. If there's a bug given the systems unwritten assumptions you may not catch it. This is completely correct. It's a very fair statement. The problem is that a developer coming into a large legacy project is in this spot regardless of the existence of AI. I've found that asking AI tools to generate a changeset in this case is actually a pretty solid way of starting to learn the mental model. I want to see where it tries to make changes, what files it wants to touch, what libraries and patterns it uses, etc. It's a poor man's proxy for having a subject matter expert in the code give you pointers. But it doesn't take anyone else's time, and as long as you're not just trying to dump output into a PR can actually be a pretty good resource. The key is not letting it dump out a lot of code, in favor of directional signaling. ex: Prompts like "Which files should I edit to implement a feature which does [detailed description of feature]?" Or "Where is [specific functionality] implemented in this codebase?" Have been real timesavers for me. The actual code generation has probably been a net time loss.
- Roscius 1y ago
  > I've found that asking AI tools to generate a changeset in this case is actually a pretty solid way of starting to learn the mental model. This. Leveraging the AI to start to develop the mental model is an advantage. But, using the AI is a non-trivial skill set that needs to be learned. Skepticism of what it's saying is important. AI can be really useful just like a 747 can be useful, but you don't want someone picked off the street at random flying it.
  - bluefirebrand 1y ago
    > This. Leveraging the AI to start to develop the mental model is an advantage Is there any evidence that AI helps you build the mental model of an unfamiliar codebase more quickly? In my experience trying to use AI for this it often leads me into the weeds
- 1y ago

gjsman-1000 1y ago

What I thought was fascinating, and should be a warning sign to everyone here: Before beginning the study, the average developer expected about a 20% productivity boost. After ending the study, the average developer (potentially: you) believed they actually were 20% more productive. In reality, they were 0% more productive at best, and 40% less productive at worst. Think about what it would be like to be that developer; off by 60% about your own output. If you can't even gauge your own output without being 40% off on average, 60% off at worst; be cautious about strong opinions on anything in life. Especially politically. Edit 1: Also consider, quite terrifyingly, if said developers were in an online group, together, like... here. The one developer who said she thought it made everyone slower (the truth in this particular case), would be unanimously considered an idiot, downvoted to the full -4, even with the benefit of hindsight. Edit 2: I suppose this goes to show, that even on Hacker News, where there are relatively high-IQ and self-aware individuals present... 95% of the crowd can still possibly be wildly delusional. Stick to your gut, regardless of the crowd, and regardless of who is in it.

pphysch 1y ago
Given how deadlines/timelines tend to (not) work in SWE, this is not surprising.
- gjsman-1000 1y ago
  Perhaps; but this is a developer's own output with an AI tool, compared against their own historical output when they didn't use it. Apparently, the average developer (read: quite possibly most people here) can't even hit the broadside of a barn in estimating their own productivity.
  - sureglymop 1y ago
    That doesn't surprise me at all. Isn't software engineering in essence about being constantly confronted with new problems to solve and having to come up with a sufficient one on the fly? It seems very hard to estimate this, even if you know yourself well.

xyst 1y ago

Not surprising. Use of LLM has only been helpful in initial exploration of unknown code bases or languages for me. Using it beyond that is just more work. First parse the broken response, remove any useless junk, have it reprocess with updated query. It’s a nice tool to have (just as search engines gave us easy access to multiple sources/forums), but its limitations are well known. Trying to use it 100% as intended is a massive waste of time and resources (energy use…)

nico 1y ago

> They are experienced open source developers, working on their own projects I just started working on a 3-month old codebase written by someone else, in a framework and architecture I had never used before Within a couple hours, with the help of Claude Code, I had already created a really nice system to replicate data from staging to local development. Something I had built before in other projects, and I new that manually it would take me a full day or two, especially without experience in the architecture That immediately sped up my development even more, as now I had better data to test things locally Then a couple hours later, I had already pushed my first PR. All code following the proper coding style and practices of the existing project and the framework. That PR, would have taken me at least a couple of days and up to 2 weeks to fully manually write out and test So sure, AI won’t speed everyone or everything up. But at least in this one case, it gave me a huge boost As I keep going, I expect things to slow down a bit, as the complexity of the project grows. However, it’s also given me the chance to get an amazing jumpstart

kevmo314 1y ago
You've missed the point of the article, which in fact agrees with your anecdote. > It's equally common for developers to work in environments where little value is placed on understanding systems, but a lot of value is placed on quickly delivering changes that mostly work. In this context, I think that AI tools have more of an advantage. They can ingest the unfamiliar codebase faster than any human can, and can often generate changes that will essentially work.
- moogleii 1y ago
  That would be an aside, or a comment, not the point of the article.
- antonvs 1y ago
  > You've missed the point of the article Sadly clickbait headlines like the OP, "AI slows down open source developers," spread this misinformation, ensuring that a majority of people will have the same misapprehension.

rosspackard 1y ago

One mediocre paper/study (it should not even be called that with all the bias and sample size issues) and now we have to put up with stories re-hashing and dissecting it. I really hope these don't get upvoted more in the future. 16 devs. And they weren't allowed to pick which tasks they used the AI on. Ridiculous. Also using it on "old and >1 million line" codebases and then extrapolating that to software engineering in general. Writers like this then theorize why AI isn't helpful, then those "theories" get repeated until it feels less like a theory and more like a fact and it all proliferates into an echo chamber of AI isn't a useful tool. There have been too many anecdotes and my own personal experience to ignore that it isn't useful. It is a tool and you have to learn it to be successful with it.

sumeno 1y ago
> And they weren't allowed to pick which tasks they used the AI on. They were allowed to pick whether or not to use AI on a subset of tasks. They weren't forced to use AI on tasks that don't make sense for AI
- rosspackard 1y ago
  Half the tasks they were not allowed to use AI.
  - sumeno 1y ago
    Yes, and the other half they had the option to use AI. That's why I said they were allowed to pick whether or not to use AI on a subset of tasks. On the other subset they were not allowed to use AI.
- throwaway284927 1y ago
  That is not true, usage of AI was decided randomly. From the paper: "To directly measure the impact of AI tools on developer productivity, we conduct a randomized controlled trial by having 16 developers complete 246 tasks (2.0 hours on average) on well-known open-source repositories (23,000 stars on average) they regularly contribute to. Each task is randomly assigned to allow or disallow AI usage, and we measure how long it takes developers to complete tasks in each condition."

uludag 1y ago

Great article and I was having very similar thoughts with regards to this productivity study and the "Programming as Theory Building" paper. I'm starting to be convinced that if you are the original author of a program and still have the program's context in the head, you are the asymptote to which any and all AI systems will approach but never surpass: maybe not in terms of raw coding speed, but in terms of understanding the program, its vision of development, its deficiencies and hacks, its context, its users and what they want, the broader culture the program exists in, etc. I really like how the author then brought up the point that for most daily work we don't have the theory built, even a small fraction of it, and that this may or may not change the equation.

conartist6 1y ago
Thanks, <3

cratermoon 1y ago

dissected https://www.fightforthehuman.com/are-developers-slowed-down-by-ai-evaluating-an-rct-and-what-it-tells-us-about-developer-productivity/ https://www.fightforthehuman.com/are-developers-slowed-down-...

omnicognate 1y ago

All these studies that show "AI makes developers x% more/less productive" are predicated on the idea that developer "productivity" can be usefully captured in a single objectively measurable number. Just one problem with that...

narush 1y ago
Thanks for the feedback! I strongly agree this is not the only measure of developer productivity -- but it's certainly one of them. I think this measure as speaks very directly to how _many_ developers (myself included) understand the impact of AI tools on their own work currently (e.g. just speeding up implementation speed). (The SPACE [1] framework is a pretty overview of considerations here; I agree with a lot of it, although I'll note that METR [2] has different motivations for studying developer productivity than Microsoft does.) [1] https://dl.acm.org/doi/10.1145/3454122.3454124 https://dl.acm.org/doi/10.1145/3454122.3454124 [2] https://metr.org/about https://metr.org/about
charcircuit 1y ago
As long as the true productivity is correlated with that number it should be fine.

yomismoaqui 1y ago

Someone on X said that these agentic AI tools (Claude Code, Amp, Gemini Cli) are to programming like the table saw was to hand-made woodworking. It can make some things faster and better than a human with a saw, but you have to learn how to use them right (or you will loose some fingers). I personally find that agentic AI tools make me be more ambitious in my projects, I can tackle some things I didn't tthougth about doing before. And I also delegate work that I don't like to them because they are going to do it better and quicker than me. So my mind is free to think on the real problems like architecture, the technical debt balance of my code... Problem is that there is the temptation of letting the AI agent do everything and just commit the result without understanding YOUR code (yes, it was generated by an AI but if you sign the commit YOU are responsible for that code). So as with any tool try to take the time to understand how to better use it and see if it works for you.

bgwalter 1y ago
"You are using it wrong!" This is insulting to all pre-2023 open source developers, who produced the entire stack that the "AI" robber barons use in their companies. It is even more insulting because no actual software of value has been demonstrably produced using "AI".
- yomismoaqui 1y ago
  > It is even more insulting because no actual software of value has been demonstrably produced using "AI". Claude Code and Amp (equivalent from Sourcegraph) are created by humans using these same tools to add new features and fix bugs. Having used both tools for some weeks I can tell you that they provide a great value to me, enough that I see paying $100 monthly as a bargain related to that value. Edit: typo
  - jdiff 1y ago
    GP is pointing out the distinct lack of AI driven development in the wild. At this point, agents should be visibly maintaining at least a few popular codebases across this world wide web. The fact that there aren't raises some eyebrows for the claims that are regularly made by proponents. Not just the breathless proponents, either. Even taking claims very conservatively, FOSS maintainer burnout should be a thing of the past, but the only noted interaction with AI seems to be amplifying it.

d00mB0t 1y ago

Blasphemy! How dare you say our Emperor has no clothes! AI is becoming a cult and I'm not here for it.

gr8beehive 1y ago

Mirror neurons got people drinking the same stupid kool aid without realizing it.

narush 1y ago

Hey HN -- study author here! (See previous thread on the paper here [1].) I think this blog post is an interesting take on one specific factor that is likely contributing to slowdown. We discuss this in the paper [2] in the section "Implicit repository context (C.1.5)" -- check it out if you want to see some developer quotes about this factor. > This is why AI coding tools, as they exist today, will generally slow someone down if they know what they are doing, and are working on a project that they understand. I made this point in the other thread discussing the study, but in general, these results being surprising makes it easy to read the paper, find one factor that resonates, and conclude "ah, this one factor probably just explains slowdown." My guess: there is no one factor -- there's a bunch of factors that contribute to this result -- at least 5 seem likely, and at least 9 we can't rule out (see the full factors table on page 11). > If there are no takers then I might try experimenting on myself. This sounds super cool! I'd be very excited to see how you set this up + how it turns out... please do shoot me an email (in the paper) if you do this! > AI slows down open source developers. Peter Naur can teach us why Nit: I appreciate how hard it is to write short titles summarizing the paper (the graph title is the best I was able to do after a lot of trying) -- but I might have written this "Early-2025 AI slows down experienced open-source developers. Peter Naur can give us more context about one specific factor." It's admittedly less of a catchy-title, but I think getting the qualifications right are really important! Thanks again for the sweet write-up! I'll hang around in the comments today as well. [1] https://news.ycombinator.com/item?id=44522772 https://news.ycombinator.com/item?id=44522772 [2] https://metr.org/Early_2025_AI_Experienced_OS_Devs_Study.pdf https://metr.org/Early_2025_AI_Experienced_OS_Devs_Study.pdf

antonvs 1y ago
> Early-2025 AI slows down experienced open-source developers. Even that's too general, because it'll depend on what the task is. It's not as if open source developers in general never work on tasks where AI could save time.
- narush 1y ago
  We call this over-generalization out specifically in the "We do not provide evidence that:" table in the blog post and paper - I agree there are tasks these developers are likely sped up on with early-2025 tools.

neuroelectron 1y ago

Good article and it makes sense. I wish I had sometime in my career worked on a codebase that was possible to be understood without 10 years of experience. Instead most of my development time was spent tracing execution paths through tangles of abstractions in nested objects in 10M LOC legacy codebases. My buddy who introduced me to the job is still doing it today and now uses AI and this has given him the free time to start working on his own side projects. So there's certain types if jobs where AI will certainly speed up your development.

bunderbunder 1y ago

> It's a really fabulous study... Ehhhh... not so much. It had serious design flaws in both the protocol and the analysis. This blog post is a fairly approachable explanation of what's wrong with it: https://www.argmin.net/p/are-developers-finally-out-of-a-job https://www.argmin.net/p/are-developers-finally-out-of-a-job

narush 1y ago
Hey, thanks for linking this! I'm a study author, and I greatly appreciate that this author dug into the appendix and provided feedback so that other folks can read it as well. A few notes if it's helpful: 1. This post is primarily worried about ordering considerations -- I think this is a valid concern. We explicitly call this out in the paper [1] as a factor we can't rule out -- see "Bias from issue completion order (C.2.4)". We have no evidence this occurred, but we also don't have evidence it didn't. 2. "I mean, rather than boring us with these robustness checks, METR could just release a CSV with three columns (developer ID, task condition, time)." Seconded :) We're planning on open-sourcing pretty much this data (and some core analysis code) later this week here: https://github.com/METR/Measuring-Early-2025-AI-on-Exp-OSS-Devs https://github.com/METR/Measuring-Early-2025-AI-on-Exp-OSS-D... - star if you want to dig in when it comes out. 3. As I said in my comment on the post, the takeaway at the end of the post is that "What we can glean from this study is that even expert developers aren’t great at predicting how long tasks will take. And despite the new coding tools being incredibly useful, people are certainly far too optimistic about the dramatic gains in productivity they will bring." I think this is a reasonable takeaway from the study overall. As we say in the "We do not provide evidence that:" section of the paper (Page 17), we don't provide evidence across all developers (or even most developers) -- and ofc, this is just a point-in-time measurement that could totally be different by now (from tooling and model improvements in the past month alone). Thanks again for linking, and to the original author for their detailed review. It's greatly appreciated! [1] https://metr.org/Early_2025_AI_Experienced_OS_Devs_Study.pdf https://metr.org/Early_2025_AI_Experienced_OS_Devs_Study.pdf
- bunderbunder 1y ago
  Thanks for the response, you make some very points. Sorry, I had missed your response on the original post. I don't know if it was there yet, or because for some reason their blog is configured to only show the first two comments by default. :/ Either way, my bad. I think my bias as someone who spends too much time looking at social science papers is that the protocol allows for spillover effects that, to me, imply that the results must be interpreted much more cautiously than a lot of people are doing. (And then on top of that I'm trying to be hyper-cautious and skeptical when I see a paper whose conclusions align with my biases on this topic.) Granted, that sort of thing is my complaint about basically every study on developer productivity when using LLMs that I've seen so far. So I appreciate how difficult this is to study in practice.

tomasz_fm 1y ago

Only one developer in this study had more than 50h of Cursor experience, including time spent using Cursor during the study. That one developer saw a 25% speed improvement. Everyone else was an absolute Cursor beginner with barely any Cursor experience. I don't find it surprising that using tools they're unfamiliar with slows software engineers down. I don't think this study can be used to reach any sort of conclusion on use of AI and development speed.

Art9681 1y ago
This is exactly my same take. Any tool an engineer is inexperienced with will slow them down. AI is no different.
- bluefirebrand 1y ago
  This runs counter to the starry eyed promises of AI letting people with no experience accomplish things
  - TeMPOraL 1y ago
    That promise is true, though, and the two claims are not opposite. The devil is in details, specifically in what you mean by "people" and "accomplish things". If by "people" you mean "general public", and by "accomplish things" you mean solving some immediate problems, that may or may not involve authoring a script or even a small app - then yes, this is already happening, and is a big reason behind the AI hype as it is. If by "people" you mean "experienced software engineers", and by "accomplish things" you mean meaningful contributions to a large software product, measured by high internal code and process quality standards, then no - AI tools may not help with that directly, though chances are greater when you have enough experience with those tools to reliably give them right context and steer away from failure modes. Still, solving one-off problems != incremental improvements to a large system.
    - bluefirebrand 1y ago
      > If by "people" you mean "experienced software engineers", My post is a single sentence and I literally wrote "people with no experience"

whatever1 1y ago

They didn’t use the latest model that was released yesterday night. Follow my paid course to learn how to vibe code/s

methuselah_in 1y ago

Those of current generation students who have access to ai might become slow over time. Because when things are not readily available then they have to struggle and work harder in that process, at that time I thing human a lot of secondary things ! Now when everything is easily available especially knowledge without knowing how to struggle with basics. It will eventually make kids dumb. But can be opposite also. Eventually even I become slow even I keep on using chat gpt or gemini.

piker 1y ago

My main two attempts at using an “agentic” coding workflow were trying to incorporate an Outlook COM interface into my rust code base and to streamline an existing abstract windows API interaction to avoid copying memory a couple of times. Both wasted tremendous amounts of time and were ultimately abandoned leaving me only slightly more educated about windows development. They make great autocompletion engines but I just cannot see them being useful in my project otherwise.

crinkly 1y ago
This is typically what I see when I’ve seen it applied. And as always trying to hammer nails in with a banana. Rather than fit two generally disparate things together it’s probably better to just use VSTO and C# (hammer and nails) rather than some unholy combination no one else has tried or suffered through. When it goes wrong there’s more info to get you unstuck.
- piker 1y ago
  To be fair though, unsafe rust (where the COM lives) is basically just C, so I totally expected it to be tractable in the same way it has been tractable for the last 20ish years? But it isn’t. Why is interacting with the OS’ API in a compiled language the wrong approach in 2025? Why must I use this managed Frankenstein’s monster of dotnet? I didn’t want to ship or expect a whole runtime for what should be a tiny convenience DLL. Insane
charcircuit 1y ago
I had the opposite experience. Gemini was able to work with COM and accomplish what I needed despite me never using COM before.
- tonyedgecombe 1y ago
  I've done a lot of work with COM over the years and that is the last technology I would trust to an AI. It's very easy to write COM code that appears to work but contains subtle bugs.

joshmarlow 1y ago

I've gotten some pretty cool things working with LLMs doing most of the heavy lifting using the following approaches: * spec out project goals and relevant context in a README and spec out all components; have the AI build out each component and compose them. I understand the high-level but don't necessarily know all of the low-level details. This is particularly helpful when I'm not deeply familiar with some of the underlying technologies/libraries. * having an AI write tests for code that I've verified is working. As we all know, testing is tedious - so of course I want to automate it. And we written tests (for well written code) can be pretty easy to review.

antimora 1y ago

I'm one of the regular code reviewers for Burn (a deep learning framework in Rust). I recently had to close a PR because the submitter's bug fix was clearly written entirely by an AI agent. The "fix" simply muted an error instead of addressing the root cause. This is exactly what AI tends to do when it can't identify the actual problem. The code was unnecessarily verbose and even included tests for muting the error. Based on the person's profile, I suspect their motivation was just to get a commit on their record. This is becoming a troubling trend with AI tools.

meindnoch 1y ago
>a deep learning framework in Rust [...] This is becoming a troubling trend with AI tools. The serpent is devouring its own tail.
- LoganDark 1y ago
  Deep learning can be incredibly cool and not just used for AI slop.
- TeMPOraL 1y ago
  OTOH when they'll start getting good AI contributions, then... it'll be too late for us all.
tomrod 1y ago
As a side question: I work in AI, but mostly python and theory work. How can I best jump into Burn? Rust has been intriguing to me for a long time
lvl155 1y ago
This is a real problem that’s only going to get worse. With the major model providers basically keeping all the data themselves, I frankly don’t like this trend long term.
dawnerd 1y ago
That's what I love about LLMs. You can spot it doesn't know the answer, tell it that it's wrong and it'll go, "You're absolutely right. Let me actually fix it" It scares me how much code is being produced by people without enough experience to spot issues or people that just gave up caring. We're going to be in for wild ride when all the exploits start flowing.

hartator 1y ago

I am not super sure how to quickly writing benchmark scripts that are one-shot used slows anyone down, but okay.

lsy 1y ago

Typically debugging, e.g., a tricky race condition in an unfamiliar code base would require adding logging, refactoring library calls, inspecting existing logs, and even rewriting parts of your program to be more modular or understandable. This is part of the theory-building. When you have an AI that says "here is the race condition and here is the code change to make to fix it", that might be "faster" in the immediate sense, but it means you aren't understanding the program better or making it easier for anyone else to understand. There is also the question of whether this process is sustainable: does an AI-edited program eventually fall so far outside what is "normal" for a program that the AI becomes unable to model correct responses?

sodapopcan 1y ago
This is always my thought whenever I hear the "AI let me build a feature in a codebase I didn't know in a language I didn't know" (which is often, there is at one in these comments). Great, but what have you learned? This is fine for small contributions, I guess, but I don't hear a lot of stories of long-term maintenance. Unpopular opinion, though, I know.
- threetonesun 1y ago
  I guess it's a question of how anyone learns. There's some value in typing code, I suppose, but with tab complete that's been gone for a long time. Letting AI write something and then reading it seems as good as copying and pasting from some other source.
  - sodapopcan 1y ago
    I'm not super qualified to answer as I haven't gone deep into AI at all. But from my limited observations I'd say yes and no. You generally aren't copy/pasting entire features, just snippets that you yourself have to string together in a sensible way. Of course there are lots of people who still do this and what's why I find most people in this industry infuriating to work with. It's all good when it's boilerplate, and that's actually my primary use of "AI"—it's essentially been a snippets replacement (and is quite good at that).

andix 1y ago

What I noticed: AI development constantly breaks my flow. It makes me more tired, and I work for shorter time periods on coding. It's a myth that you can code a whole day long. I usually do intervals of 1-3 hours for coding, with some breaks in between. Procrastination can even happen on work related things, like reading other project members code/changes for an hour. It has a benefit to some extent, but during this time I don't get my work done. Agentic AI works the best for me. Small refactoring tasks on a selected code snippet can be helpful, but isn't a huge time saver. The worst are AI code completions (first version Copilot style), they are much more noise then help.

rightbyte 1y ago
It would be interesting to record what one do in a day at the desk. Probably quite depressing to watch. Like, I think 1h would be streaching it for mature codebases.
- andix 1y ago
  The 1h I'm talking about is not all the time I might spend reading on code. It's the time I might procrastinate on my tasks with reading unrelated code. Like doom scrolling on social media: Let's see what the fancy new guy got done this week. I need to feel better, I'm just going to look at the commits of the guy in the other team that always breaks production. Let's see how close he got to that recently, ...

afro88 1y ago

I said this when the linked paper was shared and got downvotes: it's based on early 2025 data. My point isn't that it should be completely up to date, but that how we need to consider it in that context. This is pre Claude 4, Claude Code. Pre Gemini 2.5 even. These models are such a big step up from what came previously. Just like we put a (2023) on articles here so they are considered in the right context, so too this paper should be. Blanket "AI tools slow sown development" statements with a "look this rigorous paper says so!" is ignoring a key variable: the rate of effectiveness improvement. If said paper evaluated with the current models, the picture would be different. Also in 3 months time. AI tools aren't a static thing that either works or don't indefinitely.

tonyedgecombe 1y ago
>This is pre Claude 4, Claude Code. Pre Gemini 2.5 even. The most interesting point from the article wasn't about how well the AI's worked, rather it was the gap between peoples perception and their actual results.

blake1 1y ago

I think a reasonable summary of the study referenced is that: "AI creates the perception of productivity enhancements far beyond the reality." Even within the study, there were some participants who saw mild improvements to productivity, but most had a significant drop in productivity. This thread is now full of people telling their story about huge productivity gains they made with AI, but none of the comments contend with the central insight of this study: that these productivity gains are illusions. AI is a product designed to make you value the product. In matters of personal value, perception is reality, no question. Anyone relying heavily on AI should really be worried that it is mostly a tool for warping their self-perception, one that creates dependency and a false sense of accomplishment. After all, it speaks a highly optimized stream of tokens at you, and you really have to wonder what the optimization goal was.

BriggyDwiggs42 1y ago
I’ve noticed that you can definitely use them to help you learn something, but that your understanding tends to be more abstract and LLM-like that way. You definitely want to mix it up when learning too.
- daxfohl 1y ago
  I've also had bad results with hallucinations there. I was trying to learn more about multi-dimensional qubit algorithms, and spent a whole day learning a bunch of stuff that was fascinating but plain wrong. I only figured out it was wrong at the end of the day when I tried to do a simulation and the results weren't consistent. Early in the chat it substituted a `-1` for an `i`, and everything that followed was garbage. There were also some errors that I spotted real-time and got it to correct itself. But yeah, IDK, it presents itself so confidently and "knows" so much and is so easy to use, that it's hard not to try to use as a reference / teacher. But it's also quite dangerous if you're not confirming things; it can send you down incorrect paths and waste a ton of time. I haven't decided whether the cost is worth the benefit or not. Presumably they'll get better at this over time, so in the long run (probably no more than a year) it'll likely easily exceed the ROI breakeven point, but for now, you do have to remain vigilant.

Kim_Bruning 1y ago

I think different people use these tools differently. I've got mine set up to start in "rubber duck" mode, where I do rubber duck programming, before asking the AI to help me with certain tasks (if at all). Low impact utility scripts? The AI gets let off the leash. Critical core logic? I might do most of the work myself (though having a rubber duck can still be good!)

ringeryless 1y ago

not to mention the annoyance of AI assisted issues being opened, many times incorrectly due to hallucinations. these tickets hammer human teams with nonsense and suck resources away from real issues.

alganet 1y ago

This idea that some developers have some "mental model" and others not is an extraordinary claim, and I don't see extraordinary evidence. It sounds like a good thing, right? "Wow, mental model. I want that, I want to be good and have big brain", which encourages you to believe the bullshit. The truth is, this paper is irrelevant and a waste of time. It only serves the purpose of creating discussion around the subject. It's not science, it's a cupholder for marketing.

imiric 1y ago
You couldn't be more wrong. If you've ever programmed, or worked with programmers, that is not an extraordinary claim at all, but a widely accepted fact. A mental model of the software is what allows a programmer to intuitively know why the software is behaving a certain way, or what the most optimal design for a feature would be. In the vast majority of cases these intuitions are correct, and other programmers should pay attention to them. This ability is what separates those with a mental model and those without. On the other hand, LLMs are unable to do this, and are usually not used in ways that help build a mental model. At best, they can summarize the design of a system or answer questions about its behavior, which can be helpful, but a mental model is an abstract model of the software, not a textual summary of its design or behavior. Those neural pathways can only be activated by natural learning and manual programming.
- alganet 1y ago
  > You couldn't be more wrong. Explanation missing. > If you've ever programmed, or worked with programmers, that is not an extraordinary claim at all. One step ahead of you. I already say this is engineered to encourage belief "I want to be good, big brain, and open source is good, I want to be good big brain". It's marketing. > A mental model of the software is what allows a programmer [yadda yadda] I'm not saying it doesn't exist, I'm saying the paper doesn't provide any relevant information regarding the phenomena. > Those neural pathways can only be activated by natural learning and manual programming. Again, probably true. But the paper doesn't provide any relevant information regarding this phenomena. --- Your answer seems to disagree with me, but displays a disjointed understanding of what I'm really addressing. --- As a lighthearted fun analogy, I present: https://isotropic.org/papers/chicken.pdf https://isotropic.org/papers/chicken.pdf The paper does not prove the existence of chickens. It says chicken a lot, but never addresses the phenomena of chickens existing.

wellpast 1y ago

The fact that the devs thought the AI saved them time is no surprise to me… at least at this point in my career. Developers (people?) in general for some reason just simply cannot see time. It’s why so many people don’t believe in estimation. What I don’t understand is why. Is this like a general human brain limitation (like not being able to visualize four dimensions, or how some folks don’t have an internal monologue)? Or is this more psychodynamic or emotional? It’s been super clear and interesting to me how developers I work with want to believe AI (code generation) is saving them time when it’s clearly obviously not. Is it just the hope that one day it will? Is it fetishization of AI? Why in an industry that so requires clarity of thinking and expression (computer processors don’t like ambiguity), can we be so bad at talking about, thinking about… time? Don’t get me started on the static type enthusiasts who think their strong type system (another seeming fetish) is saving them time.

deleted 1y ago

[deleted]

munificent 1y ago

> The inability of developers to tell if a tool sped them up or slowed them down is fascinating in itself, probably applies to many other forms of human endeavour, and explains things as varied as why so many people think that AI has made them 10 times more productive, why I continue to use Vim, why people drive in London etc. In boating, there's a notion of a "set and drift" which describes how wind and current pushes a boat off course. If a mariner isn't careful, they'll end up far from their destination because of it. This is because when you're sitting in a boat, your perception of motion is relative and local. You feel the breeze on your face, and you see how the boat cuts through the surrounding water. You interpret that as motion towards your destination, but it can equally consist of wind and current where the medium itself is moving. I think a similar effect explains all of these. Our perception of "making progress" is mostly a sense of motion and "stuff happening" in our immediate vicinity. It's not based on a perception of the goal getting closer, which is much harder to measure and develop an intuition for. So people tend to choose strategies that make them feel like they're making progress even if it's not the most effective strategy. I think this is why people often take "shortcuts" when driving that are actually longer. All of the twists and turns keep them busy and make them feel like they're making more progress than zoning out on a boring interstate does.

thinkingemote 1y ago
Exactly! Waze the navigation app tends to route users on longer routes but which feels more fast. When driving we perceive our journey as fast or slow not by the actual length but by our memories of what happened. Waze knows human drivers are happier with driving a route that may be longer in time and distance of they feel like they are making progress with the twists and turns. Ai tools makes programming feel easier. That it might be actually less productive is interesting but we humans prefer the easier shortcuts. Our memories of coding with AI tells us that we didn't struggle and therefore we made progress.
- tjr 1y ago
  That sounds like a navigation tool that I absolutely do not want! Occasionally I do enjoy meandering around, but usually fastest / shortest path would be preferred. And I'm not sure about the other either. In my 20+ year career in aerospace software, the most memorable times were solving interesting problems, not days with no struggle just churning out code.

i_love_retros 1y ago

The AI hype will die off just like block chain and web3. LLMs are a solution in search of a problem. All the VCs are gonna lose a ton of money! OpenAI will be NopenAI, relegated to the dustbin of history. We never asked for this, nobody wants it. Companies using AI and promoting it in their products will be seen as tacky and cheap. Just like developers and artists that use it.

diamond559 1y ago

Measure twice cut once. Not cut 100 times and hope it does it right once.

trey-jones 1y ago

Doing my own post-mortem of a recent project (the first that I've leaned on "AI" tools to any extent), my feeling was the following: 1. It did not make me faster. I don't know that I expected it to. 2. It's very possible that it made me slower. 3. The quality of my work was better. Slower and better are related here, because I used these tools more to either check ideas that I had for soundness, or to get some fresh ideas if I didn't have a good one. In many cases the workflow would be: "I don't like that idea, what else do you have for me?" There were also instances of being led by my tools into a rabbit hole that I eventually just abandoned, so that also contributes to the slowness. This might happen in instances where I'm using "AI" to help cover areas that I'm less of an expert in (and these were great learning experiences). In my areas of expertise, it was much more likely that I would refine my ideas, or the "AI" tool's ideas into something that I was ultimately very pleased with, hence the improved quality. Now, some people might think that speed is the only metric that matters, and certainly it's harder to quantify quality - but it definitely felt worth it to me.

jpc0 1y ago
I do this a lot and absolutely think it might even improve it, and this is why I like the current crop of AIs that are more likely to be argumentative and not just capitulate. I will ask the AI for an idea and then start blowing holes in its idea, or will ask it to do the same for my idea. And I might end up not going with it’s idea regardless but it got me thinking about things I wouldn’t have thought about. Effectively its like chatting to a coworker that has a reasonable idea about the domain and can bounce ideas around.
- trey-jones 1y ago
  I'm on record saying it's "like the smartest coworker I've ever had" (no offense).

remorses 1y ago

Using AI agents productively requires setting up a repository for collaboration, it means writing docs and making the build process easy and fast. As any other tool AI is slow to adopt but has huge gains later on

stevekrouse 1y ago

Such a great essay! Peter Naur's thesis is also the central point in my talk about vibe coding from last month: https://www.youtube.com/watch?v=1WC8dxMC4Xw https://www.youtube.com/watch?v=1WC8dxMC4Xw I'm spending an inordinate amount of time turning that video into an essay, but I feel like I'm being scooped already, so here's my current draft in case anyone wants to get a sneak preview: https://valdottown--89ed76076a6544019f981f7d4397d736.web.val.run/vibe-code https://valdottown--89ed76076a6544019f981f7d4397d736.web.val... Feedback appreciated :)

hakfoo 1y ago

My two cents: My experience with AI is that it's workable for "approximate" things, but it's frustratingly difficult to use as a precision tool. It works great for the trivial demos, where you say "here's an API, build a client" without significant constraints, because that use case is a pretty wide, vague goal. I wasn't going to hold it accountable for matching existing corporate branding, code style, or how to use storage efficiently, so it can work fine. But most of the real work is in the "precision tool" space. You aren't building that many blank-slate API clients, many of the actual tickets are "flip bit 29 of data structure XQ33 when it's a married taxpayer filing singly and huckleberries are in season". The actual change is 3 lines of code, and the effort is in thinking and understanding the problem (and the hundreds of lines of misdocumented code surrounding the problem). I've had Claude decide it wanted to refactor a bunch of unrelated code after asking for a minor, specific change. Or the classic "here's 2000 lines of code that solve the problem in a highly Enterprise way, when the real developer would look at the problem and spit up 150 lines of actual functionality". You can either spend 30 minutes writing the prompt to do the specific precision thing you want and only that, or you can just write the fix directly.

kazinator 1y ago

> Interestingly the developers predict that AI will make them faster, and continue to believe that it did make them faster, even after completing the task slower than they otherwise would! It's like a form of gambling in which you don't have a simple indicator that you're going broke. The addiction is the same though.

sltr 1y ago

A couple of months ago I put forth Naur's program theory as an argument why LLM's can't replace human developers: > LLMs as they currently exist cannot master a theory, design, or mental construct because they don't remember beyond their context window. Only humans can can gain and retain program theory. https://news.ycombinator.com/item?id=44114631 https://news.ycombinator.com/item?id=44114631

cadamsdotcom 1y ago

The study doesn’t provide data that can be extrapolated in any way to any statement about anybody slowing down or speeding up. 1. They used Cursor, which makes you spend your whole day saying “yes” to code changes. Cursor is the windows vista “cancel or allow” of agentic coding. 2. Every user was a Cursor beginner. The study measured developers’ effectiveness at writing code during the ramp up period while they were learning to use a bad tool.