19 ms·

Measuring the impact of AI on experienced open-source developer productivity

• Jabrov 1y ago
  Very interesting methodology, but the sample size (16) is way too low. Would love to see this repeated with more participants.
  • IshKebab 1y ago
    They paid the developers about $75k in total to do this so I wouldn't hold your breath!
    • barbazoo 1y ago
      That's a lot of money for many of us. Do you know those folks were in a HCOL area?
      • IshKebab 1y ago
        No idea. They don't say who they were; just random popular GitHub projects. To be clear it wasn't $75k each.
        • narush 1y ago
          You can see a list of repositories with participating developers in the appendix! Section G.7. Paper is here: https://metr.org/Early_2025_AI_Experienced_OS_Devs_Study.pdf https://metr.org/Early_2025_AI_Experienced_OS_Devs_Study.pdf
      • mapt 1y ago
        It isn't a lot of money for industry research. Changes of +-40% in productivity are an enormous advantage/disadvantage for a large tech company moving tens of billions of dollars a year in cashflow through a pipeline that their software engineers built.
    • lawlessone 1y ago
      Neat, how to sign up??
      • IshKebab 1y ago
        Go back in time, create a popular github repo with lots of stars, be lucky.
      • asdff 1y ago
        I see these things posted on linkedin. Usually asking $40/hr though. But essentially the same thing as the OP outlines: you do some domain related task assigned either with or without an AI tool. Check linked in. They will have really vague titles like "data scientist" though even though that's not what is being described, its study subject. Maybe set 40/hr as a filter on linkedin and see if you can get a few to come up.
  • narush 1y ago
    Noting that most of our power comes from the number of tasks that developers complete; it's 246 total completed issues in the course of this study -- developers do about 15 issues (7.5 with AI and 7.5 without AI) on average.
    • deleted 1y ago
      [deleted]
    • biophysboy 1y ago
      Did you compare the variance within individuals (due to treatment) to the variance between individuals (due to other stuff)?
• kokanee 1y ago
  > developers expected AI to speed them up by 24%, and even after experiencing the slowdown, they still believed AI had sped them up by 20%. I feel like there are two challenges causing this. One is that it's difficult to get good data on how long the same person in the same context would have taken to do a task without AI vs with. The other is that it's tempting to time an AI with metrics like how long until the PR was opened or merged. But the AI workflow fundamentally shifts engineering hours so that a greater percentage of time is spent on refactoring, testing, and resolving issues later in the process, including after the code was initially approved and merged. I can see how it's easy for a developer to report that AI completed a task quickly because the PR was opened quickly, discounting the amount of future work that the PR created.
  • qsort 1y ago
    It's really hard to attribute productivity gains/losses to specific technologies or practices, I'm very wary of self-reported anecdotes in any direction precisely because it's so easy to fool ourselves. I'm not making any claim in either direction, the authors themselves recognize the study's limitations, I'm just trying to say that everyone should have far greater error bars. This technology is the weirdest shit I've seen in my lifetime, making deductions about productivity from anecdotes and dubious benchmarks is basically reading tea leaves.
  • yorwba 1y ago
    Figure 21 shows that initial implementation time (which I take to be time to PR) increased as well, although post-review time increased even more (but doesn't seem to have a significant impact on the total). But Figure 18 shows that time spent actively coding decreased (which might be where the feeling of a speed-up was coming from) and the gains were eaten up by time spent prompting, waiting for and then reviewing the AI output and generally being idle. So maybe it's not a good idea to use LLMs for tasks that you could've done yourself in under 5 minutes.
  • narush 1y ago
    Qualitatively, we don't see a drop in PR quality in between AI-allowed and AI-disallowed conditions in the study; the devs who participate are generally excellent, know their repositories standards super well, and aren't really into the 'get up a bad PR' vibe -- the median review time on the PRs in the study is about a minute. Developers totally spend time totally differently, though, this is a great callout! On page 10 of the paper [1], you can see a breakdown of how developers spend time when they have AI vs. not - in general, when these devs have AI, they spend a smaller % of time writing code, and a larger % of time working with AI (which... makes sense). [1] https://metr.org/Early_2025_AI_Experienced_OS_Devs_Study.pdf https://metr.org/Early_2025_AI_Experienced_OS_Devs_Study.pdf
• dash2 1y ago
  The authors say "High developer familiarity with repositories" is a likely reason for the surprising negative result, so I wonder if this generalizes beyond that.
  • kennywinker 1y ago
    Like if it generalizes to situations where the developer is not familiar with the repo? That doesn’t seem like generalizing, that seems like specifying. Am I wrong in saying that the majority of developer time is spent in repos that they’re familiar with? Every job and project I’ve worked has been on a fixed set of repos the entire time. If AI is only helpful for the first week or two on a project, that’s not very many cases it’s useful for.
    • jbeninger 1y ago
      I'd say I write the majority of my code in areas I'm familiar with, but spend the majority of my _time_ on sections I'm not familiar with, and ai helps a lot more with the latter than the former. I've always felt my coding life is speeding through a hundred lines of easy code then getting stuck on the 101st. Then as I get more experienced that hundred becomes 150, then 200, but always speeding through the easy part until I have to learn something new. So I never feel like I'm getting any faster. 90% of my time is still spent in frustration, even when I'm producing twice the code at higher quality
  • add-sub-mul-div 1y ago
    Without the familiarity would the work be getting done effectively? What does it mean for someone to commit AI code that they can't fully understand?
• noisy_boy 1y ago
  It is 80/20 again - it gets you 80% of the way in 20% of the time and then you spend 80% of the time to get the rest of the 20% done. And since it always feels like it is almost there, sunk-cost fallacy comes into play as well and you just don't want to give up. I think an approach that I tried recently is to use it as a friction remover instead of a solution provider. I do the programming but use it to remove pebbles such as that small bit of syntax I forgot, basically to keep up the velocity. However, I don't look at the wholesale code it offers. I think keeping the active thinking cap on results in code I actually understand while avoiding skill atrophy.
  • eknkc 1y ago
    It works great on adding stuff to an already established codebase. Things like “we have these search parameters, also add foo”. Remove anything related to x…
    • antonvs 1y ago
      Exactly. If you can give it a contract and a context, essentially, and it doesn't need to write a large amount of code to fulfill it, it can be great. I just used it to write about 80 lines of new code like that, and there's no question it saves time.
  • wmeredith 1y ago
    > and then you spend 80% of the time to get the rest of the 20% done This was my pr-AI experience anyway, so getting that first chunk of time back is helpful. Related: One of the better takes I've seen on AI from an experienced developer was, "90% of my skills just became worthless, and the other 10% just became 1,000 times more valuable." There's some hyperbole there, I but I like the gist.
    • skydhash 1y ago
      It’s not funny when you find yourself redoing the first 80%, as the only way to complete the second 80%.
    • 1y ago
• fritzo 1y ago
  As an open source maintainer on the brink of tech debt bankruptcy, I feel like AI is a savior, helping me keep up with rapid changes to dependencies, build systems, release methodology, and idioms.
  • aerhardt 1y ago
    But what about producing actual code?
    • fritzo 1y ago
      Producing code is overrated. There's lots of old code whose lifetime we can extend.
      • fhd2 1y ago
        Very, very much this.
    • resource_waste 1y ago
      I find it useful for simple algorithms and error solving.
  • reaperducer 1y ago
    [flagged]
  • candiddevmike 1y ago
    If you stewarded that much tech debt in the first place, how can you be sure LLM will help prevent it going forward? In my experience, LLMs add more tech debt due to lacking cohesion with it's edits.
• IshKebab 1y ago
  I wonder if the discrepancy is that it felt like it was taking less time because they were having to do less thinking which feels like it is easier and hence faster. Even so... I still would be really surprised if there wasn't some systematic error here skewing the results, like the developers deliberately picked "easy" tasks that they already knew how to do, so implementing them themselves was particularly fast. Seems like they authors had about as good methodology as you can get for something like this. It's just really hard to test stuff like this. I've seen studies proving that code comments don't matter for example... are you going to stop writing comments? No.
  • narush 1y ago
    > which feels like it is easier and hence faster. We explore this factor in section (C.2.5) - "Trading speed for ease" - in the paper [1]. It's labeled as a factor with an unclear effect, some developers seem to think so, and others don't! > like the developers deliberately picked "easy" tasks that they already knew how to do We explore this factor in (C.2.2) - "Unrepresentative task distribution." I think the effect here is unclear; these are certainly real tasks, but they are sampled from the smaller end of tasks developers would work on. I think the relative effect on AI vs. human performance is not super clear... [1] https://metr.org/Early_2025_AI_Experienced_OS_Devs_Study.pdf https://metr.org/Early_2025_AI_Experienced_OS_Devs_Study.pdf
    • IshKebab 1y ago
      Sounds like you've thought of everything!
• tcdent 1y ago
  This study neglects to incorporate the fact that I have forgotten how to write code.
  • resource_waste 1y ago
    I'm curious what space people are working in where AI does their job entirely. I can use it for parts of code, algorithms, error solving, and maybe sometimes a 'first draft'. But there is no way I could finish an entire piece of software with AI only.
    • asdff 1y ago
      Not a lot of people are empowered to create an entire piece of software. Most are probably in the trenches squashing tickets.
      • tcdent 1y ago
        I do create entire pieces of software, and while my workflow is always evolving, it goes something like this: Define schemas, interfaces, and perhaps some base classes that define the attributes I'm thinking about. Research libraries that support my cause, and include them. Reference patterns I have established in other parts of the codebase; internal tooling for database, HTTP services, etc. Instruct the agent to come up with a plan for a first pass at execution in markdown format. Iterate on this plan; "what about X?" Splat a bunch of code down that supports the structure I'm looking for. Iterate. Cleanup. Iterate. Implement unit tests and get them to pass. Go back through everything manually and adjust it to suit my personal style, while at the same time fully understanding what's being done and why. I use STT a lot to have conversations with the agent as we go, and very rarely allow it to make sequential edits without reviewing first; this is a great opportunity to go back and forth and refine what's being written.
        • asdff 1y ago
          You are going well above and beyond what a lot of people do to be fair. There are people in senior roles who are just futzing with json files.
• NewsaHackO 1y ago
  So they paid developers 300 x 246 = about 73K just for developer recruitment for the study, which is not in any academic journal, or has no peer reviews? The underlying paper looks quite polished and not overtly AI generated so I don't want to say it entirely made up, but how were they even able to get funding for this?
  • fabianhjr 1y ago
    Most of the world provides funding for research, the US used to provide funding but now that has been mostly gutted.
  • iLoveOncall 1y ago
    https://metr.org/about https://metr.org/about Seems like they get paid by AI companies, and they also get government funding.
    • deleted 1y ago
      [deleted]
  • narush 1y ago
    Our largest funding was through The Audacious Project -- you can see an announcement here: https://metr.org/blog/2024-10-09-new-support-through-the-audacious-project/ https://metr.org/blog/2024-10-09-new-support-through-the-aud... Per our website, “To date, April 2025, we have not accepted compensation from AI companies for the evaluations we have conducted.” You can check out the footnote on this page: https://metr.org/donate https://metr.org/donate
    • iLoveOncall 1y ago
      This is really disingenuous when you also say that OpenAI and Anthropic have provided you with access and compute credits (on https://metr.org/about https://metr.org/about). Not all payment is cash. Compute credits is still by all means compensation.
      • 1y ago
• deleted 1y ago
  [deleted]
• alganet 1y ago
  [flagged]
• 30minAdayHN 1y ago
  This study focused on experienced OSS maintainers. Here is my personal experience, but a very different persona (or opposite to the one in the study). I always wanted to contribute to OSS but never had time to. Finally was able to do that, thanks to AI. Last month, I was able to contribute to 4 different repositories which I would never have dreamed of doing it. I was using an async coding agent I built[1], to generate PRs given a GitHub issue. Some PRs took a lot of back and forth. And some PRs were accepted as is. Without AI, there is no way I would have contributed to those repositories. One thing that did work in my favor is that, I was clearly creating a failing repro test case, and adding before and after along with PR. That helped getting the PR landed. There are also a few PRs that never got accepted because the repro is not as strong or clear. [1] https://workback.ai https://workback.ai
  • ares623 1y ago
    Did you make the contributions though? Or did the LLM? This is not directed at you, but I am worried that contributors that use AI "exclusively" to contribute to OSS projects are extracting the value (street cred, being seen as part of the project community) without actually contributing anything (by being one more person that knows the codebase and can help steward it). It's the same thing we've seen out of enshittification of everything. Value extraction without giving back. Maybe I'm too much of a cynic. Maybe majority of OSS projects don't care. But I know I will be saddened if one of the OSS projects I care about get taken over by such "value extractors".
    • 30minAdayHN 1y ago
      Did not take it personal. You brought up a good point. I've slightly alternate perspective. Imo, using OSS without contributing is the value extraction without giving back. If someone can fix a bunch of chores (that still take human time), with the use of AI (even though they don't become stewards), I still see it as giving back. Of course, there is a value chain - contributing with AI without understanding code is the bottom of value creation. Like you mentioned, also being a steward is the top of the value chain. Along the way, if the contributor builds some sorta reputation that would help with their career or other outcomes, so be it. So in that sense, I don't see it as enshittification. AI might make a pathway to resolve a bunch of things which otherwise wouldn't be resolved. In fact, this was the line of thinking for the tool we built. Instead of people making these mindless PRs, can we build an agent that can take care of 'trivial' tasks. I manually created PRs to test that hypothesis. There is also a natural self selection here. If someone was able to fix something without understanding any code, that is also indicative of how trivial the task is. There is a reverse effect to my argument though. These "AI contributors" can create a ton of PRs that would create a lot of work for maintainers to review them. In my case, I was being upfront about how I'm raising PRs and requesting permissions if it is OK to work on certain issue. Maintainers are quite open and inviting.
• kristianc 1y ago
  > The developers estimated how long it would take them to complete each task (a) under normal conditions Ah, there’s your issue. There’s not a developer in human history who hasn’t drastically underestimated how long it would take to complete a task.
  • 59nadir 1y ago
    On average the developers overestimated how long tasks would take when not using AI; they undershot their estimates on average. The opposite happened with AI-assisted tasks. The conclusion isn't that "estimates are hard" (they can be), but rather that AI-assistance can lead people to believe they're being more productive than they actually are, because they incorrectly think they've spent less time. The graphs in the paper tell part of that story; the time that is being reduced is in actual programming time, "Reading & Searching", "Testing & Debugging", but that time is being spent elsewhere, notably in parts specific to LLMs (reviewing output, prompting, waiting for the AI to spit out results).
• MYEUHD 1y ago
  This does not mention the open-source developer time wasted while reviewing vibe coded PRs
  • narush 1y ago
    Yeah, I'll note that this study does _not_ capture the entire OS dev workflow -- you're totally right that reviewing PRs is a big portion of the time that many maintainers spend on their projects (and thanks to them for doing this [often hard] work). In the paper [1], we explore this factor in more detail -- see section (C.2.2) - Unrepresentative task distribution. There's some existing lit about increased contributions to OS repositories after the introduction of AI -- I've also personally heard a fear anecdotes about an increase in the number of low-quality PRs from first time contributors, seemingly as a result of AI making it easier to get started -- ofc, the tradeoff is that making it easier to get started has pros to it too! [1] https://metr.org/Early_2025_AI_Experienced_OS_Devs_Study.pdf https://metr.org/Early_2025_AI_Experienced_OS_Devs_Study.pdf
• castratikron 1y ago
  I really like those graphics, does anyone know the tool was used to create them?
  • narush 1y ago
    The graphs are all matplotlib. The methodology figure is built in Figma! (Source: I'm a paper author :)).
• narush 1y ago
  Hey HN, study author here. I'm a long-time HN user -- and I'll be in the comments today to answer questions/comments when possible! If you're short on time, I'd recommend just reading the linked blogpost or the announcement thread here [1], rather than the full paper. [1] https://x.com/METR_Evals/status/1943360399220388093 https://x.com/METR_Evals/status/1943360399220388093
  • jsnider3 1y ago
    It's good to know that Claude 3.7 isn't enough to build Skynet!
  • causal 1y ago
    Hey I just wanted to say this is one of the better studies I've seen - not clickbaity, very forthright about what is being claimed, and presented in such an easy-to-digest format. Thanks so much for doing this.
    • narush 1y ago
      Thanks for the kind words!
  • igorkraw 1y ago
    Could you either release the dataset (raw but anonymized) for independent statistical évaluation or at least add the absolute times of each dev per task to the paper? I'm curious what the absolute times of each dev with/without AI was and whether the one guy with lots of Cursor experience was actually faster than the rest of just a slow typer getting a big boost out of llms Also, cool work, very happy to see actually good evaluations instead of just vibes or observational stuies that don't account for the Hawthorne effect
    • narush 1y ago
      Yep, sorry, meant to post this somewhere but forgot in final-paper-polishing-sprint yesterday! We'll be releasing anonymized data and some basic analysis code to replicate core results within the next few weeks (probably next, depending). Our GitHub is here (http://github.com/METR/ http://github.com/METR/) -- or you can follow us (https://x.com/metr_evals https://x.com/metr_evals) and we'll probably tweet about it.
• incomingpain 1y ago
  Essentially an advertisement against Cursor Pro and/or Claude Sonnet 3.5/3.7 I think personally when i tried tools like Void IDE, I was fighting with Void too much. It is beta software, it is buggy, but also the big one... learning curve of the tool. I havent had the chance to try cursor but i imagine its going to have a learning curve as a new tool. So perhaps there is a slowdown at first expected; but later after you get your context and prompting down pat. Asking specifically for what you want. Then you get your speed up.
• achenet 1y ago
  I find agents useful for showing me how to do something I don't already know how to do, but I could see how for tasks I'm an expert on, I'd be faster without an extra thing to have to worry about (the AI).
• dboreham 1y ago
  Any time you see the word "measuring" in the context of software development, you know what follows will be nonsense and probably in service of someone's business model.
• simonw 1y ago
  Here's the full paper, which has a lot of details missing from the summary linked above: https://metr.org/Early_2025_AI_Experienced_OS_Devs_Study.pdf https://metr.org/Early_2025_AI_Experienced_OS_Devs_Study.pdf My personal theory is that getting a significant productivity boost from LLM assistance and AI tools has a much steeper learning curve than most people expect. This study had 16 participants, with a mix of previous exposure to AI tools - 56% of them had never used Cursor before, and the study was mainly about Cursor. They then had those 16 participants work on issues (about 15 each), where each issue was randomly assigned a "you can use AI" v.s. "you can't use AI" rule. So each developer worked on a mix of AI-tasks and no-AI-tasks during the study. A quarter of the participants saw increased performance, 3/4 saw reduced performance. One of the top performers for AI was also someone with the most previous Cursor experience. The paper acknowledges that here: > However, we see positive speedup for the one developer who has more than 50 hours of Cursor experience, so it's plausible that there is a high skill ceiling for using Cursor, such that developers with significant experience see positive speedup. My intuition here is that this study mainly demonstrated that the learning curve on AI-assisted development is high enough that asking developers to bake it into their existing workflows reduces their performance while they climb that learing curve.
  • nicman23 1y ago
    i just treat ai as a very long auto complete. sometimes it surprises me. on things i do not know, like windows C calls, i think i ought to just search the documentation..
  • lupusreal 1y ago
    A friend of mine, complete non-programmer, has been trying to use ChatGPT to write a phone app. I've been as hands off as I feel I can be, watching how the process goes for him. My observations so far is that it's not going well, he doesn't understand what questions he should be asking so the answers he's getting aren't useful. I encourage him to ask it to teach him the relevant programming but he asks it to help him make the app without programming at all. With more coaching from me, which I might end up doing, I think he would get further. But I expected the chatbot to get him further through the process than this. My conclusion so far is that this technology won't meaningfully shift the balance of programmers to non-programmers in the general population.
• inetknght 1y ago
  > We pay developers $150/hr as compensation for their participation in the study. Can someone point me to these 300k/yr jobs?
  • recursive 1y ago
    These are not 300k/yr jobs.
  • akavi 1y ago
    L5 ("Senior") at any FAANG co, L6 ("Staff") at pretty much any VC-backed startup in the bay.
  • phyzome 1y ago
    This is more like contractor pay rates -- not salaried. Which is appropriate to the nature of the work here.
• nestorD 1y ago
  One thing I could not find on a cursory read is how used were those developers to AI tools. I would expect someone using those regularly to benefit while someone who only played with them a couple of time would likely be slowed down as they deal with the friction of learning to be productive with the tool.
  • uludag 1y ago
    In this case though you still wouldn't necessarily know if the AI tools had a positive causal effect. For example, I practically live in Emacs. Take that away and no doubt I would be immensely less effective. That Emacs improves my productivity and without it I am much worse in no way implies that Emacs is better than the alternatives. I feel like a proper study for this would involve following multiple developers over time, tracking how their contribution patterns and social standing changes. For example, take three cohorts of relatively new developers: instruct one to go all in on agentic development, one to freely use AI tools, and one prohibited from AI tools. Then teach these developers open source (like a course off of this book: https://pragprog.com/titles/a-vbopens/forge-your-future-with-open-source/ https://pragprog.com/titles/a-vbopens/forge-your-future-with...) and have them work for a year to become part of a project of their choosing. Then in the end, track a number of metrics such as leadership position in community, coding/non-coding contributions, emotional connection to project, social connections made with community, knowledge of code base, etc. Personally, my prior probability is that the no-ai group would likely still be ahead overall.
    • iLemming 1y ago
      FWIW, LLM tooling for Emacs is great. gptel for example allows you to converse with wide-range of different models from anywhere in Emacs — you can spontaneously send requests while typing some text or even browsing M-x menu. I often do things like "summarize current paragraph in pdf document" or "create a few anki cards based on this web page content", etc.
• swayvil 1y ago
  AI by design can only repeat and recombine past material. Therefore actual invention is out.
  • 8note 1y ago
    its not a huge deal. i dont need the AI to invent a replacement to the for loop or map function; i only want it to use the those tools. I'm the one providing the invention, it's transforming my invention into an implementation; sometimes better than others.
  • elpakal 1y ago
    underrated comment
  • atleastoptimal 1y ago
    HN moment
  • luibelgo 1y ago
    Is that actually proven?
    • greenchair 1y ago
      The easiest way to see this for yourself is with an image generator. Try asking for a very specific combination of things that would not normally appear together in an artpiece.
  • keeda 1y ago
    Pretty much all invention is novel combination of known techniques. Anything that introduces a fundamental new technique is usually in the realm of groundbreaking papers and Nobel prizes.
• zzzeek 1y ago
  As a project for work, I've been using Claude CLI all week to do as many tasks as possible. So with my week's experience, I'm now an expert in this subject and can weigh in. Two things that stand out to me are 1. it depends a lot on what kind of task you are having the LLM do. and 2. if the LLM process takes more time, it is very likely your cognitive effort was still way less - for sysadmin kinds of tasks, working with less often accessed systems, LLMs can read --help, man pages, doc sites, all for you, and give you the working command right there (And then run it, and then look at the output and tell you why it failed, or how it worked, and what it did). There is absolutely no question that second part is a big deal. Sticking it onto my large open source project to fix a deep, esoteric issue or write some subtle documentation where it doesnt really "get" what I'm doing, yeah it is not as productive in that realm and you might want to skip it for the thinking part there. I think everyone is trying to figure out this question of "when and how" for LLMs. I think the sweet spot is for tasks involving systems and technologies where you'd otherwise be spending a lot of time googling, stackoverflowing, reading man pages to get just the right parameters into commands and so forth. This is cognitive grunt work and the LLMs can do that part very well. My week of effort with it was not really "coding on my open source project"; two examples were, 1. running a bunch of ansible playbooks that I wrote years ago on a new host, where OS upgrades had lots of snags; I worked with Claude to debug all the various error messages and places where the newer OS distribution had different packages, missing packages, etc. it was ENORMOUSLY helpful since I never look at these playbooks and I dont even remember what I did, Claude can read it for you and interpret it as well as you can. 2. I got a bugzilla for a fedora package that I packaged years ago, where they have some change to the directives used in specfiles that everyone has to make. I look at fedora packaging workflows once every three years. I told Claude to read the BZ and just do it. IT DID IT. I had to get involved running the "mock" suite as it needed sudo but Claude gave me the commands. zero googling. zero even reading the new format of the specfile (the bz linked to a tool that does the conversion). From bug received to bug closed and I didnt do any typing at all outside of the prompt. Had it done before breakfast since I didnt even need any glucose for mental energy expended. This would have been a painful and frustrating mental effort otherwise. so the studies have to get more nuanced and survey a lot more than 16 devs I think
  • 8note 1y ago
• geerlingguy 1y ago
  So far in my own hobby OSS projects, AI has only hampered things as code generation/scaffolding is probably the least of my concerns, whereas code review, community wrangling, etc. are more impactful. And AI tooling can only do so much. But it's hampered me in the fact that others, uninvited, toss an AI code review tool at some of my open PRs, and that spits out a 2-page document with cute emoji and formatted bullet points going over all aspects of a 30 line PR. Just adds to the noise, so now I spend time deleting or hiding those comments in PRs, which means I have even _less_ time for actual useful maintenance work. (Not that I have much already.)
• deleted 1y ago
  [deleted]
• heisenbit 1y ago
  AI sometimes points out hygiene issues that may be swept under the carpet but once pointed out can't be ignored anymore. I know I don't need that error handling, I'm certain for the near future but maybe it is needed... Also the code produced by the AI has some impedance match with my natural code. Then one needs to figure out whether that is due to moving best practices, until now ignored best practices or the AI being overwhelmed with code from beginners. This all takes time - some of it is transient, some of it is actually improving things and some of it is waste. The jury is still out there.
• ChrisMarshallNY 1y ago
  It's been very helpful for me. I find ChatGPT the easiest to use; not because it's more accurate (it isn't), but because it seems to understand the intent of my questions most clearly. I don't usually have to iterate much. I use it like a know-it-all personal assistant that I can ask any question to; even [especially] the embarrassing, "stupid" ones. > The only stupid question is the one we don't ask. - On an old art teacher's wall
• 0xmusubi 1y ago
  I find myself having discussions with AI about different design possibilities and it sometimes comes up with ideas I hadn't thought of or features I wasn't aware of. I wouldn't classify this as "overuse" as I often find the discussions useful, even if it's just to get my thoughts down. This might be more relevant for larger scoped tasks or ones where the programmer isn't as familiar with certain features or libraries though.
• deleted 1y ago
  [deleted]
  • deleted 1y ago
    [deleted]
• groos 1y ago
  One thing I've experienced in trying to use LLMs to code in an existing large code base is that it's _extremely_ hard to accurately describe what you want to do. Oftentimes, you are working on a problem with a web of interactions all over the code and describing the problem to an LLM will take far longer than just doing it manually. This is not the case with generating new (boilerplate) code for projects, which is where users report the most favorable interaction with LLMs.
  • 9dev 1y ago
    That’s my experience as well. It’s where Knuth comes in again: the program doesn’t just live in the code, but also in the minds of its creator. Unless I communicate all that context from the start, I can’t just dump years of concepts and strategy out of my brain into the LLM without missing details that would be relevant.
  • phyzome 1y ago
    Hell, a lot of times I can't even explain an idea to my coworkers in a conversation, and I eventually say "I'll just explain it in code instead of words." And I just quickly put up a small PR that makes the skeleton of the changes (or even the entire changeset) and then we continue our conversation (or just do the review).
• AvAn12 1y ago
  N = 16 developers. Is this enough to draw any meaningful conclusions?
  • sarchertech 1y ago
    That depends on the size of the effect you’re trying to measure. If cursor provides a 5x, 10x, or 100x productivity boost as many people are claiming, you’d expect to see that in a sample size of 16 unless there’s something seriously wrong with your sample selection. If you are looking for a 0.1% increase in productivity, then 16 is too small.
    • biophysboy 1y ago
      Well it depends on the variance of the random variable itself. You're right that with big, obvious effects, a larger n is less "necessary". I could see individuals having very different "productivities", especially when the idea is flattened down to completion time.
      • sarchertech 1y ago
        Individuals do have very different productivities, but they are measuring the productivity difference across a single individual.
    • AvAn12 1y ago
      “A quarter of the participants saw increased performance, 3/4 saw reduced performance.” So I think any conclusions drawn on these 16 people doesn’t signify much one way or the other. Cool paper but how is this anything other than a null finding?
      • tripletao 1y ago
        They show a 95% CI excluding zero in Figure 1. By the usual standards of social science, that's not a null finding. They give their methodology in Appendix D. For intuition on why it's insufficient to consider N alone, I assume e.g. that you'd greatly increase your belief that a coin was unfair long before 16 consecutive heads--as already noted, the size of the effect also matters. That relationship isn't intuitive in general, and attempts to replace the math with feelings tend to fail.
• atleastoptimal 1y ago
  I’m not surprised that AI doesn’t help people with 5+ years experience in open source contribution, but I’d imagine most people aren’t claiming AI tools are at senior engineer level yet. Soon once the tools and how people use them improve AI won’t be a hinderance for advanced tasks like this, and soon after AI will be able to do these prs on their own. It’s inevitable given the rate of improvement even since this study.
  • artee_49 1y ago
    Even for senior levels the claim has been that AI will speed up their coding (take it over) so they can focus on higher level decisions and abstract level concepts. These contributions are not those and based on prior predictions the productivity should have gone up.
    • atleastoptimal 1y ago
      It would be different I'm sure if they were making contributions to repos they had less familiarity with. In my experience and talking with those who use AI most effectively, it is best leveraged as a way of getting up to speed or creating code for a framework/project you have less familiarity with. The general ratio determining the effectiveness of non-AI coding vs AI coding is the familiarity the user has with the codebase * the complexity of the codebase : the amount of closed-loop abstractions in the tasks the coder needs to carry out. Currently AI is like a junior engineer, and if you don't have good experience managing junior engineers, AI isn't going to help you as much.
      • sarchertech 1y ago
        I’ve been doing this for a while now. Junior engineers are pretty near universally terrible when measured by short term ROI. The only reason you would ever want to pay a truly junior engineer is because you can teach them. If someone told me “you can have a free junior engineer, but they get swapped out each week for a new person”, I’d say no thanks. I’m sure someone could figure out a way to make money on in that situation, but it wouldn’t be by building anything I’d be comfortable attaching my name to or or would want to use myself
• pera 1y ago
  Wow these are extremely interesting results, specially this part: > This gap between perception and reality is striking: developers expected AI to speed them up by 24%, and even after experiencing the slowdown, they still believed AI had sped them up by 20%. I wonder what could explain such large difference between estimation/experience vs reality, any ideas? Maybe our brains are measuring mental effort and distorting our experience of time?
  • rsynnott 1y ago
    > I wonder what could explain such large difference between estimation/experience vs reality, any ideas? This bit I wasn't at all surprised by, because this is _very common_. People who are doing a [magic thing] which they believe in often claim that it is improving things even where it empirically isn't; very, very common with fad diets and exercise regimens, say. You really can't trust subjects' claims of efficacy of something that's being tested on them, or that they're testing on themselves. And particularly for LLM tools, there is this strong sense amongst many fans that they are The Future, that anyone who doesn't get onboard is being Left Behind, and so forth. I'd assume a lot of users aren't thinking particularly rationally about them.
  • alfalfasprout 1y ago
    I would speculate that it's because there's been a huge concerted effort to make people want to believe that these tools are better than they are. The "economic experts" and "ml experts" are in many cases effectively the same group-- companies pushing AI coding tools have a vested interest in people believing they're more useful than they are. Executives take this at face value and broadly promise major wins. Economic experts take this at face value and use this for their forecasts. This propagates further, and now novices and casual individuals begin to believe in the hype. Eventually, as an experienced engineer it moves the "baseline" expectation much higher. Unfortunately this is very difficult to capture empirically.
  • longwave 1y ago
    I also wonder how many of the numerous AI proponents in HN comments are subject to the same effect. Unless they are truly measuring their own performance, is AI really making them more productive?
• afro88 1y ago
  Early 2025. I imagine the results would be quite different with mid 2025 models and tools.
  • phyzome 1y ago
    If they used mid 2025 models and tools, the paper would have come out in late 2025, and you would have had the same complaint.
• gmaster1440 1y ago
  What if the slowdown isn't a bug but a feature? What if AI tools are forcing developers to think more carefully about their code, making them slower but potentially producing better results? AFAIK the study measured speed, not quality, maintainability, or correctness. The developers might feel more productive because they're engaging with their code at a higher level of abstraction, even if it takes longer. This would be consistent with why they maintained positive perceptions despite the slowdown.
  • PessimalDecimal 1y ago
    In my experience, LLMs are not causing people to think more carefully about their code.
• doctoboggan 1y ago
  For me, the measurable gain in productiviy comes when I am working with a new language or new technology. If I were to use claude code to help implement a feature of a python library I've worked on for years then I don't think it would help much (Maybe even hurt). However, if I use claude code on some go code I have very little experience with, or using it to write/modify helm charts then I can definitely say it speeds me up. But, taking a broader view its possible that these initial speed ups are negated by the fact that I never really learn go or helm charts as deeply now that I use claude code. Over time, its possible that my net productiviy is still reduced. Hard to say for sure, especially considering I might not have even considered talking these more difficult go library modifications if I didn't have claude code to hold my hand. Regardless, these tools are out there, increasing in effectiveness and I do feel like I need to jump on the train before it leaves me at the station.
  • mattxxx 1y ago
    agreed - it helps transpose skills. That said, this comes up often in my office. It's just not giving really good advice in many situations - especially novel ones. AI is super good at coming up with things that have been written ad nauseam for coding-interview-prep website
• LegNeato 1y ago
  For certain tasks it can speed me up 30x compared to an expert in the space: https://rust-gpu.github.io/blog/2025/06/24/vulkan-shader-port/ https://rust-gpu.github.io/blog/2025/06/24/vulkan-shader-por...
  • lpghatguy 1y ago
    This is very disingenuous: we don't know how much spare time Sascha spent, and much of that time was likely spent learning, experimenting, and reporting issues to Slang.
    • LegNeato 1y ago
      It's not disingenuous, we have his and my git commit logs. I also had to deal with issues with rust-gpu. He is an expert in both the space and his project, I had never seen his project nor written any graphics shaders before. You can never get apples to apples comparisons but this is the closest I have personally experienced.
• _jayhack_ 1y ago
  This does not take into account the fact that experienced developers working with AI have shifted into roles of management and triage, working on several tasks simultaneously. Would be interesting (and in fact necessary to derive conclusions from this study) to see aggregate number of tasks completed per developer with AI augmentation. That is, if time per task has gone up by 20% but we clear 2x as many tasks, that is a pretty important caveat to the results published here
• isoprophlex 1y ago
  Ed Zitron was 100% right. The mask is off and the AI subprime crisis is coming. Reading TFA, it would be hilarious if the bubble burst AND it turns out there's actually no value to be had, at ANY price. I for one can't wait for this era of hype to end. We'll see. you're addicted to the FEELING of productivity more than actual productivity. even knowing this, even seeing the data, even acknowledging the complete fuckery of it all, you're still gonna use me. i'm still gonna exist. you're all still gonna pretend this helps because the alternative is admitting you spent billions of dollars on spicy autocomplete.
• keerthiko 1y ago
  IME AI coding is excellent for one-off scripts, personal automation tooling (I iterate on a tool to scrape receipts and submit expenses for my specific needs) and generally stuff that can be run in environments where the creator and the end user are effectively the same (and only) entity. Scaled up slightly, we use it to build plenty of internal tooling in our video content production pipeline (syncing between encoding tools and a status dashboard for our non-technical content team). Using it for anything more than boilerplate code, well-defined but tedious refactors, or quickly demonstrating how to use an unfamiliar API in production code, before a human takes a full pass at everything is something I'm going to be wary of for a long time.
• mrwaffle 1y ago
  My overall concern has to do with our developer ecosystem from the important points mentioned by simonw and narush. I've been concerned about this for years but AI reliance seems to be pouring jet fuel on the fire. Particularly troubling is the lack of understanding less-experienced devs will have over time. Does anyone have a counter-argument for this they can share on why this is a good thing?
  • partdavid 1y ago
    The shallow analogy is like "why worry about not being able to do arithmetic without a calculator"? Like... the dev of the future just won't need it. I feel like programming has become increasingly specialized and even before AI tool explosion, it's way more possible to be ignorant of an enormous amount of "computing" than it used to be. I feel like a lot of "full stack" developers only understand things to the margin of their frameworks but above and below it they kind of barely know how a computer works or what different wire protocols actually are or what an OS might actually do at a lower level. Let alone the context in which in application sits beyond let's say, a level above a kubernetes pod and a kind of trial-end-error approach to poking at some YAML templates. Do we all need to know about processor architectures and microcode and L2 caches and paging and OS distributions and system software and installers and openssl engines and how to make sure you have the one that uses native instructions and TCP packets and envoy and controllers and raft systems and topic partitions and cloud IAM and CDN and DNS? Since that's not the case--nearly everyone has vast areas of ignorance yet still does a bunch of stuff--it's harder to sell the idea that whatever AI tools are doing that we lose skills in will somehow vaguely matter in the future. I kind of miss when you had to know a little of everything and it also seemed like "a little bit" was a bigger slice of what there was to know. Now you talk to people who use a different framework in your own language and you feel like you're talking to deep specialists whose concerns you can barely understand the existence of, let alone have an opinion on.
    • jpc0 1y ago
      > Do we all need to know about processor architectures and microcode and L2 caches and paging and OS distributions and system software… Have you used modern software… or just software in general to be honest. We have had orders of magnitude improvement in hardware performance and much fewer orders of magnitude increase in software performance and features. May I present the windows start menu as a perfect exhibit, we put a web browser in there and made actually finding the software you want to use harder than ever, even search is completely broken 99% of the time (really try powertoys run or even windows + s for a night and day difference). We add boundless complexity to things that doesn’t need it, millions of lines of code, then waste millions of cycles running security tools to heuristically prevent malicious actors from exploiting our millions of lines of code that is impossible to know because it is deemed to difficult to learn the underlying semantics of the problem domain.
• OpenSourceWard 1y ago
  Very cool work! And I love the nuance in your methodology and findings. Anyway, I'm preparing myself for all the "Bombshell news: AI is slowing down developers" posts that are coming.
  • mwigdahl 1y ago
    Plus the gaslighting to follow for anyone claiming AI improved their productivity.
    • phyzome 1y ago
      Well, it would be a nice counterweight to all the gaslighting of people who claim AI doesn't improve their productivity...
      • mwigdahl 1y ago
        I've never seen a post telling someone who claimed that AI didn't improve their productivity that they were mistaken and that actually it had. Plenty of "you're holding it wrong" and "it works for me", but not directly telling someone that their perceptions were inaccurate.
• asciimov 1y ago
  I'll be curious of the long term impacts of AI. Such as: do you end up spending more time to find and fix issues, does AI use reduce institutional knowledge, will you be more inclined to start projects over from scratch.
• lmeyerov 1y ago
  As someone has been doing hardcore genai for 2+ years, my experience has been, and what we advise internally: * 3 weeks to transition from ai pairing to AI Delegation to ai multitasking. So work gains are mostly week 3+. That's 120+ hours in, as someone pretty senior here. * Speedup is the wrong metric. Think throughput, not latency. Some finite amount of work might take longer, but the volume of work should go up because AI can do more on a task and diff tasks/projects in parallel. Both perspectives seem consistent with the paper description...
  • phyzome 1y ago
    Have you actually measured this? Because one of the big takeaways from this study is that people are bad at predicting and observing their own time spent.
    • lmeyerov 1y ago
      yes, I keep prompt plan logs At the same time... that's not why I'm comfortable writing this. It's pretty obvious when you know what good vs bad feels like here and adjust accordingly: 1. Good: You are able to generate a long plan and that plan mostly works. These are big wins _as long as you are multitasking_: you are high throughput, even if the AI is slow. Think running 5-20min at a time for pretty good progress, for just a few minutes of your planning that you'd largely have to do anyways. 2. Bad: You are wasting a lot of attention chatting (so 1-2min runs) and repairing (re-planning from the top, vs progressing). There is no multitasking win. It's pretty clear what situation you're in, with run duration on its own being a ~10X level difference. Ex: I'll have ~3 projects going at the same time, and/or whatever else I'm doing. I'm not interacting "much" so I know it's a win. If a project is requiring interaction, well, now I need to jump in, and it's no longer agentic coding IMO, but chat assistant stuff. At the same time, I power through case #2 in practice because we're investing in AI automation. We're retooling everything to enable long runs, so we'll still do the "hard" tasks via AI to identify & smooth the bumps. Similar to infrastructure-as-code and SDLC tooling, we're investing in automating as much of our stack as we can, so that means we figure out prompt templates, CI tooling, etc to enable the AI to do these so we can benefit later.
• ieie3366 1y ago
  LLMs are godtier if you know what you’re doing, and prompt them with ”do X”, where x is a SELF-CONTAINED change you would manually know how to implement For example, today I asked claude to implement per-user rate-limiting into my nestjs service, then iterated by asking implementing specific unit tests and some refactoring. It one-shot everything. I would say 90% time savings. Unskilled people ask them ”i have giant problem X solve it” and end up with slop
  • lloeki 1y ago
    I tried exactly that, several times, over and over. Except on "hello world" situations (which I guess is a solid part of the corpus LLMs are trained with) these tools were consistently slower. Last time was an area where several files were subtly different in a section that essentially does about the same thing, and needed to be aligned and made consistent†. Time to - begrudgingly - do it manually: 5min Time to come up with a one-shot shell incantation: 10min Time to very dumbly manually mark the areas with ===BEGIN=== and ===END=== and come up with a one-shot shell incantation: 3min Time to do it for the LLM: 45min††; also it required regular petting every 20ish command so zero chance of letting it run and doing something else†††. Time to review + manually fix the LLM output which missed two sections, left obsolete comments, and modified four files that were entirely unrelated yet clearly declared as out of scope in the prompt: 5min Consistently, proponents have been telling me "yeah you need to practice more, I'm getting fine results so you're holding it wrong, we can do a session together and I'll show you how to do it", which they do, and then it doesn't work, and they're like "well I'll look into it and circle back" and I never hear from them again. As for suggestions, for every good completion where I accept saying "oh well, why not", 99 get rejected: the majority are complete hallucinations absolutely unrelated to the surrounding logic, a third are either broken or introduce non-working code, and 1-5 _actively dangerous_ in some way. The only places where I found LLMs vaguely useful are: - Asking questions about an unknown codebase. It still hallucinates and misdirects or is excessively repetitive about some things (even with rules) but it can crudely draw a rough "map" and make non-obvious connections about two distant areas, which can be welcome. - Asking for a quick code review in addition to the one I ask to humans; 70% of such output is laughably useless (although harmless beyond the noise + energy cost), 30% is duplicate of human reviews but I can get it earlier, and sometimes it unearths a good point that has been overlooked. † No, the specific section cannot+should not be factored out †† And that's because I interrupted it because it was going about modifying files that it should not have. ††† A bit of a lie because I did the other three ways during that time. Which also is telling because the time to do the other ways would actually be _lower_ because I was interrupted by / had to keep tabs on what the AI agent was doing.
• thepasswordis 1y ago
  I actually think that pasting questions into chatGPT etc. and then getting general answers to put into your code is the way. “One shotting” apps, or even cursor and so forth seem like a waste of time. It feels like if you prompt it just right it might help but then it never really does.
  • partdavid 1y ago
    I've done okay with copilot as a very smart autocomplete on: a) very typical codebase, with b) lots of boilerplate, where c) I'm not terribly familiar with the languages and frameworks, which are d) very, very popular but e) I don't really like, so I'm not particularly motivated to become familiar with them. I'm not a frontend developer, I don't like it, but I'm in a position now where I need to do frontend things with a verbose Typescript/React application which is not interesting from a technical point of view (good product, it's just not good because it has an interesting or demanding front end). Copilot (I use Emacs, so cursor is a non-starter, but copilot-mode works very well for Typescript) has been pretty invaluable to just sort of slogging through stuff. For everything else, I think you're right, and actually the dialog-oriented method is way better. If I learn an approach and apply some general example from ChatGPT, but I do the typing and implementation myself so I need to understand what I'm doing, I'm actually leveling up and I know what I'm finished with. If I weren't "experienced", I'd worry about what it was doing to my critical thinking skills, but I know enough about learning on my own at this point to know I'm doing something. I'm not interested in vibe coding at all--it seems like a one-way process to automate what was already not the hard part of software engineering; generating tutorial-level initial implementations. Just more scaffolding that eventually needs to be cleared away.
• deleted 1y ago
  [deleted]
• thesz 1y ago
  What is interesting here is that all predictions were positive, but results are negative. This shows that everyone in the study (economic experts, ML experts and even developers themselves, even after getting experience) are novices if we look at them from the Dunning-Kruger effect [1] perspective. [1] https://en.wikipedia.org/wiki/Dunning%E2%80%93Kruger_effect https://en.wikipedia.org/wiki/Dunning%E2%80%93Kruger_effect "The Dunning–Kruger effect is a cognitive bias in which people with limited competence in a particular domain overestimate their abilities."
  • 59nadir 1y ago
    > "The Dunning–Kruger effect is a cognitive bias in which people with limited competence in a particular domain overestimate their abilities." No, they underestimated their own abilities for the most part; the estimates for AI-disallowed tasks were all undershot in terms of real implementation time. What they overestimated was the ability of LLMs to provide real productivity gains on a given task.
    • thesz 1y ago
      > What they overestimated was the ability of LLMs to provide real productivity gains on a given task. This is exactly my point. This is not about the ability of LLM overestimated by developers, this is about the ability of developer interacting with LLM overestimated by developers themselves, economic experts and ML experts. LLMs are not "able" per se, they are "prompted" to be "able." They are not agents, but behave as agents on someone's behalf - and no one have a clue whether use of LLMs is positive or detrimental, with the bias being "LLM's use is net positive". The overestimation of LLM's abilities by everyone calls for Dunning-Kruger.
• mattl 1y ago
  I don't understand how anyone doing open source can use something trained on other people's code as a tool for contributions. I wouldn't accept someone's copy and pasted code from another project if it were under an incompatible license, let alone something with unknown origin.
• AIorNot 1y ago
  Hey guys why are we making it so complicated? do we really need a paper and study? anyway -AI as the tech currently stand is a new skill to use and takes us humans time to learn, but once we do well, its becomes force multiplier ie see this: https://claude.ai/public/artifacts/221821f0-0677-409b-8294-3fd48d3ffcb8 https://claude.ai/public/artifacts/221821f0-0677-409b-8294-3...
  • lompad 1y ago
    Because for now, that's just what those financially profiting from the AI-hype tell us. Be it sama, hyung or nadella, they all profit if people _believe_ AI is a force multiplier for everybody. Reality is much more muddy though and it's absolutely not as obvious as those people claim. And keep in mind that a 5-10x price hike is to be expected if those companies keep spending billions to make millions. Right now, there is a consistent stream of papers incoming which indicates that AI might be much more of a specialized tool for very particular situations instead of the "solve everything"-tool the hype makes people believe. This is highly significant. "Just believe me bro" is just not enough.
• tarofchaos 1y ago
  Totally flawed study
  • phyzome 1y ago
    This comment isn't very useful if you don't actually point out some flaws. :-)
• bit1993 1y ago
  It used to be that all you required to program was a computer and to RTFM but now we need to pay for API "tokens" and pray that there are no rug pull in the future.
  • danparsonson 1y ago
    "It used to be that all you required to write was a pen and paper but now we need to pay for 'electricity'..." You can still do those things.
• cadamsdotcom 1y ago
  My hot take: Cursor is a bad tool for agentic coding. Had a subscription and canceled it in favor of Claude Code. I don’t want to spend 90% of my time approving every line the agent wants to write. With Claude Code I review whole diffs - 1-2 minutes of the agent’s work at a time. Then I work with the agent at a level of what its approach is, almost never asking about specific lines of code. I can look at 5 files at once in git diff and then ask “why’d you choose that way?” “Can we undo that and try to find a simpler way?” Cursor’s workflow exposes how differently different people track context. The best ways to work with Cursor may simply not work for some of us. If Cursor isn’t working for you, I strongly encourage you to try CLI agents like Claude Code.
• codyb 1y ago
  So slow until a learning curve is hit (or as one user posited "until you forget how to work without it"). But isn't the important thing to measure... how long does it take to debug the resulting code at 3AM when you get a PagerDuty alert? Similarly... how about the quality of this code over time? It's taken a lot of effort to bring some of the code bases I work in into a more portable, less coupled, more concise state through the hard work of - bringing shared business logic up into shared folders - working to ensure call chains flow top down towards root then back up through exposed APIs from other modules as opposed to criss-crossing through the directory structure - working to separate business logic from API logic from display logic - working to provide encapsulation through the use of wrapper functions creating portability - using techniques like dependency injection to decouple concepts allowing for easier testing etc So, do we end up with better code quality that ends up being more maintainable, extensible, portable, and composable? Or do we just end up with lots of poor quality code that eventually grows to become a tangled mess we spend 50% of our time fighting bugs on?
• solid_fuel 1y ago
  I would love to see a comparison of the pull requests generated by each workflow, if possible. My experience with Copilot has generally been that it suggests far more code than I would actually write to solve a specific problem - sometimes adding extra checks where they aren't needed, sometimes just being more verbose than I would be, and oftentimes repeating itself where it would be better to use an abstraction. My personal hypothesis is that seeing the LLM write _so much_ code may create the feeling that the problems it is solving would take longer to solve by yourself.
  • narush 1y ago
    Check out section AI increasing issue scope (C.2.3) in the paper -- https://metr.org/Early_2025_AI_Experienced_OS_Devs_Study.pdf https://metr.org/Early_2025_AI_Experienced_OS_Devs_Study.pdf We speak (the best we can) to changes in amount of code -- I'll note that this metric is quite messy and hard to reason about!
• budududuroiu 1y ago
  My theory is that, outside of programming skill, amazement by AI tools is inversely proportional to typing/navigating speed. I already know what I need to write, I just need to get it into the editor. I wouldn’t trade the precision I have with vim macros flying across multiple files for an AI workflow. I do think AI is a good rubber ducky sometimes tho, but I despise letting it take over editing files.
• Amaury-El 1y ago
  I've been using LLMs almost every day for the past year. They're definitely helpful for small tasks, but in real, complex projects, reviewing and fixing their output can sometimes take more time than just writing the code myself. We probably need a bit less wishful thinking. Blindly trusting what the AI suggests tends to backfire. The real challenge is figuring out where it actually helps, and where it quietly gets in the way.
• Leo-thorne 1y ago
  I really admire stories like this. Reaching $1M ARR without any funding is rare and feels real. It shows what building something truly takes. Late nights, tough moments, losing users. It's not about big bursts of growth but staying consistent, solving real problems, and growing revenue little by little. There's a lot to learn from that.
  • semireg 1y ago
    Wrong thread
• semireg 1y ago
  Something I don’t see mentioned that’s been helpful to me is having an agent add strict type safety to my typescript. I avoid the use of “any” type and berating an agent to “make it work” really opens my eyes and forces me to learn how advanced typescript can be leveraged. I feel that creating complex types that make my code simpler, makes autocomplete work(!), is a great tradeoff in some meta dimension of software dev.
• journal 1y ago
  would you be worse without it? now prepare to pay $1000+/month for chatgpt in a few years when dust settles.
• DrNosferatu 1y ago
  I would say AI has very different impacts on individual coding styles.
• ternaus 1y ago
  I guess, I am experienced open-source developer (https://github.com/albumentations-team/Albumentations https://github.com/albumentations-team/Albumentations) 15k stars, 5 million monthly downloads ---- It may happen that Cursor in the agentic mode writes code slower than I am. But! It frees me from being in the IDE 100% of the time. There is infinite list of educational videos, blog posts, scientific papers, hacker news, twitter, reddit that I want to read and going through them, while agents do their job is ultra convenient. => If I think about "productivity" in a broader way => with Cursor + agents, my overall productivity moved to a whole another level.
• iainctduncan 1y ago
  I believe the important and interesting take away of the paper of the paper is that so many developers thought they were going faster when they were actually going slower. This is especially striking when you consider that the study was not broad enought to capture long-term tech debt.
• ishita159 1y ago
  This was very surprising for me given all the noise.
• kgryte 1y ago
  Here's a reflection from one of the study's participants, which covers some of the common criticisms and where he see opportunities for further study. https://blog.stdlib.io/reflection-on-the-metr-study-2025/ https://blog.stdlib.io/reflection-on-the-metr-study-2025/
• GregDavidson 1y ago
  AI used to refer to the extensive range of techniques of the field of Artificial Intelligence. Now it refers to LLMs and maybe other multi-layer networks trained on vast datasets. LLMs are great for some tasks and are also great as parts of hybrid systems like the IBM Watson Jepardy system. There's much more to Artificial Intelligence, e.g. https://en.m.wikipedia.org/wiki/Knowledge_representation_and_reasoning https://en.m.wikipedia.org/wiki/Knowledge_representation_and... et al.
• smcleod 1y ago
  Claude 4 Opus when provided with the full paper, blog post and asked to perform a critical review: The methodology contains several fundamental flaws that likely explain this anomalous result. Most critically, the study examines a highly specific scenario - expert developers working on codebases they've contributed to for years (averaging 1,500 commits over 5 years) - which creates a ceiling effect where AI has minimal room to provide value. The 30-minute Cursor training for developers, 56% of whom had never used the tool before, is woefully inadequate for learning effective AI pair programming techniques. With only 16 participants and a non-blinded design where developers knew their condition and were paid $150/hour, the study lacks both statistical power and ecological validity. The restriction to a single tool configuration (Cursor Pro with specific Claude models) and acknowledgment that the tool doesn't optimise token sampling or prompting strategies further limits the findings' applicability to the broader question of AI's impact on developer productivity. - Concerningly small sample size: Only 16 developers across 246 tasks provides insufficient statistical power for broad generalisations about AI's impact on millions of developers - Inadequate AI training: 30-minute basic Cursor tutorial for developers where 56% had never used the tool - completely insufficient for developing effective AI collaboration skills - Selection bias towards ceiling effects: Developers averaged 5 years and 1,500 commits on their repositories, creating an expertise level where AI assistance has minimal value-add potential - Single tool learning: Study restricted to education Cursor Pro with specific Claude models, not representative of the diverse AI tooling ecosystem (Windsurf, Cline, Roo Code etc.) - Artificial task constraints: Tasks were acknowledged as "shorter than average" and broken into ≤2 hour chunks, not representative of real development work - No experimental blinding: Developers knew their condition and were being observed/recorded, potentially affecting natural work patterns - Suboptimal AI usage: Study acknowledges Cursor doesn't sample sufficient tokens and developers reported overusing AI due to experimental conditions - Narrow context: All repositories were large (1.1M LoC average), mature (10 years old), with high quality standards - a specific niche not representative of most development - Self-reported metrics: Time tracking was self-reported with only 29% verified through screen recordings, introducing measurement bias - Expertise mismatch: Focusing on experts contradicts established findings that AI tools provide greater benefits to less experienced developers - Tool proficiency confound: No control for varying levels of AI tool proficiency or different prompting strategies between participants - Limited generalisability: The specific combination of expert developers + familiar codebases + large repositories + short tasks creates an artificial scenario unlike typical development workflows