6 ms·

There was a study recently that showed how not only did devs overestimate the time saved using AI, but that they were net negative compared to the control grou

There was a study recently that showed how not only did devs overestimate the time saved using AI, but that they were net negative compared to the control group. Anyway, that about sums up my experience with AI. It may save some time here and there, but on net, you’re better off without it.

keeda 1y ago
That study gets mentioned all the time, somehow this one and many of the others it cites don't get much airtime: https://www.stlouisfed.org/on-the-economy/2024/sep/rapid-adoption-generative-ai https://www.stlouisfed.org/on-the-economy/2024/sep/rapid-ado... >This implies that each hour spent using genAI increases the worker’s productivity for that hour by 33%. This is similar in magnitude to the average productivity gain of 27% from several randomized experiments of genAI usage (Cui et al., 2024; Dell’Acqua et al., 2023; Noy and Zhang, 2023; Peng et al., 2023) Our estimated aggregate productivity gain from genAI (1.1%) exceeds the 0.7% estimate by Acemoglu (2024) based on a similar framework. To be clear, they are surmising that GenAI is already having a productivity gain.
- agent_turtle 1y ago
  The article you gave is derived from a poll, not a study. As for the quote, I can’t find it in the article. Can you point me to it? I did click on one of the studies and it indicated productivity gains specifically on writing tasks. Which reminded me of this recent BBC article about a copywriter making bank fixing expensive mistakes caused by AI: https://www.bbc.com/news/articles/cyvm1dyp9v2o https://www.bbc.com/news/articles/cyvm1dyp9v2o
  - keeda 1y ago
    The quote is from the paper linked in the article: https://s3.amazonaws.com/real.stlouisfed.org/wp/2024/2024-027.pdf https://s3.amazonaws.com/real.stlouisfed.org/wp/2024/2024-02... It's actually based on the results of three surveys conducted by two different parties. While surveys are subject to all kinds of biases and the gains are self-reported, their findings of 25% - 33% producitivity do match the gains shown by at least 3 other randomized studies, one of which was specifically about programming. Those studies are worth looking at as well.
    - foolswisdom 1y ago
      It's worth noting that the METR paper that found decreased productivity also found that many of the developers thought the work was being sped up.
      - keeda 1y ago
        Yes, self-reporting has biases and estimating tasks is still a fool's errand, which is why I noted that the estimates from these surveys matched the findings from other RTC studies. However, what doesn't get discussed enough about the METR study is that there was a spike in overall idle time as they waited for the AI to finish. I haven't run the numbers so I don't know how much of the increased completion time it accounts for, but if your cognitive load drops almost to 0, it will of course feel like your work is sped up, even though calendar time has increased. I wonder if that is the more important finding of that paper.
rockemsockem 1y ago
I'm not talking about time saving. AI seems to speed up my searching a bit since I can get results quicker without having to find the right query then find a site that actually answers my question, but that's minor, as nice as it is. I use AI in my personal life to learn about things I never would have without it because it makes the cost of finding any basic knowledge basically 0. Diet improvement ideas based on several quick questions about gut functioning, etc, recently learning how to gauge tsunami severity, and tons of other things. Once you have several fundamental terms and phrases for new topics it's easy to then validate the information with some quick googling too. How much have you actually tried using LLMs and did you just use normal chat or some big grand complex tool? I mostly just use chat and prefer to enter my code in artisanally.
- fzeroracer 1y ago
  Why not just look up the information directly instead of asking a machine that you can never truly validate? If I need information, I can just keyword search wikipedia, then follow the chain there and then validate the sources along with outside information. An LLM would actually cost me time because I would still need to do all of the above anyways, making it a meaningless step. If you don't do the above then it's 'cheaper' but you're implicitly trusting the lying machine to not lie to you.
  - rockemsockem 1y ago
    See my previous statement > Once you have several fundamental terms and phrases for new topics it's easy to then validate the information with some quick googling too. You're practically saying that looking at an index in the back of a book is a meaningless step. It is significantly faster, so much so that I am able to ask it things that would have taken an indeterminate amount of time to research before, for just simple information, not deep understanding. Edit: Also I can truly validate literally any piece of information it gives me. Like I said previously, it makes it very easy to validate via Wikipedia or other places with the right terms, which I may not have known ahead of time.
    - 1y ago
ares623 1y ago
Is the "here and there" tasks that were previously so little value that they are always stuck in the backlog? i.e. the parts where it helps have very little value in the first place.

keeda 1y ago

That study gets mentioned all the time, somehow this one and many of the others it cites don't get much airtime: https://www.stlouisfed.org/on-the-economy/2024/sep/rapid-adoption-generative-ai https://www.stlouisfed.org/on-the-economy/2024/sep/rapid-ado... >This implies that each hour spent using genAI increases the worker’s productivity for that hour by 33%. This is similar in magnitude to the average productivity gain of 27% from several randomized experiments of genAI usage (Cui et al., 2024; Dell’Acqua et al., 2023; Noy and Zhang, 2023; Peng et al., 2023) Our estimated aggregate productivity gain from genAI (1.1%) exceeds the 0.7% estimate by Acemoglu (2024) based on a similar framework. To be clear, they are surmising that GenAI is already having a productivity gain.

agent_turtle 1y ago
The article you gave is derived from a poll, not a study. As for the quote, I can’t find it in the article. Can you point me to it? I did click on one of the studies and it indicated productivity gains specifically on writing tasks. Which reminded me of this recent BBC article about a copywriter making bank fixing expensive mistakes caused by AI: https://www.bbc.com/news/articles/cyvm1dyp9v2o https://www.bbc.com/news/articles/cyvm1dyp9v2o
- keeda 1y ago
  The quote is from the paper linked in the article: https://s3.amazonaws.com/real.stlouisfed.org/wp/2024/2024-027.pdf https://s3.amazonaws.com/real.stlouisfed.org/wp/2024/2024-02... It's actually based on the results of three surveys conducted by two different parties. While surveys are subject to all kinds of biases and the gains are self-reported, their findings of 25% - 33% producitivity do match the gains shown by at least 3 other randomized studies, one of which was specifically about programming. Those studies are worth looking at as well.
  - foolswisdom 1y ago
    It's worth noting that the METR paper that found decreased productivity also found that many of the developers thought the work was being sped up.
    - keeda 1y ago
      Yes, self-reporting has biases and estimating tasks is still a fool's errand, which is why I noted that the estimates from these surveys matched the findings from other RTC studies. However, what doesn't get discussed enough about the METR study is that there was a spike in overall idle time as they waited for the AI to finish. I haven't run the numbers so I don't know how much of the increased completion time it accounts for, but if your cognitive load drops almost to 0, it will of course feel like your work is sped up, even though calendar time has increased. I wonder if that is the more important finding of that paper.

rockemsockem 1y ago

I'm not talking about time saving. AI seems to speed up my searching a bit since I can get results quicker without having to find the right query then find a site that actually answers my question, but that's minor, as nice as it is. I use AI in my personal life to learn about things I never would have without it because it makes the cost of finding any basic knowledge basically 0. Diet improvement ideas based on several quick questions about gut functioning, etc, recently learning how to gauge tsunami severity, and tons of other things. Once you have several fundamental terms and phrases for new topics it's easy to then validate the information with some quick googling too. How much have you actually tried using LLMs and did you just use normal chat or some big grand complex tool? I mostly just use chat and prefer to enter my code in artisanally.

fzeroracer 1y ago
Why not just look up the information directly instead of asking a machine that you can never truly validate? If I need information, I can just keyword search wikipedia, then follow the chain there and then validate the sources along with outside information. An LLM would actually cost me time because I would still need to do all of the above anyways, making it a meaningless step. If you don't do the above then it's 'cheaper' but you're implicitly trusting the lying machine to not lie to you.
- rockemsockem 1y ago
  See my previous statement > Once you have several fundamental terms and phrases for new topics it's easy to then validate the information with some quick googling too. You're practically saying that looking at an index in the back of a book is a meaningless step. It is significantly faster, so much so that I am able to ask it things that would have taken an indeterminate amount of time to research before, for just simple information, not deep understanding. Edit: Also I can truly validate literally any piece of information it gives me. Like I said previously, it makes it very easy to validate via Wikipedia or other places with the right terms, which I may not have known ahead of time.
  - 1y ago

ares623 1y ago

Is the "here and there" tasks that were previously so little value that they are always stuck in the backlog? i.e. the parts where it helps have very little value in the first place.