Ianjit

9 ms·

1.
by Ianjit 8mo ago
I guess it all comes down to what a meaningful gain is. I agree that 10-30% is meaningful and if “software is a gas” this will lead to more software. But my expectations had become anchored to the frontier labs marketing (10x), and in that
2.
by Ianjit 8mo ago
I don’t work in SWE so I am just reacting to the claims that LLMs 10x productivity and are leading to mass layoff in the industry. In that context the 6-12% productivity gain at a company “all in” on AI didn’t seem impressive. LMMs can be a
3.
by Ianjit 8mo ago
A few studies over different time frames: [1] https://www.youtube.com/watch?v=1OzxYK2-qsI [2] https://www.coderabbit.ai/blog/state-of-ai-vs-human-code-gen... [3] https://www.youtube.com&#x
4.
by Ianjit 8mo ago
Sorry for the late response! My guess is they didn't publish it because they only measured it at one company, if they had the data across the cohort they would have published. The general result that review/re-wrok can cancel out
5.
by Ianjit 8mo ago
Studies have shown that software engineers are very bad at judging their own productivity. When a software engineer feels more productive the inverse is just as likely to be true. Thats why anecdotal data can't be trusted.
6.
by Ianjit 8mo ago
Please provide links to the studies, I am genuinely curious. I have been looking for data but most studies I find showing an uplift are just looking at LOC or PRs, which of course is nonsense. Meta measured a 6-12% uplift in productivity fr
7.
by Ianjit 8mo ago
I think an OpenAI paper showed 25% of GPT usage is “seeking information”. In that case Google also has a an advantage from being the default search provider on iOS and Android. I do find myself using the address bar in a browser like a chat
8.
by Ianjit 8mo ago
The productivity studies on software engineers directly don't show much of a productivity gain certainly nowhere near the 10x the frontier labs would like to claim. When including re-work of bugs in the AI generated code some studies f
9.
by Ianjit 8mo ago
"History doesn't repeat itself, but it often rhymes", except in the world of computer science where history does repeat.
10.
by Ianjit 8mo ago
Radiology has proven to be one of the most defensive jobs in medicine, radiologists beat AI once already! https://www.worksinprogress.news/p/why-ai-isnt-replacing-rad...
11.
by Ianjit 9mo ago
My bad. What was the result when they measured productivity after rework across the entire co hort?
12.
by Ianjit 9mo ago
I think you are quoting productivity measured before checking the code actually works and correcting it. After re-work productivity drops to 1%. Tinestamp 14:04.
13.
by Ianjit 9mo ago
This study does a good job of measuring the productivity impact. It found 1% uplift in dev productivity from using AI. https://youtu.be/JvosMkuNxF8?si=J9qCjE-RvfU6qoU0
14.
by Ianjit 9mo ago
This Stanford study on developer productivity found 0 correlation between developers assessment of their own productivity and independent measures of their productivity. Any anecdotal evidence from developers on how AI has made them more or
15.
by Ianjit 9mo ago
Isn’t this what Tao is addressing in the link, that LLMs haven’t encoded reasoning? Success in IMO is misleading because they are synthetic problems with known solutions that are subject to contamination (answers to similar questions are av
16.
by Ianjit 9mo ago
I think people make comments on LLMs not being smart in reaction to the comments from the leaders of AI labs that LLMs are so smart they could/will lead to mass unemployment.
17.
by Ianjit 9mo ago
Are SWE’s really experiencing a productivity uplift? When studies attempt to measure the productivity impact of AI in software the results I have seen are underwhelming compared to the frontier labs marketing.
18.
by Ianjit 9mo ago
My impression from Tao’s answers is that there is a gulf between the reality of what mathematicians actually do and the benchmarks that the frontier labs tout. It doesn’t seem trivial to close that gap and create an LLM that can replace mat
19.
Terry Tao: "LLMs are simpler than you think"(youtube.com)
15 points by Ianjit 9mo ago|9 comments
20.
by Ianjit 9mo ago
Meta internal study showed a 6-12% productivity uplift. https://youtu.be/1OzxYK2-qsI?si=8Tew5BPhV2LhtOg0
21.
by Ianjit 9mo ago
The productivity uplift is massive, Meta got a 6-12% productivity uplift from AI coding! https://youtu.be/1OzxYK2-qsI?si=8Tew5BPhV2LhtOg0
22.
by Ianjit 9mo ago
The studies I have seen show that AI written software is 70% more buggy than human written code [1]. I am curious where you get your data on AI code having “a similar reliability to accepted software”? [1] https://www.coderabbit.
23.
by Ianjit 9mo ago
Maybe you are right, but maybe it’s radiology all over again.
24.
by Ianjit 9mo ago
Why assume the breakdown between benchmarks and RoI is due to humans? The map is not the territory, the benchmark is not reality, there world is more complex than computer scientists understand.
25.
by Ianjit 9mo ago
Good deal for Donald Trump Jr.
26.
by Ianjit 9mo ago
Superficially it looks like a lot of these AI features are available elsewhere, what differentiates Aluo for e-commerce?
27.
Measuring the impact of AI on developer productivity at Meta(youtube.com)
2 points by Ianjit 9mo ago|1 comments
28.
by Ianjit 9mo ago
Meta engineers who were above median users of agentic coding saw a 6-12% increase in throughput.
29.
by Ianjit 9mo ago
Is AI really topping the charts in country music? https://youtu.be/rGremoYVMPc?si=EXrmyGltrvo2Ps8E
30.
by Ianjit 11mo ago
Forever, tech has become a never ending sequence of bubbles and hype cycles.

More ›