6 ms·

Probably averaging 70. 9 devs and 1 QA engineer.

hansvm 4d ago
Assuming the LLM never got anything wrong or otherwise had to be re-prompted, that means your devs were reviewing 130 SLOC per hour, on what was described as moderately greenfield (examining new implementations rather than comparing to old historical accidents). How? I don't want to sound flippant, but if the point is to add human thought to the mix, that's a high review rate even when examining small tweaks to an existing, working product, even with substantial AI help to pre-filter major gotchas before you bother spending a lot of human effort on the review. That's only 20-30wpm, but a review isn't just scanning or reading code, especially if you're trying to figure out how a new system which doesn't run yet will fit together.
- skrtskrt 4d ago
  No don’t worry they said it’s not slop because there’s tests lol
- kalaksi 4d ago
  Is that actually a high review rate? Especially if you know the language and domain. Sure, initially there's a learning curve for a new codebase structure, but lots of lines will also be trivial and many changes might also be similar to each other.
  - IsTom 4d ago
    For small separate changes in isolation then maybe it's ok? But not for whole days 8 hours each. But then you need to watch for bugs coming from interaction with previous changes and in 700k loc that might be nontrivial. How do you know which states are reachable and which are not? That takes time. It only takes a botched condition here (forgot a "not"? swapped "and"/"or"?), a swapped variable name there, code that looks ok, but isn't.
- lelanthran 4d ago
  > Assuming the LLM never got anything wrong or otherwise had to be re-prompted, that means your devs were reviewing 130 SLOC per hour, It's possible, I think, but humans cannot do repetitive mental work (like reviewing code) constantly without atrophy. Can I review 130 SLoC/hour? Sure. Can I do it 80 hours a week? No. Not even a single 8-hour workday.
  - hansvm 3d ago
    The fatigue aspect matters too, but another important point I was trying to get across is that not all reviews are created equally. In a working, well tested, properly factored system, with a high quality PR, I can do a cursory, local analysis of the new code and be very confident that it works. AI code I've seen is ... not that. I wouldn't be comfortable in that review without manually checking preconditions and postconditions, defining invariants, examining lifetimes, and a host of other activities. Moreover, for a greenfield replacement of lots of SLOC, I'd expect that most of the code for the early PRs can't even meaningfully run or do anything important, making any sort of broader architectural analysis impossible. I could see a world potentially where they came up with a magic prompt allowing each proposed PR to be cohesive, shippable, well factored, and everything else you need to be able to actually review it at a higher level and be comfortable with the results, but I'm skeptical. That's a major innovation if they managed to do so even as a one-off, and that wasn't the thing they highlighted when talking about the project.

Assuming the LLM never got anything wrong or otherwise had to be re-prompted, that means your devs were reviewing 130 SLOC per hour, on what was described as moderately greenfield (examining new implementations rather than comparing to old historical accidents). How? I don't want to sound flippant, but if the point is to add human thought to the mix, that's a high review rate even when examining small tweaks to an existing, working product, even with substantial AI help to pre-filter major gotchas before you bother spending a lot of human effort on the review. That's only 20-30wpm, but a review isn't just scanning or reading code, especially if you're trying to figure out how a new system which doesn't run yet will fit together.

skrtskrt 4d ago
No don’t worry they said it’s not slop because there’s tests lol
kalaksi 4d ago
Is that actually a high review rate? Especially if you know the language and domain. Sure, initially there's a learning curve for a new codebase structure, but lots of lines will also be trivial and many changes might also be similar to each other.
- IsTom 4d ago
  For small separate changes in isolation then maybe it's ok? But not for whole days 8 hours each. But then you need to watch for bugs coming from interaction with previous changes and in 700k loc that might be nontrivial. How do you know which states are reachable and which are not? That takes time. It only takes a botched condition here (forgot a "not"? swapped "and"/"or"?), a swapped variable name there, code that looks ok, but isn't.
lelanthran 4d ago
> Assuming the LLM never got anything wrong or otherwise had to be re-prompted, that means your devs were reviewing 130 SLOC per hour, It's possible, I think, but humans cannot do repetitive mental work (like reviewing code) constantly without atrophy. Can I review 130 SLoC/hour? Sure. Can I do it 80 hours a week? No. Not even a single 8-hour workday.
- hansvm 3d ago
  The fatigue aspect matters too, but another important point I was trying to get across is that not all reviews are created equally. In a working, well tested, properly factored system, with a high quality PR, I can do a cursory, local analysis of the new code and be very confident that it works. AI code I've seen is ... not that. I wouldn't be comfortable in that review without manually checking preconditions and postconditions, defining invariants, examining lifetimes, and a host of other activities. Moreover, for a greenfield replacement of lots of SLOC, I'd expect that most of the code for the early PRs can't even meaningfully run or do anything important, making any sort of broader architectural analysis impossible. I could see a world potentially where they came up with a magic prompt allowing each proposed PR to be cohesive, shippable, well factored, and everything else you need to be able to actually review it at a higher level and be comfortable with the results, but I'm skeptical. That's a major innovation if they managed to do so even as a one-off, and that wasn't the thing they highlighted when talking about the project.