6 ms·

How do you guys review AI-generated code ? In our team, frontend work is vibe-coded by the PO and merged as-is without review. Backend is coded by developers,

How do you guys review AI-generated code ? In our team, frontend work is vibe-coded by the PO and merged as-is without review. Backend is coded by developers, using AI but in a slower, more controlled way. Recently, our PO has been trying his hand at vibe-coding the backend. I must say he is a smart guy, almost technical but not quite a developer. We've just been handed a burst of stacked PRs amounting for ~15k LOC backend. We do not quite know what do to about it. I know we are not the only ones in the situation. What's your experience and context ? What do you do ? What works for you what doesn't ?

oscarestudillom 7d ago
we are using Revix AI, works really good on repos that already have some standards and patterns from good devs. the reviewer catches most of the things so when the senior reviews he just have to focus on more elevated things like architecture, etc. moreover, try to enforce having AGENTS.md files on your repos, and rules created by the senior devs specific to your repo, its not perfect but also helps quite a lot
mobelkh 11d ago
throw his garbage out, the time and effort taken to review that is magnitudes more than what it took to prompt it. have him start with an overall design doc if his change is 15k, it's definitely worth a design doc. and then have his contributions reviewed in pieces of 200-300 LoC PRs. any other solution is trading stability and system knowledge, that's 15k LoC no one is truly familiar with, even if you do try to review it
- embedding-shape 11d ago
  Yeah, it's the "eager apprentice" problem, common almost everywhere. Solution is to make them stop and double-check before running ahead, in software development, concise design documents outlining what the problem is, what possible solutions are and what the chosen solution is, and why, then review this together with the person, before they can move on to implement it.
  - sdeframond 11d ago
    This particular apprentice is also my boss, an overall reasonable guy and has more experience in the software industry than myself, so there's that. He's just not a developer.
- sdeframond 11d ago
  He's made a bunch of 1-2k LOC PRs and there is a design doc. Everything is AI generated. The issue is, if he generates all of that without reviewing the code, he will always be far faster than us. And he can't review the code. No matter how he slices it. Also, he is the CPO/CTO. So we can say no, but there is a natural incentive to go his way. He still doesn't feel confident enough to just bypass the programmers and he's probably right. But it'd nice to find a way to use my expertise to review this amount of code meaningfully, somehow.
bushbaba 11d ago
I have AI confirm the logic works as expected, but review for system design. Often in both web/backend I’ve found AI to produce overly duplicative code, or have aspects that could be hard to maintain. Generally less due to the AI, and more because of the prompt itself. That and even if you’re going to AI slop it up, I’d still demand it be broken up into 1-2k LOC chunks or per meaningful “thing”. This also lets us gradually ramp the change to confirm it actually works earlier on
yomismoaqui 11d ago
Invest in having a good test suite that validates the functionality introduced by that code. Also AI can review code in an adversarial way and apply those fixes (that ideally will keep the previous tests you did on green)
fabianlindfors 11d ago
We try to avoid reviewing AI-generated code and built our own testing framework and platform to make that possible. Our principle is that our tests should give us enough confidence to not have to look at the code (which ends up being true for most changes we make). The core thing that makes this possible is that we run our entire code and infra (including fakes for external dependencies) in isolated, forkable environments and write tests against that, so they are as E2E as can possibly be. The problem then shifts from reviewing code to reviewing tests and that's why we built our own platform. We have a UI that can diff tests, so we know what changed, and a visual way to inspect what the tests actually did. A test could drive a browser like a user would, and in our UI we get a replay of that browser interaction to look at. The browser is talking to a real version of our backend, and the tests can perform assertions against the database and fakes and really anything in our system.
- eterps 11d ago
  But with full blown e2e browser tests the test suite duration can go through the roof. How do you deal with that?
  - fabianlindfors 11d ago
    Forking! We run the entire stack (browser, frontend, backend, database, etc) in a Linux VM, so latency between each of the pieces is as tiny as can be. This is quite different from "standard" E2E tests I've seen where the test browsers uses something like a persistent staging environment. The real key is that we can fork that entire Linux VM to take different paths down our testing scenarios, and can run multiple of them in parallel. Tests may look something like: new user signs up: |- creates a todo |- ... |- ... |- creates a list The two nested tests then start from the exact same point, where the previous test left off, but can run in parallel. With enough hardware, the full suite will run as fast as the slowest branch of the test tree. When we switched away from our previous integration test suite to this (not E2E), our tests actually became faster because they share setup through the forking.
    - 11d ago
ninkendo 11d ago
The answer to this is gonna vary wildly depending on what kind of codebase it is. A large, mature codebase that predates LLM’s and for which changes need a high level of scrutiny regardless of who made them (think llvm, WebKit, important foundational software), you’re going to want humans in the loop as much as ever… I think reviewing LLM output is the most important thing a human can provide. But for vibe coded apps where you can just one-shot another one if anything goes wrong? Just vibe the reviews too, who cares. Let the robots review the robots. Be careful with doing AI review if your codebase is in the former category. Or your codebase will quickly turn into the latter. Complete with “you can just one-shot another one”, because if nobody understands the code any more, there’s not much lost by just you (or your competitors, etc) replacing it wholesale with an AI-written alternative. I struggle with this a lot. 2 years ago we had a half dozen PR’s a day with a lot of careful review, and now there’s more like 30 of them per day and most people are just rubber stamping them after the AI reviews it. I’m still fighting the good fight trying to review every line of the PR’s I have time to look at, but that constitutes maybe 10% of them. Not only am I barely making a dent, but it’s awkward when I post nitpicks like “this function should go in this module”, etc, the author usually looks at me funny like “why are you even reading this”. Our codebase is gradually becoming more and more vibe coded, and it’s depressing me.
CuriouslyC 11d ago
Code review is soon to be an outmoded concept, (un)fortunately. You have to design orthogonal code (e.g. independent modules in a modular monolith, or microservices) and soak test using canaries.
- sdeframond 9d ago
  No code is truly orthogonal if we want it to interact in some way. One microservice might DoS antoher one. In a monolith, some process may take up all resources, and so on.
myappengineer 9d ago
[flagged]

oscarestudillom 7d ago

we are using Revix AI, works really good on repos that already have some standards and patterns from good devs. the reviewer catches most of the things so when the senior reviews he just have to focus on more elevated things like architecture, etc. moreover, try to enforce having AGENTS.md files on your repos, and rules created by the senior devs specific to your repo, its not perfect but also helps quite a lot

mobelkh 11d ago

throw his garbage out, the time and effort taken to review that is magnitudes more than what it took to prompt it. have him start with an overall design doc if his change is 15k, it's definitely worth a design doc. and then have his contributions reviewed in pieces of 200-300 LoC PRs. any other solution is trading stability and system knowledge, that's 15k LoC no one is truly familiar with, even if you do try to review it

embedding-shape 11d ago
Yeah, it's the "eager apprentice" problem, common almost everywhere. Solution is to make them stop and double-check before running ahead, in software development, concise design documents outlining what the problem is, what possible solutions are and what the chosen solution is, and why, then review this together with the person, before they can move on to implement it.
- sdeframond 11d ago
  This particular apprentice is also my boss, an overall reasonable guy and has more experience in the software industry than myself, so there's that. He's just not a developer.
sdeframond 11d ago
He's made a bunch of 1-2k LOC PRs and there is a design doc. Everything is AI generated. The issue is, if he generates all of that without reviewing the code, he will always be far faster than us. And he can't review the code. No matter how he slices it. Also, he is the CPO/CTO. So we can say no, but there is a natural incentive to go his way. He still doesn't feel confident enough to just bypass the programmers and he's probably right. But it'd nice to find a way to use my expertise to review this amount of code meaningfully, somehow.

bushbaba 11d ago

I have AI confirm the logic works as expected, but review for system design. Often in both web/backend I’ve found AI to produce overly duplicative code, or have aspects that could be hard to maintain. Generally less due to the AI, and more because of the prompt itself. That and even if you’re going to AI slop it up, I’d still demand it be broken up into 1-2k LOC chunks or per meaningful “thing”. This also lets us gradually ramp the change to confirm it actually works earlier on

yomismoaqui 11d ago

Invest in having a good test suite that validates the functionality introduced by that code. Also AI can review code in an adversarial way and apply those fixes (that ideally will keep the previous tests you did on green)

fabianlindfors 11d ago

We try to avoid reviewing AI-generated code and built our own testing framework and platform to make that possible. Our principle is that our tests should give us enough confidence to not have to look at the code (which ends up being true for most changes we make). The core thing that makes this possible is that we run our entire code and infra (including fakes for external dependencies) in isolated, forkable environments and write tests against that, so they are as E2E as can possibly be. The problem then shifts from reviewing code to reviewing tests and that's why we built our own platform. We have a UI that can diff tests, so we know what changed, and a visual way to inspect what the tests actually did. A test could drive a browser like a user would, and in our UI we get a replay of that browser interaction to look at. The browser is talking to a real version of our backend, and the tests can perform assertions against the database and fakes and really anything in our system.

eterps 11d ago
But with full blown e2e browser tests the test suite duration can go through the roof. How do you deal with that?
- fabianlindfors 11d ago
  Forking! We run the entire stack (browser, frontend, backend, database, etc) in a Linux VM, so latency between each of the pieces is as tiny as can be. This is quite different from "standard" E2E tests I've seen where the test browsers uses something like a persistent staging environment. The real key is that we can fork that entire Linux VM to take different paths down our testing scenarios, and can run multiple of them in parallel. Tests may look something like: new user signs up: |- creates a todo |- ... |- ... |- creates a list The two nested tests then start from the exact same point, where the previous test left off, but can run in parallel. With enough hardware, the full suite will run as fast as the slowest branch of the test tree. When we switched away from our previous integration test suite to this (not E2E), our tests actually became faster because they share setup through the forking.
  - 11d ago

ninkendo 11d ago

The answer to this is gonna vary wildly depending on what kind of codebase it is. A large, mature codebase that predates LLM’s and for which changes need a high level of scrutiny regardless of who made them (think llvm, WebKit, important foundational software), you’re going to want humans in the loop as much as ever… I think reviewing LLM output is the most important thing a human can provide. But for vibe coded apps where you can just one-shot another one if anything goes wrong? Just vibe the reviews too, who cares. Let the robots review the robots. Be careful with doing AI review if your codebase is in the former category. Or your codebase will quickly turn into the latter. Complete with “you can just one-shot another one”, because if nobody understands the code any more, there’s not much lost by just you (or your competitors, etc) replacing it wholesale with an AI-written alternative. I struggle with this a lot. 2 years ago we had a half dozen PR’s a day with a lot of careful review, and now there’s more like 30 of them per day and most people are just rubber stamping them after the AI reviews it. I’m still fighting the good fight trying to review every line of the PR’s I have time to look at, but that constitutes maybe 10% of them. Not only am I barely making a dent, but it’s awkward when I post nitpicks like “this function should go in this module”, etc, the author usually looks at me funny like “why are you even reading this”. Our codebase is gradually becoming more and more vibe coded, and it’s depressing me.

CuriouslyC 11d ago

Code review is soon to be an outmoded concept, (un)fortunately. You have to design orthogonal code (e.g. independent modules in a modular monolith, or microservices) and soak test using canaries.

sdeframond 9d ago
No code is truly orthogonal if we want it to interact in some way. One microservice might DoS antoher one. In a monolith, some process may take up all resources, and so on.

myappengineer 9d ago

[flagged]