10 ms·

From the blog post: > We observe that with AI-based suggestions, the code author increasingly becomes a reviewer, and it is important to find a balance between

From the blog post: > We observe that with AI-based suggestions, the code author increasingly becomes a reviewer, and it is important to find a balance between the cost of review and added value. We typically address the tradeoff with acceptance rate targets. In the past year since GPT-4 came out, I've also found this to be the case. I'm an ML/backend engineer with little experience in frontend development. Yet, I've been able to generate React UIs and Python UIs with GPT-4 in a matter of minutes and simply review the code to understand how it works. I find this to be very useful!

jprete 2y ago
This isn't a good trend. Reviewers never have the depth of knowledge or understanding of authors.
- bdjsiqoocwk 2y ago
  Also, people shouldn't be allowed to use computers unless they understand how transistors work. If you don't have the depth of knowledge you get nothing.
  - desolved 2y ago
    It took me way too long to realize this was a joke.
    - notpachet 2y ago
      It's HN so you can't be entirely sure.
  - BOOSTERHIDROGEN 2y ago
    Although this is well-expressed, I'm left with an unexplainable feeling that I can't quite put my finger on.
    - bdjsiqoocwk 2y ago
      Let me try and articulate it. The person Im responding to was gatekeeping. I responded by sarcastically doing the same to an extreme degree. A lot of people Will have agreed with the person i'm responding to. "Oh yeah of course You should understand these things, the things that I already understand", genuinely not realizing that there's no basis for that. When they reae my response they realize what they were doing, and are less feeling embarrassed for their senseless (and pretentious!) gatekeeping.
- refulgentis 2y ago
  Source? This is unintuitive to me, I can't come up with a rationale.
  - jprete 2y ago
    I know of no review process that produces the same level of understanding as does authorship, because the author must build the model from scratch and so must see all the details, while the reviewer is able to do less work because they're fundamentally riding on the author's understanding. In fact, in a high-trust system, e.g. a good engineering culture in a tech company, the reviewer will learn even less, because they won't be worried about the author making serious mistakes so they'll make less effort to understand. I've experienced this from both sides of the transaction for code, scientific papers, and general discussion. There's no shortcut to the level of understanding given by synthesizing the ideas yourself.
    - refulgentis 2y ago
      So the software lifecycle ends up with a sort of Zeno's paradox, each incremental maintainer understands the system less...fascinating, ty!
      - Tao3300 2y ago
        On sufficiently large and old code bases, yes, this is exactly the case.
      - codr7 2y ago
        I believe pretty much anyone who has observed a few cycles can tell as much. Often the major trigger for a rewrite is that the knowledge has mostly left the building. But then there's the cognitive dissonance; because the we like pretending that the system is the knowledge and thus has economic value in itself, and that people are interchangeable. None of which is true.
    - _the_inflator 2y ago
      I totally agree. Reviewing the solution is limited. What you don’t get are the myriads of other ways that didn’t work out. Elegant solutions are the result of weeding out dozens of other messy ways. So what gets perpetuated here then is the Dunnimg Kruger effect. While it might be speed things up in many normal circumstances, it devalues hard work in the long run. Not good.
- bongodongobob 2y ago
  Right because it's impossible for people to learn things themselves. Knowledge must only be passed down by The Gatekeeper. I'm starting to feel like the programming community is just mad things are easier to learn now.
  - imabotbeep2937 2y ago
    "Easier to learn" is abjectly false. "Easier to get code submitted to a code base" is only marginally correlated with learning anything.
  - irrational 2y ago
    Are they learning? Or are they just accepting the results, as long as it appears to more or less do what they want, and moving on? The danger seems to be code that is syntactically correct and compiles without errors, but is logically incorrect.
    - hombre_fatal 2y ago
      This is just how learning happens either way. As a noob I copied code from Railscasts or Stack Overflow or docs or IRC without understanding it just to get things working. And then at some point I was doing less and less of it, and then rarely at all. But what if the code I copied isn't correct?! Didn't the sky fall down? Well, things would break and I would have to figure out why or steal a better solution, and then I could observe the delta between what didn't work vs what worked. And boom, learning happened. LLMs just speed that cycle up tremendously. The concern trolling over LLMs basically imagines a hypothetical person who can't learn anything and doesn't care. More power to them imo if they can build what they want without understanding it. That's a cracked lazy person we all should fear.
    - bongodongobob 2y ago
      Oh come on, the user is talking about building UIs. I don't know how else you learn. Your attitude just reeks of high-horse. As if it was better to learn things from stackoverflow.
- trybackprop 2y ago
  [dead]
- kajecounterhack 2y ago
  Depends on what's being authored. Many times more experienced engineers review PRs from junior engineers and do have a higher level of knowledge.
- elboru 2y ago
  Sometimes you simply don’t need that knowledge. Not needing to understand something frees your mind.
- WalterSear 2y ago
  IMHO, review is a misnomer for where software engineering is going. I'm not sure where we are going, but review implies less responsibility for the outcome. But I do think that we will have less depth of knowledge of the underlying processes. That's the point of having a machine do it. I expect this, however, to be a good trend: the systems will need to be up to a task before it makes sense to rely on them.
  - reacharavindh 2y ago
    This is how progress (in developer productivity) has always been made. We coded in assembler, then used macros, then a language like C, Fortran, then more of Java/Go/Puthon/Rust/Ruby et al. A developer writing a for loop over a list in Python need to necessarily know about linked lists and memory patterns because Python takes care of it. This frees up that developer from abstracted details and think one level closer to the problem at a higher speed. LLMs _can_ be a good tool under the right hands. They certainly have some ways to become a reliable assistant. I suppose in the way of LLMs, they need better training before they can get there.
    - discreteevent 2y ago
      > We coded in assembler, then used macros, then a language like C, Fortran, then more of Java/Go/Puthon/Rust/Ruby et al. The difference is that: 1) All of those things are deterministic [1] 2) In all of those cases I can debug at the level of the abstraction. [1] Meaning: Do exactly what I say. Don't make it up.
      - CoastalCoder 2y ago
        I agree with your overall point. In a certain sense I'd say optimizing compilers aren't deterministic: The same source code can produce different object code, depending on data and algorithms into which a typical programmer has little insight.
        vitiral 2y ago
- vineyardmike 2y ago
  Frankly, its fine more often than we may care to admit. As the parent comment suggested, UI elements are a great candidate for this. Often very similar (how many apps have a menu bar, side bar, etc) and full of boilerplate. And at the rate things change on the front-end, it's often a candidate for frequent re-writes, so code quality and health don't need to be as strict. It'd be nice if every piece of software ever written was done so by wise experts with hand-crafted libraries, but sometimes it's just a job and just needs to be done.
  - Rumudiez 2y ago
    UI is a terrible example to make your point. Tell me you don’t know frontend development… Accessibility, cross browser+platform support, design systems, SEO, consistency and polish, you name it. You are most certainly not getting that from an LLM and most engineers don’t know how or don’t have a good eye for it to catch when the agent has gone astray or suggested a common mistake
    - pquki4 2y ago
      You definitely have a point, but the reality is that LLMs are about as good as an "average" UI developers in some cases -- lots of people who work on UI every day think very little about accessibility and don't understand if their code actually runs in a non-chromium browser.
- Mmskynettio 2y ago
  I do and i found plenty of issues while doing code review. But i started to read a lot more code than what i did 10 years ago
- williamcotton 2y ago
  This is a false equivalence. When someone is using an LLM they are still the author. Think about it like someone who is searching through record crates for a good sample. They're going to "review" certain drum breaks and then decide if it should be included in an artwork. The reviewing that you're alluding to is like a book reviewer who has nothing to do with the finished product.
- anonyfox 2y ago
  not every developer knows how exactly his modern CPU oder memory layers work, or how electromagnetic waves are building up a signal. people use tools to make things. Its okay. Some "hardcore folks" advance the "lower level" tooling, other creative folks build actually useful things for daily life, and mostly these two groups have very little overlap IMO.
- mewpmewp2 2y ago
  Well it is not actually review where you have a PR. It is more like you are guiding and reviewing a very fast typer in your decided order that in any simple cases handles it 99 percent of the time.
- naijaboiler 2y ago
  Yup that’s an old reviewer/author problem. Reviewer has a huge blind spot because they don’t even know what they don’t know. The author knows what he knows but more importantly also has a bigger grasp on what he doesn’t know. So he understands what’s safe to do and what’s not
- fnordpiglet 2y ago
  Does everything ever written need to be crafted by an artisan? And awful lot of useful stuff written “good enough” is good enough. Depth of knowledge or understanding is irrelevant to a lot of front end UI development where the key is the design itself and that the behavior of the design is accurate and reliable not that the engineer -really- understand at a core of their soul graphql and react with the passion of a Japanese craftsman when they’re building a user interface for the ML backend that internal users use for non critical tasks. There does exist a hierarchy of when depth matters and it’s not homogeneously “literally everything you do.”
zackproser 2y ago
Having a similar experience but in the other direction. I have a lot of backend and frontend development experience but no ml background. Being able to ask stupid questions to get further faster has been making difference for me.

jprete 2y ago

This isn't a good trend. Reviewers never have the depth of knowledge or understanding of authors.

bdjsiqoocwk 2y ago
Also, people shouldn't be allowed to use computers unless they understand how transistors work. If you don't have the depth of knowledge you get nothing.
- desolved 2y ago
  It took me way too long to realize this was a joke.
  - notpachet 2y ago
    It's HN so you can't be entirely sure.
- BOOSTERHIDROGEN 2y ago
  Although this is well-expressed, I'm left with an unexplainable feeling that I can't quite put my finger on.
  - bdjsiqoocwk 2y ago
    Let me try and articulate it. The person Im responding to was gatekeeping. I responded by sarcastically doing the same to an extreme degree. A lot of people Will have agreed with the person i'm responding to. "Oh yeah of course You should understand these things, the things that I already understand", genuinely not realizing that there's no basis for that. When they reae my response they realize what they were doing, and are less feeling embarrassed for their senseless (and pretentious!) gatekeeping.
refulgentis 2y ago
Source? This is unintuitive to me, I can't come up with a rationale.
- jprete 2y ago
  I know of no review process that produces the same level of understanding as does authorship, because the author must build the model from scratch and so must see all the details, while the reviewer is able to do less work because they're fundamentally riding on the author's understanding. In fact, in a high-trust system, e.g. a good engineering culture in a tech company, the reviewer will learn even less, because they won't be worried about the author making serious mistakes so they'll make less effort to understand. I've experienced this from both sides of the transaction for code, scientific papers, and general discussion. There's no shortcut to the level of understanding given by synthesizing the ideas yourself.
  - refulgentis 2y ago
    So the software lifecycle ends up with a sort of Zeno's paradox, each incremental maintainer understands the system less...fascinating, ty!
    - Tao3300 2y ago
      On sufficiently large and old code bases, yes, this is exactly the case.
    - codr7 2y ago
      I believe pretty much anyone who has observed a few cycles can tell as much. Often the major trigger for a rewrite is that the knowledge has mostly left the building. But then there's the cognitive dissonance; because the we like pretending that the system is the knowledge and thus has economic value in itself, and that people are interchangeable. None of which is true.
  - _the_inflator 2y ago
    I totally agree. Reviewing the solution is limited. What you don’t get are the myriads of other ways that didn’t work out. Elegant solutions are the result of weeding out dozens of other messy ways. So what gets perpetuated here then is the Dunnimg Kruger effect. While it might be speed things up in many normal circumstances, it devalues hard work in the long run. Not good.
bongodongobob 2y ago
Right because it's impossible for people to learn things themselves. Knowledge must only be passed down by The Gatekeeper. I'm starting to feel like the programming community is just mad things are easier to learn now.
- imabotbeep2937 2y ago
  "Easier to learn" is abjectly false. "Easier to get code submitted to a code base" is only marginally correlated with learning anything.
- irrational 2y ago
  Are they learning? Or are they just accepting the results, as long as it appears to more or less do what they want, and moving on? The danger seems to be code that is syntactically correct and compiles without errors, but is logically incorrect.
  - hombre_fatal 2y ago
    This is just how learning happens either way. As a noob I copied code from Railscasts or Stack Overflow or docs or IRC without understanding it just to get things working. And then at some point I was doing less and less of it, and then rarely at all. But what if the code I copied isn't correct?! Didn't the sky fall down? Well, things would break and I would have to figure out why or steal a better solution, and then I could observe the delta between what didn't work vs what worked. And boom, learning happened. LLMs just speed that cycle up tremendously. The concern trolling over LLMs basically imagines a hypothetical person who can't learn anything and doesn't care. More power to them imo if they can build what they want without understanding it. That's a cracked lazy person we all should fear.
  - bongodongobob 2y ago
    Oh come on, the user is talking about building UIs. I don't know how else you learn. Your attitude just reeks of high-horse. As if it was better to learn things from stackoverflow.
trybackprop 2y ago
[dead]
kajecounterhack 2y ago
Depends on what's being authored. Many times more experienced engineers review PRs from junior engineers and do have a higher level of knowledge.
elboru 2y ago
Sometimes you simply don’t need that knowledge. Not needing to understand something frees your mind.
WalterSear 2y ago
IMHO, review is a misnomer for where software engineering is going. I'm not sure where we are going, but review implies less responsibility for the outcome. But I do think that we will have less depth of knowledge of the underlying processes. That's the point of having a machine do it. I expect this, however, to be a good trend: the systems will need to be up to a task before it makes sense to rely on them.
- reacharavindh 2y ago
  This is how progress (in developer productivity) has always been made. We coded in assembler, then used macros, then a language like C, Fortran, then more of Java/Go/Puthon/Rust/Ruby et al. A developer writing a for loop over a list in Python need to necessarily know about linked lists and memory patterns because Python takes care of it. This frees up that developer from abstracted details and think one level closer to the problem at a higher speed. LLMs _can_ be a good tool under the right hands. They certainly have some ways to become a reliable assistant. I suppose in the way of LLMs, they need better training before they can get there.
  - discreteevent 2y ago
    > We coded in assembler, then used macros, then a language like C, Fortran, then more of Java/Go/Puthon/Rust/Ruby et al. The difference is that: 1) All of those things are deterministic [1] 2) In all of those cases I can debug at the level of the abstraction. [1] Meaning: Do exactly what I say. Don't make it up.
    - CoastalCoder 2y ago
      I agree with your overall point. In a certain sense I'd say optimizing compilers aren't deterministic: The same source code can produce different object code, depending on data and algorithms into which a typical programmer has little insight.
      - vitiral 2y ago
vineyardmike 2y ago
Frankly, its fine more often than we may care to admit. As the parent comment suggested, UI elements are a great candidate for this. Often very similar (how many apps have a menu bar, side bar, etc) and full of boilerplate. And at the rate things change on the front-end, it's often a candidate for frequent re-writes, so code quality and health don't need to be as strict. It'd be nice if every piece of software ever written was done so by wise experts with hand-crafted libraries, but sometimes it's just a job and just needs to be done.
- Rumudiez 2y ago
  UI is a terrible example to make your point. Tell me you don’t know frontend development… Accessibility, cross browser+platform support, design systems, SEO, consistency and polish, you name it. You are most certainly not getting that from an LLM and most engineers don’t know how or don’t have a good eye for it to catch when the agent has gone astray or suggested a common mistake
  - pquki4 2y ago
    You definitely have a point, but the reality is that LLMs are about as good as an "average" UI developers in some cases -- lots of people who work on UI every day think very little about accessibility and don't understand if their code actually runs in a non-chromium browser.
Mmskynettio 2y ago
I do and i found plenty of issues while doing code review. But i started to read a lot more code than what i did 10 years ago
williamcotton 2y ago
This is a false equivalence. When someone is using an LLM they are still the author. Think about it like someone who is searching through record crates for a good sample. They're going to "review" certain drum breaks and then decide if it should be included in an artwork. The reviewing that you're alluding to is like a book reviewer who has nothing to do with the finished product.
anonyfox 2y ago
not every developer knows how exactly his modern CPU oder memory layers work, or how electromagnetic waves are building up a signal. people use tools to make things. Its okay. Some "hardcore folks" advance the "lower level" tooling, other creative folks build actually useful things for daily life, and mostly these two groups have very little overlap IMO.
mewpmewp2 2y ago
Well it is not actually review where you have a PR. It is more like you are guiding and reviewing a very fast typer in your decided order that in any simple cases handles it 99 percent of the time.
naijaboiler 2y ago
Yup that’s an old reviewer/author problem. Reviewer has a huge blind spot because they don’t even know what they don’t know. The author knows what he knows but more importantly also has a bigger grasp on what he doesn’t know. So he understands what’s safe to do and what’s not
fnordpiglet 2y ago
Does everything ever written need to be crafted by an artisan? And awful lot of useful stuff written “good enough” is good enough. Depth of knowledge or understanding is irrelevant to a lot of front end UI development where the key is the design itself and that the behavior of the design is accurate and reliable not that the engineer -really- understand at a core of their soul graphql and react with the passion of a Japanese craftsman when they’re building a user interface for the ML backend that internal users use for non critical tasks. There does exist a hierarchy of when depth matters and it’s not homogeneously “literally everything you do.”

zackproser 2y ago

Having a similar experience but in the other direction. I have a lot of backend and frontend development experience but no ml background. Being able to ask stupid questions to get further faster has been making difference for me.