6 ms·

Hello, author here. Thanks for your comment. I agree with your first point, maybe AI will close some of those gaps with future advances, but I think a large pa

Hello, author here. Thanks for your comment. I agree with your first point, maybe AI will close some of those gaps with future advances, but I think a large part of the damage will have been done by then. Regarding the memory of reasoning from LLMs, I think the issue is that even if you can solve it in the future, you already have code for which you've lost the artifacts associated with the original generation. Overall I find there's a lot of talks (especially in the mainstream media) about AI "always learning" when they don't actually learn new anything until a new model is released. > Why does it require 100% accuracy 100% of the time? Humans are not 100% accurate 100% of the time and we seem to trust them with our code. Correct, but humans writing code don't lead to a Bus Factor of 0, so it's easier to go back, understand what is wrong and address it. If the other gaps mentioned above are addressed, then I agree that this also partially goes away.

mmhmmmmmm 1y ago
[dead]
scarface_74 1y ago
How is that any different than any other legacy code where the reason for the decisions being made have long been forgotten?
johnfn 1y ago
> Regarding the memory of reasoning from LLMs, I think the issue is that even if you can solve it in the future, you already have code for which you've lost the artifacts associated with the original generation. Overall I find there's a lot of talks (especially in the mainstream media) about AI "always learning" when they don't actually learn new anything until a new model is released. But this already exists! At work, our code is full of code where the original reasoning for the code is lost. Sometimes someone has forgotten, sometimes the person who wrote it is no longer at the company any more, and so on. > Correct, but humans writing code don't lead to a Bus Factor of 0, so it's easier to go back, understand what is wrong and address it. But there are plenty of instances where I work with code that has a bus factor of 0. The conclusion of your article is that vibe coding is "fundamentally flawed". But every aspect you've described about vibe coding has an analog in normal software engineering, and I don't think you would claim that is "fundamentally flawed".
- matt_kantor 1y ago
  I can't speak for the author, but I would definitely claim that having a bus factor of zero for any remotely-mission-critical piece of software is "fundamentally flawed", no matter the cause. I'd say the same for a bus factor of one in most settings.
  - johnfn 1y ago
    I think that's moving goalposts. The original post never talks about vibe-coding mission-critical software - and I wouldn't advocate for that, either. The post says that all vibe coding is fundamentally flawed.
    - matt_kantor 1y ago
      That's fair, and I agree with you that the generalizations in the article's conclusion go too far. I added the "remotely-mission-critical" qualifier to capture additional nuance. Tolerance for a low bus factor should be inversely correlated with a project's importance. That wasn't explicitly stated in the article, but it seems uncontroversial, and I suspect the author would agree with me.
- matt_kantor 1y ago
  To make sure I understand your position: > But there are plenty of instances where I work with code that has a bus factor of 0. Do you think this is a problem? As per my other replies, if all of these instances are in completely unimportant projects, then I could see you answering "no" (but I'd be concerned if you're spending a lot of time on unimportant things). If they are important, isn't the fact that knowledge about them has been lost indicative of a flaw in how your team/company operates?
  - johnfn 1y ago
    I do think this is a problem. But the article goes one step further than claiming it's a problem - it claims that a bus factor of 0, and by extension, vibe coding, is "fundamentally flawed". I don't think that a bus factor of 0 is indicative of a "fundamental flaw".
elliotto 1y ago
Why would the damage have been done by then? What does that even mean?
dinfinity 1y ago
The solution is almost trivial and well known: Document. Whenever a member of the team leaves, you should always make them document what is in their head and not already documented (or even better, make documenting a required part of normal development). The thing to understand is that every (new) AI chat session is effectively a member entering and leaving your team. So to make that work well you need great onboarding (provide documentation) and great offboarding (require documentation).

mmhmmmmmm 1y ago

[dead]

scarface_74 1y ago

How is that any different than any other legacy code where the reason for the decisions being made have long been forgotten?

johnfn 1y ago

> Regarding the memory of reasoning from LLMs, I think the issue is that even if you can solve it in the future, you already have code for which you've lost the artifacts associated with the original generation. Overall I find there's a lot of talks (especially in the mainstream media) about AI "always learning" when they don't actually learn new anything until a new model is released. But this already exists! At work, our code is full of code where the original reasoning for the code is lost. Sometimes someone has forgotten, sometimes the person who wrote it is no longer at the company any more, and so on. > Correct, but humans writing code don't lead to a Bus Factor of 0, so it's easier to go back, understand what is wrong and address it. But there are plenty of instances where I work with code that has a bus factor of 0. The conclusion of your article is that vibe coding is "fundamentally flawed". But every aspect you've described about vibe coding has an analog in normal software engineering, and I don't think you would claim that is "fundamentally flawed".

matt_kantor 1y ago
I can't speak for the author, but I would definitely claim that having a bus factor of zero for any remotely-mission-critical piece of software is "fundamentally flawed", no matter the cause. I'd say the same for a bus factor of one in most settings.
- johnfn 1y ago
  I think that's moving goalposts. The original post never talks about vibe-coding mission-critical software - and I wouldn't advocate for that, either. The post says that all vibe coding is fundamentally flawed.
  - matt_kantor 1y ago
    That's fair, and I agree with you that the generalizations in the article's conclusion go too far. I added the "remotely-mission-critical" qualifier to capture additional nuance. Tolerance for a low bus factor should be inversely correlated with a project's importance. That wasn't explicitly stated in the article, but it seems uncontroversial, and I suspect the author would agree with me.
matt_kantor 1y ago
To make sure I understand your position: > But there are plenty of instances where I work with code that has a bus factor of 0. Do you think this is a problem? As per my other replies, if all of these instances are in completely unimportant projects, then I could see you answering "no" (but I'd be concerned if you're spending a lot of time on unimportant things). If they are important, isn't the fact that knowledge about them has been lost indicative of a flaw in how your team/company operates?
- johnfn 1y ago
  I do think this is a problem. But the article goes one step further than claiming it's a problem - it claims that a bus factor of 0, and by extension, vibe coding, is "fundamentally flawed". I don't think that a bus factor of 0 is indicative of a "fundamental flaw".

elliotto 1y ago

Why would the damage have been done by then? What does that even mean?

dinfinity 1y ago

The solution is almost trivial and well known: Document. Whenever a member of the team leaves, you should always make them document what is in their head and not already documented (or even better, make documenting a required part of normal development). The thing to understand is that every (new) AI chat session is effectively a member entering and leaving your team. So to make that work well you need great onboarding (provide documentation) and great offboarding (require documentation).