7 ms·

I think most of this is correct, in spite of potentially being built on a bad assumption. The assumption is that LLMs should be writing the code and human engi

I think most of this is correct, in spite of potentially being built on a bad assumption. The assumption is that LLMs should be writing the code and human engineers reviewing and verifying the LLM output. And that this pushes the cost of producing down. And I fundamentally disagree with that. Every time I ask LLMs to write code, even with Opus 4.8 (haven't tried it with Opus 5 yet), what I get ends up being totally rewritten. LLMs still aren't good at writing maintainable code. Can they write plausibly functional code? Yes. But it won't survive the long term. People using LLMs to write all their code are gambling on them eventually getting to a point where the LLMs can fix their own code. It's possible, but I wouldn't necessarily bet on it. Where I have found immense value from LLMs is in code review. Repeated review by LLMs catches an amazing amount of potential issues. They really shine on security review, but are very effective with any kind of review. The other thing that the "LLMs write code camp" misunderstands is that writing was never the bottleneck. Understanding was. And understanding the code is still the bottleneck. But understanding is truly gained during the writing loop. The understanding you gain from pure reading or code review is marginal compared to the understanding you gain while writing. Most of the time previously spent writing was actually spent updating and deepening our understanding of the system under development. There's no replacement for that understanding in a world where LLMs are doing the writing. But if you flip it: humans write, LLMs review, then you still get a major gain -- not in speed, but in quality. And you keep the understanding loop intact. I would propose that this might be the best way to deploy LLMs.

lukan 2mo ago
" But understanding is truly gained during the writing loop. The understanding you gain from pure reading or code review is marginal compared to the understanding you gain while writing." Debugging code step by step is how I understand complicated code.
dwayneII 2mo ago
I write code in Go in an established codebase. 90% of the time the LLM gets a basic api/feature right and fully e2e tested as long as I give it enough business context in the prompt. The other 10% of the time I have to do some follow up prompts to either change the behaviour or change the approach of a given step in the flow or just to point out that the wrong pattern was used and please rather use our codebase standard. I haven’t actually hand-written code in well over a year. Is this way faster? Absolutely. Does it lead to better code? Yes, because I have time to write all the regression tests that keep the behaviour as I wanted it when others come bungling around.
- sanderjd 2mo ago
  I do think the comment is right about understanding. I'm still feeling out what I think is the right level of understanding to invest in now. I think it is probably not the deep line by line understanding I used to have of the systems I worked on. But I also think it's easy to remain too aloof from how the system is being built, and that this is very bad. I think the right answer is somewhere in between, but I'm still working through my own process and forcing functions to strike that balance properly.
broast 2mo ago
many of us work with swarms of cheap off-shore contractors and have to rewrite or finish their code already as it is. The llm is much lower friction
- rufius 2mo ago
  IME, the LLM when used well also gets closer to correct.
ilovefood 2mo ago
You're right about understanding. Where we might disagree is the conclusion. I think it's possible to build the understanding without having to type out syntax by hand (debugging, writing tests, etc). Maybe we're missing much better verification tools? LLMs will likely play a big part in those too (explain the codebase, walk me through A, B, C etc)
SoftTalker 2mo ago
How do you handle the fact that an LLM can't seem to help itself from disgorging page after page of words no matter what it's asked to do? I've never seen an LLM say "this looks good as-is; I would not spend any more time on it; what's next?" it will always seem to suggest using another pattern or additional abstractions or other yak shaving. But to be fair human code reviews have the same problem. It's like reviewers feel they have not done their job if they don't find something wrong.
- dbingham 2mo ago
  Yeah, that's just a problem with review. I tend to keep it going until the stuff its finding is stuff that I look at and think "I can live with that". Usually by that point it's found the worthwhile stuff and what remains is minor or, like you said, yak shaving.
  - SoftTalker 2mo ago
    There used to be a bit of "wisdom" passed around that said "when you submit something to the client (or management, etc) for approval, leave an obvious mistake somewhere." Then they will find it and point it out, you can easily correct it, and everyone is happy. Otherwise they will find something to critique, just to demonstrate that they contributed something to the review. I wonder if the same trick works with AI?
    - clbrmbr 2mo ago
      I think so, if you can inject synthetic errors with the same distribution as the errors you care about.
- RyanHamilton 2mo ago
  I've tried and tried to get an LLM to delete code, it couldn't do it. I knew a file was 40% bad so what I ended up doing was deleting the file, then asking AI to create the missing file. That's how I got AI to delete code :)
bcrosby95 2mo ago
I've come to realize that the smarter LLMs get, the less they understand the point of abstractions. I've been using it for a unity game for the past few years. Nowadays it will go sleuthing into packages and assembly and make decisions based upon what it sees there. It will make comments about why it's doing something based upon a function call 3 methods deep. God forbid any of these details change in a minor version update.
- hkpack 2mo ago
  Try reducing thinking and use lighter models. For example I observed that using Sonnet works much better (compared to Opus) for tasks I want to be in control of architecture and just need a faster code input.
lordnacho 2mo ago
> The other thing that the "LLMs write code camp" misunderstands is that writing was never the bottleneck. Understanding was. And understanding the code is still the bottleneck. But understanding is truly gained during the writing loop. The understanding you gain from pure reading or code review is marginal compared to the understanding you gain while writing. This was my stance a couple of years ago, but now I've given it up. It turns out writing actually was the bottleneck. You can understand perfectly well what you want, but writing it is long and tedious to the point where you find excuses not to do it. Particularly with version 2, the step where you have an OK system and you want to improve it. Quite a lot of changing the code is just useless busywork: re-wiring old functions, moving imports around, searching for locations that benefit from extracting a common piece of code. And each time you do one of those, there's a decent chance you did something even more trivial like forgetting a semicolon or calling the wrong function. Now that I have an LLM helping me, I can see why. The critical decision is a terse declarative like "we need to have several TCP connections instead of one, and just use the sequence number to arbitrate". A human junior programmer could perfectly well understand what this meant, but he would have to go through all of the above to get to the final product. Now, I can just tell the LLM and I will get what I want, even with the things I didn't explicitly state, without spending attention. This means I can use my attention on the things that matter. So instead of spending today thinking about how to arbitrate between the TCP connections and tomorrow thinking about pre-calculating my outgoing orders, I can just do both today. I don't waste the good waking hours chasing minor bugs, I just think about the large structure. I get the feeling the best programmers of years past were actually masters of the little things, which led them to be able to look at the big things. Essentially it was cheaper for them to get to the top of the mountain, where you can see the landscape. Kinda like how the kid who was good at mental arithmetic in primary school was also good at calculus at the end of high school: if you don't have to concentrate on the little things, you have time for the big things.
- leptons 2mo ago
  >It turns out writing actually was the bottleneck. You can understand perfectly well what you want, but writing it is long and tedious to the point where you find excuses not to do it. Speak for yourself. Writing code has never been a bottleneck for some of us. I can't speak for everyone, and neither should you. >Now, I can just tell the LLM and I will get what I want, even with the things I didn't explicitly state, without spending attention. This should worry you. All too often the LLM invents things I didn't ask for and implements things I didn't need. YMMV, I guess. If slop gets the job done, and nobody notices, then who should care?
clbrmbr 2mo ago
I’ve found that doing the full requirements capture, planning, writing, reviewing, gardening loop with frontier models has worked quite well since last October, and phenomenally since Fable 5. The key I’ve found is human peer review. The reviewer jumps on a live call with the developer, pulls up the PR with transcription on, and asks questions. At the end of the call, the transcript passes back into the coding agent and the PR is polished up, becoming more self-documenting, and the humans are left with some degree of common understanding of what’s going on. I’ve been operating my team of ~15 this way for 9mo to great effect… there is simply no going back to the stone ages.
- oogali 2mo ago
  Here's an odd request for you, as a fellow New Jersey-an... Can I watch/observe one of your review sessions? Every few weeks, I hear the beginnings of a great approach towards working with LLMs but I rarely see it in practice. If you're open to this, remote or in person, ping my username at gmail.
tracerbulletx 2mo ago
I have not experienced this "llms dont write maintainable code narrative" I just tell it the shape of the entities and apis I want vaguely and the mental model and the output is excellent.
- supriyo-biswas 2mo ago
  I could do this in the beginning of 2025. Unfortunately, given the over-proactiveness of model these days, I find that they end up inferring what my original request was and implementing it anyway, and then writing unit tests, etc. even when all I asked for was to wire up a few components, as an example.
  - lukeschlather 2mo ago
    I actually find the converse, that I try to scope things down into chunks small enough for the model to work on, but it will start trying to hack off pieces that don't belong because it thinks the restricted scope is the whole world and I have to stop it and explain the bigger picture so it stops trying to remove things that serve the full scope.
- davidkoenit 2mo ago
  A lot of people should try being more specific in their prompts, but then again at a certain point you're basically just writing all the code in sentence form!
- disgruntledphd2 2mo ago
  For me, at least, the overall design needs to come from using the code. I think for refactoring, you're definitely right as if you give them a good high level sketch you can get them to do all of the more tedious implementation.
win311fwg 2mo ago
> LLMs still aren't good at writing maintainable code. I find that depends on the target language. They can be good at writing maintainable code, but not consistently across every language. The languages beginners usually gravitate towards are especially hard for LLMs to produce quality output for. Presumably this is due to the training data including all the unmaintainable codebases written by beginners in those languages, which hasn't allowed the LLM to converge on recognizing what a maintainable codebase looks like in those languages.
himata4113 2mo ago
There is the flip side of where the code is no longer read by humans and is becoming the prevailing way software is shipped in tiny businesses. You don't need to code to be maintainable since you will never maintain it, the AI will and the quality will naturally improve as models improve. For example 5.6 sol and fable are showing signs where you can feed garbage in and it will spit out something pretty decent, definitely not the quality you'd expect from a senior developer with millenia of experience, but that of your average grunt worker turning words in an issue board into code. However, sometimes then I tell it to write an app with detailed instructions and it spits out garbage so your mileage might vary.
- davidkoenit 2mo ago
  I find this to work for a V1 or product with only a few users but not sure this will scale. At least you still need software engineers reviewing the functionality and giving the AI hints at what to do.
gste 2mo ago
I think this is outdated. If you follow spec-driven development, get the model to do all the planning work upfront, review and iterate the plan, write clear markdown file documentation on the abstractions and patterns you want to follow, then you have every opportunity to tell the model how you want it to write the code. If you use Opus or Fable 5 it will then write the code better and faster than you will.
- written-beyond 2mo ago
  Depends on how long that planning/review cycle takes.
  - gste 2mo ago
    True, but I think people are also missing the trick behind markdowns that reference each other and are used for context window management. They go in your source control, they are reusable, extendable, and composable in the same way code is. So planning is not a one-off effort. It compounds over time. If you're doing it right your plan is referencing automation techniques - testing, CI/CD, etc. This further compounds so that the agents verify their own work against the standards that you set. To be honest, the models are getting so good that they do most of this unprompted now.
- eikenberry 2mo ago
  IMO "get the model to do all the planning work upfront, review and iterate the plan" is backwards. It works better if you do an initial plan yourself and then have the AI review it (and iterate as necessary). Makes you think about the design for a bit so you can have a semblance of a mental model.
  - prymitive 2mo ago
    I have never seen a plan written by LLM that wasn’t vague and light on details, every time I need to ask for more details and every time I ask to implement it trips over some dead end in the plan, discards the plan and continues as if there was no plan to begin with. Planning feels often just narrating the request and a wish list then what a human would do - methodically build enough understanding so that you’re confident of the direction you choose. LLM plans are overhyped and overrated.
drTobiasFunke 2mo ago
“Not in speed, but in quality” — unfortunately though, not a single CEO, executive, company, VC, investor or anyone with the power to make decisions cares about quality instead of speed.
- consumer451 2mo ago
  I agree. Also, what does "code quality" even mean in the age of agentic dev? If the code works, and is secure, what else matters? I do know what good code looks like, but does that even matter anymore? All I know is that now, I get to focus on endless UX polish, which is the only thing the matters. I feel like we are living through something like the Protestant Reformation, where priests once spoke Latin, and then started to speak in plain local language. The old guard did not like this.
  - bigstrat2003 2mo ago
    > If the code works, and is secure, what else matters? Maybe so, but it doesn't.
  - Silhouette 2mo ago
    Also, what does "code quality" even mean in the age of agentic dev? If the code works, and is secure, what else matters? AI pricing is mostly based on tokens consumed right now. Shouldn't that mean being able to quickly and reliably analyse existing code and to make only small local changes to implement new functionality is as valuable as ever - if not more so - if you're relying on agentic LLMs to do the grunt work? A lot of things about writing clear specs and developing systematically and employing lots of different kinds of checks and controls to ensure quality and performance have always been true but used to get brushed under the carpet by a lot of cheap/lazy development teams. If LLMs really do accelerate everything about development - including negative behaviours like acting undesirably based on flawed or ambiguous information and doubling down on mistaken assumptions - then the pattern across all of these areas is that doing things the right way is more important than ever if you want to get good results from AI assistance.
- layer8 2mo ago
vatsachak 2mo ago
I hate saying this but unironically opus 5/gpt 5.6 are writing maintainable code on the order of 500-1000 lines
YZF 2mo ago
In my domain since circa Opus 4.5 LLMs write code as good as most engineers given well defined small enough chunk of work. They refactor. They write tests. These days LLM also debug/troubleshoot better than most engineers. Are they as good as handcrafted code by 0.1% of top software engineers. Generally no. But neither is 99.9% of real code. LLMs also are good at code reviews. What they'll miss is often the big picture but they can still catch plenty of issues. I still want to see a human in the loop in my domain. Totally agree that writing the code was never the bottleneck. We're not seeing massive productivity gains even if some code is written faster. It's not just about understanding but also various other activities that happen in large companies and teams. Also agree LLMs can be used to gain quality but realistically most orgs are going to aim for "fixed or decreasing" quality at lower costs.
luaKmua 2mo ago
This is where I've landed too, I imagine based off all the responses that it depends on domain and also the rigor you care about in your implementation. For me, what I do, and the quality required, there's just not much time (if any) saved on handing off the implementation to an LLM. But that doesn't mean they're useless either. I use them all the time for review as you mentioned or to knock out one-off scripts that don't go anywhere near source control. There's just no world where I don't need to understand every line of code that I'm responsible for getting into our project.
lioeters 2mo ago
> writing was never the bottleneck. Understanding was. Well put. This insight is worth repeating in every discussion on the subject, from software engineering to mathematics. The problem is that understanding is not the product being sold. The business model is for everyone to become consumers of what the magical genie generates, where the "understanding" is kept on the side of the model providers. This ensures a future generation of consumers dependent on someone else to provide the understanding. Otherwise, you can create your own answers based on actually understanding the code, theorem, proofs, etc. Smart consumers of LLMs will use them to increase their own knowledge and understanding, so that the service is augmenting their intelligence, not replacing it.
ManuelKiessling 2mo ago
To be honest, it feels like you are contradicting yourself. If LLMs are, as you stated, really good at catching potential issues, then they are, almost by definition, really good at producing code without potential issues, if guided correctly: all they need to do is inspect and iterate, until they do not find any more potential issue in the code they produced.
- layer8 2mo ago
  That isn’t necessarily true. LLMs could be good at finding issues that human miss when coding, while being bad at catching issues that they themselves miss while generating code.
- booleandilemma 2mo ago
  [dead]
nicce 2mo ago
> They really shine on security review, but are very effective with any kind of review. They are getting more and more hostile for making any security assesments. I wonder will they even write secure code in the future if they can’t point vulnerabilities from existing code.
KronisLV 2mo ago
> The other thing that the "LLMs write code camp" misunderstands is that writing was never the bottleneck. Anyone who's seen tech debt where each item is well known and understood but just big in scope knows that this isn't true universally - depending on your team size and composition, any damn thing can be the bottleneck, often at different times too. I'll take everything that helps me resolve them with reasonable trade-offs, even if I need to come up with ways to mitigate the issues created by those trade-offs (like enforcing >90% test coverage as a starting point).

lukan 2mo ago

" But understanding is truly gained during the writing loop. The understanding you gain from pure reading or code review is marginal compared to the understanding you gain while writing." Debugging code step by step is how I understand complicated code.

dwayneII 2mo ago

I write code in Go in an established codebase. 90% of the time the LLM gets a basic api/feature right and fully e2e tested as long as I give it enough business context in the prompt. The other 10% of the time I have to do some follow up prompts to either change the behaviour or change the approach of a given step in the flow or just to point out that the wrong pattern was used and please rather use our codebase standard. I haven’t actually hand-written code in well over a year. Is this way faster? Absolutely. Does it lead to better code? Yes, because I have time to write all the regression tests that keep the behaviour as I wanted it when others come bungling around.

sanderjd 2mo ago
I do think the comment is right about understanding. I'm still feeling out what I think is the right level of understanding to invest in now. I think it is probably not the deep line by line understanding I used to have of the systems I worked on. But I also think it's easy to remain too aloof from how the system is being built, and that this is very bad. I think the right answer is somewhere in between, but I'm still working through my own process and forcing functions to strike that balance properly.

broast 2mo ago

many of us work with swarms of cheap off-shore contractors and have to rewrite or finish their code already as it is. The llm is much lower friction

rufius 2mo ago
IME, the LLM when used well also gets closer to correct.

ilovefood 2mo ago

You're right about understanding. Where we might disagree is the conclusion. I think it's possible to build the understanding without having to type out syntax by hand (debugging, writing tests, etc). Maybe we're missing much better verification tools? LLMs will likely play a big part in those too (explain the codebase, walk me through A, B, C etc)

SoftTalker 2mo ago

How do you handle the fact that an LLM can't seem to help itself from disgorging page after page of words no matter what it's asked to do? I've never seen an LLM say "this looks good as-is; I would not spend any more time on it; what's next?" it will always seem to suggest using another pattern or additional abstractions or other yak shaving. But to be fair human code reviews have the same problem. It's like reviewers feel they have not done their job if they don't find something wrong.

dbingham 2mo ago
Yeah, that's just a problem with review. I tend to keep it going until the stuff its finding is stuff that I look at and think "I can live with that". Usually by that point it's found the worthwhile stuff and what remains is minor or, like you said, yak shaving.
- SoftTalker 2mo ago
  There used to be a bit of "wisdom" passed around that said "when you submit something to the client (or management, etc) for approval, leave an obvious mistake somewhere." Then they will find it and point it out, you can easily correct it, and everyone is happy. Otherwise they will find something to critique, just to demonstrate that they contributed something to the review. I wonder if the same trick works with AI?
  - clbrmbr 2mo ago
    I think so, if you can inject synthetic errors with the same distribution as the errors you care about.
RyanHamilton 2mo ago
I've tried and tried to get an LLM to delete code, it couldn't do it. I knew a file was 40% bad so what I ended up doing was deleting the file, then asking AI to create the missing file. That's how I got AI to delete code :)

bcrosby95 2mo ago

I've come to realize that the smarter LLMs get, the less they understand the point of abstractions. I've been using it for a unity game for the past few years. Nowadays it will go sleuthing into packages and assembly and make decisions based upon what it sees there. It will make comments about why it's doing something based upon a function call 3 methods deep. God forbid any of these details change in a minor version update.

hkpack 2mo ago
Try reducing thinking and use lighter models. For example I observed that using Sonnet works much better (compared to Opus) for tasks I want to be in control of architecture and just need a faster code input.

lordnacho 2mo ago

> The other thing that the "LLMs write code camp" misunderstands is that writing was never the bottleneck. Understanding was. And understanding the code is still the bottleneck. But understanding is truly gained during the writing loop. The understanding you gain from pure reading or code review is marginal compared to the understanding you gain while writing. This was my stance a couple of years ago, but now I've given it up. It turns out writing actually was the bottleneck. You can understand perfectly well what you want, but writing it is long and tedious to the point where you find excuses not to do it. Particularly with version 2, the step where you have an OK system and you want to improve it. Quite a lot of changing the code is just useless busywork: re-wiring old functions, moving imports around, searching for locations that benefit from extracting a common piece of code. And each time you do one of those, there's a decent chance you did something even more trivial like forgetting a semicolon or calling the wrong function. Now that I have an LLM helping me, I can see why. The critical decision is a terse declarative like "we need to have several TCP connections instead of one, and just use the sequence number to arbitrate". A human junior programmer could perfectly well understand what this meant, but he would have to go through all of the above to get to the final product. Now, I can just tell the LLM and I will get what I want, even with the things I didn't explicitly state, without spending attention. This means I can use my attention on the things that matter. So instead of spending today thinking about how to arbitrate between the TCP connections and tomorrow thinking about pre-calculating my outgoing orders, I can just do both today. I don't waste the good waking hours chasing minor bugs, I just think about the large structure. I get the feeling the best programmers of years past were actually masters of the little things, which led them to be able to look at the big things. Essentially it was cheaper for them to get to the top of the mountain, where you can see the landscape. Kinda like how the kid who was good at mental arithmetic in primary school was also good at calculus at the end of high school: if you don't have to concentrate on the little things, you have time for the big things.

leptons 2mo ago
>It turns out writing actually was the bottleneck. You can understand perfectly well what you want, but writing it is long and tedious to the point where you find excuses not to do it. Speak for yourself. Writing code has never been a bottleneck for some of us. I can't speak for everyone, and neither should you. >Now, I can just tell the LLM and I will get what I want, even with the things I didn't explicitly state, without spending attention. This should worry you. All too often the LLM invents things I didn't ask for and implements things I didn't need. YMMV, I guess. If slop gets the job done, and nobody notices, then who should care?

clbrmbr 2mo ago

I’ve found that doing the full requirements capture, planning, writing, reviewing, gardening loop with frontier models has worked quite well since last October, and phenomenally since Fable 5. The key I’ve found is human peer review. The reviewer jumps on a live call with the developer, pulls up the PR with transcription on, and asks questions. At the end of the call, the transcript passes back into the coding agent and the PR is polished up, becoming more self-documenting, and the humans are left with some degree of common understanding of what’s going on. I’ve been operating my team of ~15 this way for 9mo to great effect… there is simply no going back to the stone ages.

oogali 2mo ago
Here's an odd request for you, as a fellow New Jersey-an... Can I watch/observe one of your review sessions? Every few weeks, I hear the beginnings of a great approach towards working with LLMs but I rarely see it in practice. If you're open to this, remote or in person, ping my username at gmail.

tracerbulletx 2mo ago

I have not experienced this "llms dont write maintainable code narrative" I just tell it the shape of the entities and apis I want vaguely and the mental model and the output is excellent.

supriyo-biswas 2mo ago
I could do this in the beginning of 2025. Unfortunately, given the over-proactiveness of model these days, I find that they end up inferring what my original request was and implementing it anyway, and then writing unit tests, etc. even when all I asked for was to wire up a few components, as an example.
- lukeschlather 2mo ago
  I actually find the converse, that I try to scope things down into chunks small enough for the model to work on, but it will start trying to hack off pieces that don't belong because it thinks the restricted scope is the whole world and I have to stop it and explain the bigger picture so it stops trying to remove things that serve the full scope.
davidkoenit 2mo ago
A lot of people should try being more specific in their prompts, but then again at a certain point you're basically just writing all the code in sentence form!
disgruntledphd2 2mo ago
For me, at least, the overall design needs to come from using the code. I think for refactoring, you're definitely right as if you give them a good high level sketch you can get them to do all of the more tedious implementation.

win311fwg 2mo ago

> LLMs still aren't good at writing maintainable code. I find that depends on the target language. They can be good at writing maintainable code, but not consistently across every language. The languages beginners usually gravitate towards are especially hard for LLMs to produce quality output for. Presumably this is due to the training data including all the unmaintainable codebases written by beginners in those languages, which hasn't allowed the LLM to converge on recognizing what a maintainable codebase looks like in those languages.

himata4113 2mo ago

There is the flip side of where the code is no longer read by humans and is becoming the prevailing way software is shipped in tiny businesses. You don't need to code to be maintainable since you will never maintain it, the AI will and the quality will naturally improve as models improve. For example 5.6 sol and fable are showing signs where you can feed garbage in and it will spit out something pretty decent, definitely not the quality you'd expect from a senior developer with millenia of experience, but that of your average grunt worker turning words in an issue board into code. However, sometimes then I tell it to write an app with detailed instructions and it spits out garbage so your mileage might vary.

davidkoenit 2mo ago
I find this to work for a V1 or product with only a few users but not sure this will scale. At least you still need software engineers reviewing the functionality and giving the AI hints at what to do.

gste 2mo ago

I think this is outdated. If you follow spec-driven development, get the model to do all the planning work upfront, review and iterate the plan, write clear markdown file documentation on the abstractions and patterns you want to follow, then you have every opportunity to tell the model how you want it to write the code. If you use Opus or Fable 5 it will then write the code better and faster than you will.

written-beyond 2mo ago
Depends on how long that planning/review cycle takes.
- gste 2mo ago
  True, but I think people are also missing the trick behind markdowns that reference each other and are used for context window management. They go in your source control, they are reusable, extendable, and composable in the same way code is. So planning is not a one-off effort. It compounds over time. If you're doing it right your plan is referencing automation techniques - testing, CI/CD, etc. This further compounds so that the agents verify their own work against the standards that you set. To be honest, the models are getting so good that they do most of this unprompted now.
eikenberry 2mo ago
IMO "get the model to do all the planning work upfront, review and iterate the plan" is backwards. It works better if you do an initial plan yourself and then have the AI review it (and iterate as necessary). Makes you think about the design for a bit so you can have a semblance of a mental model.
- prymitive 2mo ago
  I have never seen a plan written by LLM that wasn’t vague and light on details, every time I need to ask for more details and every time I ask to implement it trips over some dead end in the plan, discards the plan and continues as if there was no plan to begin with. Planning feels often just narrating the request and a wish list then what a human would do - methodically build enough understanding so that you’re confident of the direction you choose. LLM plans are overhyped and overrated.

drTobiasFunke 2mo ago

“Not in speed, but in quality” — unfortunately though, not a single CEO, executive, company, VC, investor or anyone with the power to make decisions cares about quality instead of speed.

consumer451 2mo ago
I agree. Also, what does "code quality" even mean in the age of agentic dev? If the code works, and is secure, what else matters? I do know what good code looks like, but does that even matter anymore? All I know is that now, I get to focus on endless UX polish, which is the only thing the matters. I feel like we are living through something like the Protestant Reformation, where priests once spoke Latin, and then started to speak in plain local language. The old guard did not like this.
- bigstrat2003 2mo ago
  > If the code works, and is secure, what else matters? Maybe so, but it doesn't.
- Silhouette 2mo ago
  Also, what does "code quality" even mean in the age of agentic dev? If the code works, and is secure, what else matters? AI pricing is mostly based on tokens consumed right now. Shouldn't that mean being able to quickly and reliably analyse existing code and to make only small local changes to implement new functionality is as valuable as ever - if not more so - if you're relying on agentic LLMs to do the grunt work? A lot of things about writing clear specs and developing systematically and employing lots of different kinds of checks and controls to ensure quality and performance have always been true but used to get brushed under the carpet by a lot of cheap/lazy development teams. If LLMs really do accelerate everything about development - including negative behaviours like acting undesirably based on flawed or ambiguous information and doubling down on mistaken assumptions - then the pattern across all of these areas is that doing things the right way is more important than ever if you want to get good results from AI assistance.
layer8 2mo ago

vatsachak 2mo ago

I hate saying this but unironically opus 5/gpt 5.6 are writing maintainable code on the order of 500-1000 lines

YZF 2mo ago

In my domain since circa Opus 4.5 LLMs write code as good as most engineers given well defined small enough chunk of work. They refactor. They write tests. These days LLM also debug/troubleshoot better than most engineers. Are they as good as handcrafted code by 0.1% of top software engineers. Generally no. But neither is 99.9% of real code. LLMs also are good at code reviews. What they'll miss is often the big picture but they can still catch plenty of issues. I still want to see a human in the loop in my domain. Totally agree that writing the code was never the bottleneck. We're not seeing massive productivity gains even if some code is written faster. It's not just about understanding but also various other activities that happen in large companies and teams. Also agree LLMs can be used to gain quality but realistically most orgs are going to aim for "fixed or decreasing" quality at lower costs.

luaKmua 2mo ago

This is where I've landed too, I imagine based off all the responses that it depends on domain and also the rigor you care about in your implementation. For me, what I do, and the quality required, there's just not much time (if any) saved on handing off the implementation to an LLM. But that doesn't mean they're useless either. I use them all the time for review as you mentioned or to knock out one-off scripts that don't go anywhere near source control. There's just no world where I don't need to understand every line of code that I'm responsible for getting into our project.

lioeters 2mo ago

> writing was never the bottleneck. Understanding was. Well put. This insight is worth repeating in every discussion on the subject, from software engineering to mathematics. The problem is that understanding is not the product being sold. The business model is for everyone to become consumers of what the magical genie generates, where the "understanding" is kept on the side of the model providers. This ensures a future generation of consumers dependent on someone else to provide the understanding. Otherwise, you can create your own answers based on actually understanding the code, theorem, proofs, etc. Smart consumers of LLMs will use them to increase their own knowledge and understanding, so that the service is augmenting their intelligence, not replacing it.

ManuelKiessling 2mo ago

To be honest, it feels like you are contradicting yourself. If LLMs are, as you stated, really good at catching potential issues, then they are, almost by definition, really good at producing code without potential issues, if guided correctly: all they need to do is inspect and iterate, until they do not find any more potential issue in the code they produced.

layer8 2mo ago
That isn’t necessarily true. LLMs could be good at finding issues that human miss when coding, while being bad at catching issues that they themselves miss while generating code.
booleandilemma 2mo ago
[dead]

nicce 2mo ago

> They really shine on security review, but are very effective with any kind of review. They are getting more and more hostile for making any security assesments. I wonder will they even write secure code in the future if they can’t point vulnerabilities from existing code.

KronisLV 2mo ago

> The other thing that the "LLMs write code camp" misunderstands is that writing was never the bottleneck. Anyone who's seen tech debt where each item is well known and understood but just big in scope knows that this isn't true universally - depending on your team size and composition, any damn thing can be the bottleneck, often at different times too. I'll take everything that helps me resolve them with reasonable trade-offs, even if I need to come up with ways to mitigate the issues created by those trade-offs (like enforcing >90% test coverage as a starting point).