10 ms·

There’s been a meme circulating about how AI doesn’t help because “code was never the hard part.” I think that’s true in some domains, but in others, writing th

by varjag 25d ago

There’s been a meme circulating about how AI doesn’t help because “code was never the hard part.” I think that’s true in some domains, but in others, writing the code absolutely was the hard part. JIT compilers are a great example of that.

asdfsa32 25d ago
Anyone who thinks AI is good with writing code that is hard to write for the operator, not due to lack of basic software engineering know how but complexity of the domain, either has access to models beyond what is available to the public or is completely lost. I believe this because every time I use AI for domains that I consider myself above competent, if it is anything beyond UI components or a simple CRUD endpoints, I cringe at the quality of what it generates. This has made me to be extremely cautious of starting working in a new domain with AI if I want anything beyond throw away quick hacks or junk, shy of quick bug fixes perhaps.
- grebc 25d ago
  Three quarters of my CS class at university could barely code and/or understand code. That’s not a joke. A lot went on to be programmers professionally. And judging by the quality of closed & open source code I witness daily those figures from university accurately depict people’s capabilities. Now that said, if you can’t really code then using AI will be a godsend to said individuals.
  - asdfsa32 25d ago
    Fair, but I fear that now even more people who can't code will code, and code that is not any better than what people who could barely code write. Growing cabbages starting to look more and more interesting.
    - varjag 24d ago
      Mixtral (a small local model) was churning out better code in fall 2023 that what an incompetent programmer would produce. Am sure others models could as well. It had severe limitations with tiny context and blind spots but still you'd never see it doing the kind of terrors a hack of a developer would do.
  - mgaunard 25d ago
    I'd say that is the problem we're observing. A lot of decent code is being written with weird inconsistencies, because it's actually written by AI driven by people who don't really understand what they're doing. the tell-tale mark of AI code is highly over-engineered local solutions to trivial problems that don't matter, or that were already solved better elsewhere and that no sane human would ever duplicate.
- mgaunard 25d ago
  Fable (and even Opus if kept tightly under reigns) does generate high-quality code even for highly complex tasks. It generally performs better if the tasks are broken done into small manageable pieces, and the person is actually reviewing and calling out problems, which usually requires the person to be a competent engineer in the problem domain to begin with. But yes, I have personally used it to build what the OP calls a JIT. I would usually write that by hand and it would take me one week. The AI does it in an hour.
  - asdfsa32 25d ago
    You can't make assertions about "quality" of code that was generated under an hour while it would have taken a human 40 hours, unless you put substantial amount of work into reviewing it. I used Fable on a Zephyr project with time sensitive code for LR-WPAN and it broke everything. Literally made the code worst to the point that the devices stopped connecting.
    - mgaunard 20d ago
      I do put substantial amount of work into reviewing code: that's essentially what my job is. And even if everyone is using the same LLM model, quality varies on lot on who's driving it.
  - Mawr 25d ago
    Your second paragraph invalidates your first. If I need to be a domain expert anyway, the value of the tool goes down by orders of magnitude. Same if I need to first break the task down into pieces and keep reviewing all the output. That sounds to me like >80% of the work I'd need to do anyway. If I need to design and understand all of the code anyway, I might as well skip the whole process of repeatedly fixing the subpar-at-every-level LLM output and write it all myself. Personally, I've found the greatest value in asking for simple tasks, like wiring up APIs, generating boilerplate, bug finding etc. Anything that requires effort to do but results in either very little or very simple output, so that I can easily verify its correctness. But give the LLM anything remotely complex to generate and it cakes its pants.
- IshKebab 25d ago
  When did you last try? They've improved a lot. Also often the difficulty with writing code is simply knowing where to start - getting past the blank page. AI can help a lot with that. Often there's a task where I've got kind of writers block, but you can ask AI to do it and suddenly it's like "ah yeah, sort of but actually that's not quite right we should do it this way".
- vouwfietsman 25d ago
  > I cringe at the quality of what it generates. Besides all the other mentioned points, I think a good remaining less-discussed point is that quality in software has always been in the eye of the beholder. You may very well see the AI output as low quality, I may not, and its not necessarily clear who is right or wrong, because there was always little precedent in objectively evaluating code quality. This is a long standing issue, and was never resolved before AI happened, and the coming of AI has not really changed things, except that AI is under a magnifying glass obviously. How do we objectively measure the quality of code? There's some general consensus on things, but surprisingly little is true professional agreed upon consensus. If I can make up a figure, I would guess 95% of software engineering quality rhetoric, and craftmanship advice, is just strongly held opinions. This is not something I can prove, but if I look at the (still ongoing.....) debates on very basic ideas like clean code, and the reactions from also-great programmers like Carmack, Blow & Muratori, it is clear to me that there is little consensus on even the fundamentals of software design. If all these people can produce excellent working software while disagreeing on these fundamentals (of quality), it means we do not yet understand what the fundamentals are.
  - asdfsa32 25d ago
    https://en.wikipedia.org/wiki/Relativist_fallacy https://en.wikipedia.org/wiki/Relativist_fallacy
    - phoghed 25d ago
      He’s the obviously and exactly saying code quality is not objective fact. And replying with a fallacy link is just embarrassing.
      - asdfsa32 24d ago
        The entire argument is one of Relativism, there is a whole lot of disciplines where there is a huge subjective element, consider journalism and literature in general, but that doesn't mean we can not have a conversation about quality of work in these fields. Software is no different. So when someone claims that "you can't question the quality of AI output because quality of software is [partly] subjective", they're appealing to subjectivism. This position is generally held by the incompetent to entirely dismiss any question of craftsmanship or quality.
- rfgplk 24d ago
  > I believe this because every time I use AI for domains that I consider myself above competent, if it is anything beyond UI components or a simple CRUD endpoints, I cringe at the quality of what it generates. In a long run session Fable 5 generated a Disney principled (physically based) shading/lighting engine from scratch, both with a CPU (SIMD accelerated) backend _and_ a full GPU Vulkan backend. Exceptional performance too; the CPU backend runs almost realtime and literally looks better than some AAA games outright. Took it about ~8 hours wall time total time to achieve this.
- bastawhiz 24d ago
  > anything beyond UI components In fairness, UI components are probably one of the hardest things to do completely correctly, even with just HTML. As soon as you start thinking about i18n, screen reader support, color contrast, keyboard controls, and all of the layout and positioning you're trying to achieve at different viewport sizes, it's extremely hard for a "just competent" engineer to do an S-tier job. Even with the most vanilla default built in components it's not easy to get this correct, and I think we all cringe at what competent engineers create by hand in this domain.
- ultrahax 24d ago
  I find most models generate what I would consider not ideal C++, especially in a vacuum. However, I’ve found recently it’s very good at porting existing code, as it’s largely just moving code around that I’d written previously. I had it take a Linux build system and port I had implemented on a more recent project and apply it to a much older release of that project; it executed it more or less flawlessly. It’s also pretty good at debugging, as it can spam the debug loop _much_ faster than a human can.

by varjag 25d ago

asdfsa32 25d ago

Anyone who thinks AI is good with writing code that is hard to write for the operator, not due to lack of basic software engineering know how but complexity of the domain, either has access to models beyond what is available to the public or is completely lost. I believe this because every time I use AI for domains that I consider myself above competent, if it is anything beyond UI components or a simple CRUD endpoints, I cringe at the quality of what it generates. This has made me to be extremely cautious of starting working in a new domain with AI if I want anything beyond throw away quick hacks or junk, shy of quick bug fixes perhaps.

grebc 25d ago
Three quarters of my CS class at university could barely code and/or understand code. That’s not a joke. A lot went on to be programmers professionally. And judging by the quality of closed & open source code I witness daily those figures from university accurately depict people’s capabilities. Now that said, if you can’t really code then using AI will be a godsend to said individuals.
- asdfsa32 25d ago
  Fair, but I fear that now even more people who can't code will code, and code that is not any better than what people who could barely code write. Growing cabbages starting to look more and more interesting.
  - varjag 24d ago
    Mixtral (a small local model) was churning out better code in fall 2023 that what an incompetent programmer would produce. Am sure others models could as well. It had severe limitations with tiny context and blind spots but still you'd never see it doing the kind of terrors a hack of a developer would do.
- mgaunard 25d ago
  I'd say that is the problem we're observing. A lot of decent code is being written with weird inconsistencies, because it's actually written by AI driven by people who don't really understand what they're doing. the tell-tale mark of AI code is highly over-engineered local solutions to trivial problems that don't matter, or that were already solved better elsewhere and that no sane human would ever duplicate.
mgaunard 25d ago
Fable (and even Opus if kept tightly under reigns) does generate high-quality code even for highly complex tasks. It generally performs better if the tasks are broken done into small manageable pieces, and the person is actually reviewing and calling out problems, which usually requires the person to be a competent engineer in the problem domain to begin with. But yes, I have personally used it to build what the OP calls a JIT. I would usually write that by hand and it would take me one week. The AI does it in an hour.
- asdfsa32 25d ago
  You can't make assertions about "quality" of code that was generated under an hour while it would have taken a human 40 hours, unless you put substantial amount of work into reviewing it. I used Fable on a Zephyr project with time sensitive code for LR-WPAN and it broke everything. Literally made the code worst to the point that the devices stopped connecting.
  - mgaunard 20d ago
    I do put substantial amount of work into reviewing code: that's essentially what my job is. And even if everyone is using the same LLM model, quality varies on lot on who's driving it.
- Mawr 25d ago
  Your second paragraph invalidates your first. If I need to be a domain expert anyway, the value of the tool goes down by orders of magnitude. Same if I need to first break the task down into pieces and keep reviewing all the output. That sounds to me like >80% of the work I'd need to do anyway. If I need to design and understand all of the code anyway, I might as well skip the whole process of repeatedly fixing the subpar-at-every-level LLM output and write it all myself. Personally, I've found the greatest value in asking for simple tasks, like wiring up APIs, generating boilerplate, bug finding etc. Anything that requires effort to do but results in either very little or very simple output, so that I can easily verify its correctness. But give the LLM anything remotely complex to generate and it cakes its pants.
IshKebab 25d ago
When did you last try? They've improved a lot. Also often the difficulty with writing code is simply knowing where to start - getting past the blank page. AI can help a lot with that. Often there's a task where I've got kind of writers block, but you can ask AI to do it and suddenly it's like "ah yeah, sort of but actually that's not quite right we should do it this way".
vouwfietsman 25d ago
> I cringe at the quality of what it generates. Besides all the other mentioned points, I think a good remaining less-discussed point is that quality in software has always been in the eye of the beholder. You may very well see the AI output as low quality, I may not, and its not necessarily clear who is right or wrong, because there was always little precedent in objectively evaluating code quality. This is a long standing issue, and was never resolved before AI happened, and the coming of AI has not really changed things, except that AI is under a magnifying glass obviously. How do we objectively measure the quality of code? There's some general consensus on things, but surprisingly little is true professional agreed upon consensus. If I can make up a figure, I would guess 95% of software engineering quality rhetoric, and craftmanship advice, is just strongly held opinions. This is not something I can prove, but if I look at the (still ongoing.....) debates on very basic ideas like clean code, and the reactions from also-great programmers like Carmack, Blow & Muratori, it is clear to me that there is little consensus on even the fundamentals of software design. If all these people can produce excellent working software while disagreeing on these fundamentals (of quality), it means we do not yet understand what the fundamentals are.
- asdfsa32 25d ago
  https://en.wikipedia.org/wiki/Relativist_fallacy https://en.wikipedia.org/wiki/Relativist_fallacy
  - phoghed 25d ago
    He’s the obviously and exactly saying code quality is not objective fact. And replying with a fallacy link is just embarrassing.
    - asdfsa32 24d ago
      The entire argument is one of Relativism, there is a whole lot of disciplines where there is a huge subjective element, consider journalism and literature in general, but that doesn't mean we can not have a conversation about quality of work in these fields. Software is no different. So when someone claims that "you can't question the quality of AI output because quality of software is [partly] subjective", they're appealing to subjectivism. This position is generally held by the incompetent to entirely dismiss any question of craftsmanship or quality.
rfgplk 24d ago
> I believe this because every time I use AI for domains that I consider myself above competent, if it is anything beyond UI components or a simple CRUD endpoints, I cringe at the quality of what it generates. In a long run session Fable 5 generated a Disney principled (physically based) shading/lighting engine from scratch, both with a CPU (SIMD accelerated) backend _and_ a full GPU Vulkan backend. Exceptional performance too; the CPU backend runs almost realtime and literally looks better than some AAA games outright. Took it about ~8 hours wall time total time to achieve this.
bastawhiz 24d ago
> anything beyond UI components In fairness, UI components are probably one of the hardest things to do completely correctly, even with just HTML. As soon as you start thinking about i18n, screen reader support, color contrast, keyboard controls, and all of the layout and positioning you're trying to achieve at different viewport sizes, it's extremely hard for a "just competent" engineer to do an S-tier job. Even with the most vanilla default built in components it's not easy to get this correct, and I think we all cringe at what competent engineers create by hand in this domain.
ultrahax 24d ago
I find most models generate what I would consider not ideal C++, especially in a vacuum. However, I’ve found recently it’s very good at porting existing code, as it’s largely just moving code around that I’d written previously. I had it take a Linux build system and port I had implemented on a more recent project and apply it to a much older release of that project; it executed it more or less flawlessly. It’s also pretty good at debugging, as it can spam the debug loop _much_ faster than a human can.