7 ms·

Yes. This happens because the training data contains countless SotA "to-do" apps. This argument does not scale well to other types of software.

baxtr 6mo ago
Isn’t most standard software these days a permutation of things already done before?
- mfabbri77 6mo ago
  In my experience, the further you move away from the user and toward the hardware and fundamental theoretical algorithms, the less true this becomes. This is very true for an email client, but very untrue for an innovative 3D rendering engine technology (just an example).
  - layer8 6mo ago
    An email client is highly nontrivial, due to the complexities of the underlying standards, and how the real implementations you have to be compatible with don’t strictly follow them. Making an email client that doesn’t suck and is fully interoperable is quite an ambitious endeavor.
    - mfabbri77 6mo ago
      The point was to answer the question: "Can every piece of software be viewed as a permutation of software that has already been developed?" In my opinion, an email client is a more favorable example than a 3D engine. In fields where it is necessary to differentiate, improve, or innovate at the algorithmic level, where research and development play a fundamental role, it is not simply a matter of permuting software or leveraging existing software components by simply assembling them more effectively.
      - Archer6621 6mo ago
        Actually, in the specific case of a 3D program it's the current generation of LLM's complete lack of ability in spatial reasoning that prevents them from "understanding" what you want when you ask it to e.g. "make a camera that flies in the direction you are looking at". It necessarily has to derive it from examples of cameras that fly forward that it knows about, without understanding the exact mathematical underpinnings that allow you to rotate a 3D perspective camera and move along its local coordinate system, let alone knowing how to verify whether its implementation functions as desired, often resulting in dysfunctional garbage. Even with a human in the loop that provides it with feedback and grounds it (I tried), it can't figure this out, and that's just a tiny example. Math is precise, and an LLM's fuzzy approach is therefore a bad fit for it. It will need an obscene amount of examples to reliably "parrot" mathematical constructs.
        debugnik 6mo ago
        > "make a camera that flies in the direction you are looking at" That's not the task of a renderer though, but its client, so you're talking past your parent comment. And given that I've seen peers one-shot tiny Unity prototypes with agents, I don't really believe they're that bad at taking an educated guess at such a simple prompt, as much as I wish it were true.
        Archer6621 6mo ago
        You're right. My point was more that LLMs are bad at (3D) math and spatial reasoning, which applies to renderers. Since Unity neatly abstracts the complexity away of this through an API that corresponds well to spoken language, and is quite popular, that same example and similar prototypes should have a higher success rate. I guess the less detailed a spec has to be thanks to the tooling, the more likely it is that the LLM will come up with something usable. But it's unclear to me whether that is because of more examples existing due to higher user adoption, or because of fewer decisions/predictions having to be made by the LLM. Maybe it is a bit of both.
    - School-Cotton 6mo ago
      What complexities specifically? Implementing SMTP (from the client’s perspective) that other SMTP servers can understand is not very hard. I have done it. Does it follow every nuance of the standard? I don’t know, but it works for me. I haven’t implemented IMAP but I don’t see why it should be much harder. Is there a particular example you have in mind?
  - fmbb 6mo ago
    I would be surprised if there are more working email clients out there than working 3D engines. The gaming market is huge, most people do not pay to use email, hobbyists love creating game engines.
    - School-Cotton 6mo ago
      Idk, a working basic email client is just not that hard to write though. SMTP and IMAP are simple protocols and the required graphical interface is a very straightforward combination of standard widgets.
      - isaachinman 6mo ago
        I don't mean to be contrarian, but this is completely false. IMAP _seems_ to be a straightforward (but nasty and stateful) protocol, until you find out that every major provider ignores RFCs and does things slightly differently. It's a hellscape.
- roarcher 6mo ago
  I'd say that's pretty much the definition of standard, yeah. And it's why you can't make a profit selling a simple ToDo app. If you expect people to pay for what you build, you have to build something that doesn't have a thousand free clones on the app store.
  - baxtr 6mo ago
    I politely disagree. I think you’re conflating software and product. A product can be a recombination of standard software components and yet be something completely new.
- deleted 6mo ago
  [deleted]
- loveparade 6mo ago
  But what's the point of re-building "standard software" if it is so standard that it already exists 100 times in the training data with slight variations?
  - ChrisGreenHeur 6mo ago
    The point is the small variations
  - lynx97 6mo ago
    I read this attitude very often on HN. "If someone else has already built it before, your effort is a waste of time." To me, it has this "Someone else already makes money from it, go somewhere else where you dont have competition." Well, I get the drift... But... Not everyone is into getting rich. You know, some of us just have fun building things and learning while doing so. It really doesn't matter if the path has been walked before. Not everything has to be plain novelty to count.
    - loveparade 6mo ago
      If you do it for fun then why do you care whether an LLM can do it well or not, which was the original argument? Shouldn't matter to you in that case.
      - lynx97 6mo ago
        If you dont know the answer to the question you just raised, having a discussion is pretty much useless, because it is so obvious. Just because I have fun building thinks doesn't mean I don't care about the tools to do so?
  - baxtr 6mo ago
    See here: https://news.ycombinator.com/item?id=47435808 https://news.ycombinator.com/item?id=47435808
- layer8 6mo ago
  That isn’t saying much. Every software is a permutation of zeros and ones. The novelty or ingenuity, or just quality and fitness for purpose, can lie in the permutation you come up with. And an LLM is limited by its training in the permutations it is likely to come up with, unless you give it heaps of specific guidance on what to do.
- Gabriel439 6mo ago
  Author here: it's not even clear that agents can reliably permute their training data (I'm not saying that it's impossible or never happens but that it's not something we can take for granted as a reliable feature of agentic coding). As I mentioned in one of the footnotes in the post: > People often tell me "you would get better results if you generated code in a more mainstream language rather than Haskell" to which I reply: if the agent has difficulty generating Haskell code then that suggests agents aren't capable of reliably generalizing beyond their training data. If an agent can't consistently apply concepts learned in one language to generate code in another language, then that calls into question how good they are at reliably permuting the training dataset in the way you just suggested.
  - ChrisGreenHeur 6mo ago
    It can also mean that the other programming language is above the cognitive abilities of the LLM
  - rytis 6mo ago
    > if the agent has difficulty generating Haskell code then that suggests agents aren't capable of reliably generalizing beyond their training data. doesn't that apply to flesh-and-bone developers? ask someone who's only working in python to implement their current project in haskell and I'm not so sure you'll get very satisfying results.
    - debugnik 6mo ago
      But the model has seen pretty much all the public Haskell code around, and possibly been trained to write it in different settings.
    - Frieren 6mo ago
      > doesn't that apply to flesh-and-bone developers? No, it does not. If you have a developer that knows C++, Java, Haskell, etc. and you ask that developer to re-implement something from one language to another the result will be good. That is because a developer knows how to generalize from one language (e.g. C++) and then write something concrete in the other (e.g. Haskell).
- nitwit005 6mo ago
  There's this weird schizophrenia around this in the software world. In the past I have had people here suggest they're just writing boilerplate CRUD software, and I've suggested that means they could just use low code tools instead. They then suggest it's too complex for that to work. I think we tend to view ourselves as just hooking together basic operations, which might be technically true, but that becomes complex very quickly. A product can be built off of straight forward REST and database operations, but take you months of learning to get up to speed on.
wongarsu 6mo ago
Most software written today (or 10 years ago, or 50 years ago) is not particularly unique. And even in that software that is unusual you usually find a lot of run-of-the-mill code for the more mundane aspects
- nostrademons 6mo ago
  Most of the economically valuable software written is pretty unique, or at least is one of few competitors in a new and growing niche. This is because software that is not particularly unique is by definition a commodity, with few differentiators. Commodity software gets its margins competed away, because if you try to price high, everybody just uses a competitor. So goes the AI paradox: it's really effective at writing lots and lots of software that is low value and probably never needed to get written anyway. But at least right now (this is changing rapidly), executives are very willing to hire lots of coders to write software that is low value and probably doesn't need to be written, and VCs are willing to fund lots of startups to automate the writing of lots of software that is low value and probably doesn't need to be written.
  - philipp-gayret 6mo ago
    Could you give some examples? I can only imagine completely proprietary technology like trading or developing medicine. I have worked in software for many years and was always paid well for it. None of it was particularly unique in any way. Some of it better than others, but if you could show that there exists software people pay well for that AI cannot make I would be really impressed. With my limited view as software engineer it seems to me that the data in the product / its users is what makes it valuable. For example Google Maps, Twitter, AirBnB or HN.
    - krethh 6mo ago
      You know how whenever you shuffle a deck of cards you almost certainly create an order that has never existed before in the universe? Most software does something similar. Individual components are pretty simple and well understood, but as you scale your product beyond the simple use cases ("TODO apps"), the interactions between these components create novel challenges. This applies to both functional and non-functional aspects. So if "cannot make with AI" means "the algorithms involved are so novel that AI literally couldn't write one line of them", then no - there isn't a lot of commercial software like that. But that doesn't mean most software systems aren't novel.

baxtr 6mo ago

Isn’t most standard software these days a permutation of things already done before?

mfabbri77 6mo ago
In my experience, the further you move away from the user and toward the hardware and fundamental theoretical algorithms, the less true this becomes. This is very true for an email client, but very untrue for an innovative 3D rendering engine technology (just an example).
- layer8 6mo ago
  An email client is highly nontrivial, due to the complexities of the underlying standards, and how the real implementations you have to be compatible with don’t strictly follow them. Making an email client that doesn’t suck and is fully interoperable is quite an ambitious endeavor.
  - mfabbri77 6mo ago
    The point was to answer the question: "Can every piece of software be viewed as a permutation of software that has already been developed?" In my opinion, an email client is a more favorable example than a 3D engine. In fields where it is necessary to differentiate, improve, or innovate at the algorithmic level, where research and development play a fundamental role, it is not simply a matter of permuting software or leveraging existing software components by simply assembling them more effectively.
    - Archer6621 6mo ago
      Actually, in the specific case of a 3D program it's the current generation of LLM's complete lack of ability in spatial reasoning that prevents them from "understanding" what you want when you ask it to e.g. "make a camera that flies in the direction you are looking at". It necessarily has to derive it from examples of cameras that fly forward that it knows about, without understanding the exact mathematical underpinnings that allow you to rotate a 3D perspective camera and move along its local coordinate system, let alone knowing how to verify whether its implementation functions as desired, often resulting in dysfunctional garbage. Even with a human in the loop that provides it with feedback and grounds it (I tried), it can't figure this out, and that's just a tiny example. Math is precise, and an LLM's fuzzy approach is therefore a bad fit for it. It will need an obscene amount of examples to reliably "parrot" mathematical constructs.
      - debugnik 6mo ago
        > "make a camera that flies in the direction you are looking at" That's not the task of a renderer though, but its client, so you're talking past your parent comment. And given that I've seen peers one-shot tiny Unity prototypes with agents, I don't really believe they're that bad at taking an educated guess at such a simple prompt, as much as I wish it were true.
        Archer6621 6mo ago
        You're right. My point was more that LLMs are bad at (3D) math and spatial reasoning, which applies to renderers. Since Unity neatly abstracts the complexity away of this through an API that corresponds well to spoken language, and is quite popular, that same example and similar prototypes should have a higher success rate. I guess the less detailed a spec has to be thanks to the tooling, the more likely it is that the LLM will come up with something usable. But it's unclear to me whether that is because of more examples existing due to higher user adoption, or because of fewer decisions/predictions having to be made by the LLM. Maybe it is a bit of both.
  - School-Cotton 6mo ago
    What complexities specifically? Implementing SMTP (from the client’s perspective) that other SMTP servers can understand is not very hard. I have done it. Does it follow every nuance of the standard? I don’t know, but it works for me. I haven’t implemented IMAP but I don’t see why it should be much harder. Is there a particular example you have in mind?
- fmbb 6mo ago
  I would be surprised if there are more working email clients out there than working 3D engines. The gaming market is huge, most people do not pay to use email, hobbyists love creating game engines.
  - School-Cotton 6mo ago
    Idk, a working basic email client is just not that hard to write though. SMTP and IMAP are simple protocols and the required graphical interface is a very straightforward combination of standard widgets.
    - isaachinman 6mo ago
      I don't mean to be contrarian, but this is completely false. IMAP _seems_ to be a straightforward (but nasty and stateful) protocol, until you find out that every major provider ignores RFCs and does things slightly differently. It's a hellscape.
roarcher 6mo ago
I'd say that's pretty much the definition of standard, yeah. And it's why you can't make a profit selling a simple ToDo app. If you expect people to pay for what you build, you have to build something that doesn't have a thousand free clones on the app store.
- baxtr 6mo ago
  I politely disagree. I think you’re conflating software and product. A product can be a recombination of standard software components and yet be something completely new.
deleted 6mo ago
[deleted]
loveparade 6mo ago
But what's the point of re-building "standard software" if it is so standard that it already exists 100 times in the training data with slight variations?
- ChrisGreenHeur 6mo ago
  The point is the small variations
- lynx97 6mo ago
  I read this attitude very often on HN. "If someone else has already built it before, your effort is a waste of time." To me, it has this "Someone else already makes money from it, go somewhere else where you dont have competition." Well, I get the drift... But... Not everyone is into getting rich. You know, some of us just have fun building things and learning while doing so. It really doesn't matter if the path has been walked before. Not everything has to be plain novelty to count.
  - loveparade 6mo ago
    If you do it for fun then why do you care whether an LLM can do it well or not, which was the original argument? Shouldn't matter to you in that case.
    - lynx97 6mo ago
      If you dont know the answer to the question you just raised, having a discussion is pretty much useless, because it is so obvious. Just because I have fun building thinks doesn't mean I don't care about the tools to do so?
- baxtr 6mo ago
  See here: https://news.ycombinator.com/item?id=47435808 https://news.ycombinator.com/item?id=47435808
layer8 6mo ago
That isn’t saying much. Every software is a permutation of zeros and ones. The novelty or ingenuity, or just quality and fitness for purpose, can lie in the permutation you come up with. And an LLM is limited by its training in the permutations it is likely to come up with, unless you give it heaps of specific guidance on what to do.
Gabriel439 6mo ago
Author here: it's not even clear that agents can reliably permute their training data (I'm not saying that it's impossible or never happens but that it's not something we can take for granted as a reliable feature of agentic coding). As I mentioned in one of the footnotes in the post: > People often tell me "you would get better results if you generated code in a more mainstream language rather than Haskell" to which I reply: if the agent has difficulty generating Haskell code then that suggests agents aren't capable of reliably generalizing beyond their training data. If an agent can't consistently apply concepts learned in one language to generate code in another language, then that calls into question how good they are at reliably permuting the training dataset in the way you just suggested.
- ChrisGreenHeur 6mo ago
  It can also mean that the other programming language is above the cognitive abilities of the LLM
- rytis 6mo ago
  > if the agent has difficulty generating Haskell code then that suggests agents aren't capable of reliably generalizing beyond their training data. doesn't that apply to flesh-and-bone developers? ask someone who's only working in python to implement their current project in haskell and I'm not so sure you'll get very satisfying results.
  - debugnik 6mo ago
    But the model has seen pretty much all the public Haskell code around, and possibly been trained to write it in different settings.
  - Frieren 6mo ago
    > doesn't that apply to flesh-and-bone developers? No, it does not. If you have a developer that knows C++, Java, Haskell, etc. and you ask that developer to re-implement something from one language to another the result will be good. That is because a developer knows how to generalize from one language (e.g. C++) and then write something concrete in the other (e.g. Haskell).
nitwit005 6mo ago
There's this weird schizophrenia around this in the software world. In the past I have had people here suggest they're just writing boilerplate CRUD software, and I've suggested that means they could just use low code tools instead. They then suggest it's too complex for that to work. I think we tend to view ourselves as just hooking together basic operations, which might be technically true, but that becomes complex very quickly. A product can be built off of straight forward REST and database operations, but take you months of learning to get up to speed on.

wongarsu 6mo ago

Most software written today (or 10 years ago, or 50 years ago) is not particularly unique. And even in that software that is unusual you usually find a lot of run-of-the-mill code for the more mundane aspects

nostrademons 6mo ago
Most of the economically valuable software written is pretty unique, or at least is one of few competitors in a new and growing niche. This is because software that is not particularly unique is by definition a commodity, with few differentiators. Commodity software gets its margins competed away, because if you try to price high, everybody just uses a competitor. So goes the AI paradox: it's really effective at writing lots and lots of software that is low value and probably never needed to get written anyway. But at least right now (this is changing rapidly), executives are very willing to hire lots of coders to write software that is low value and probably doesn't need to be written, and VCs are willing to fund lots of startups to automate the writing of lots of software that is low value and probably doesn't need to be written.
- philipp-gayret 6mo ago
  Could you give some examples? I can only imagine completely proprietary technology like trading or developing medicine. I have worked in software for many years and was always paid well for it. None of it was particularly unique in any way. Some of it better than others, but if you could show that there exists software people pay well for that AI cannot make I would be really impressed. With my limited view as software engineer it seems to me that the data in the product / its users is what makes it valuable. For example Google Maps, Twitter, AirBnB or HN.
  - krethh 6mo ago
    You know how whenever you shuffle a deck of cards you almost certainly create an order that has never existed before in the universe? Most software does something similar. Individual components are pretty simple and well understood, but as you scale your product beyond the simple use cases ("TODO apps"), the interactions between these components create novel challenges. This applies to both functional and non-functional aspects. So if "cannot make with AI" means "the algorithms involved are so novel that AI literally couldn't write one line of them", then no - there isn't a lot of commercial software like that. But that doesn't mean most software systems aren't novel.