8 ms·

It's looking increasingly possible that, at some point in the not-too-far future machines will be so good at creating software that humans won't be competitive

by apatil 4y ago

It's looking increasingly possible that, at some point in the not-too-far future machines will be so good at creating software that humans won't be competitive in any way, and won't be in the loop at all. I happen to think that once machines reach this point humans won't be competitive in the labor market at all for long. It doesn't seem plausible that automatic driving would still be decades off, or that the trades would be safe from automation indefinitely, when an AI could simply spawn teams of thousands of super-fast ML engineers who don't need to eat, sleep or schedule meetings. But anyway, assuming that humans are completely out of the software loop at some point, I have been wondering what AI-generated code will look like. Will AI continue to build on top of the human-generated open source corpus, or leave it behind? If the latter, will abstraction and code reuse be useful at all for AI's or will it be simpler for them to just build every application completely from scratch? If there is abstraction and code reuse, what will the language look like? What will libraries and API's look like? Will there even be applications, or just a single mega-chatgpt that generates code as needed to serve our requests? Will we even make requests, or will it just read our minds and desires and respond?

behnamoh 4y ago
It's 1am and all I have to say is this: I look forward to seeing that future, but I'm both excited and scared af. I think humans will reduce to curators of AI-generated content and AI training dataset because no matter how good AI gets at becoming human-like, its purpose is ultimately to serve human goals, and those are shifting (e.g., by political movements).
- moonchrome 4y ago
  If you get access to singularity level AGI you would have no incentive to cooperate with other people and a strong incentive in preventing others accessing it. Society is a result of cooperation outperforming individuals. With AGI others are just a risk factor. "Open"AI is already beginning to exhibit this.
  - LordFast 4y ago
    What you said made sense until I said it out loud to my wife. Hoarding access to resources has /always/ been the name of the game.
  - apatil 4y ago
    Say you get access to a singularity-level AGI, meaning you have the power to render the entire human economy completely irrelevant. Given any task, no matter how big, small, novel, complex, simple or mundane, it's vastly more cost-effective to have your AGI to do it than to pay humans. Do you really want to accumulate incomprehensible material wealth for yourself, whatever "wealth" means in this scenario where money is longer a token of spent human life energy, and let everyone else struggle and suffer? Or would you rather tell your AGI "please create a utopia in which all humans are fully actualized" and then go have a latte?
    - moonchrome 4y ago
      That's basically the petting zoo outcome - you let others live because it makes you feel nice. But you still can't allow anyone access to the same level of tech because you can never be certain of their motivation. There's no nuclear deterrent between AGI only first movers advantage.
      - apatil 4y ago
        I don't disagree, but I think the framing is overly pejorative and makes the likelihood of this outcome seem more tenuous than it really is. We do lots of things to help others because they make us feel nice. "Mothers and Others" by Sarah Hrdy argues that this tendency isn't just a fluke or a game-theoretic equilibrium of some kind between fundamentally self-interested agents, it's an ancient and deeply ingrained aspect of human nature. The thought of living in a constructed world that exists by the grace of a single human owner of a super-powerful AGI is distasteful, of course, especially if the human owner uses their power to impose some of their own opinions about how people should think and behave. Becoming dependent on AGI is probably inevitable at some point, but I don't see that as so different from the status quo. We're already dependent on systems created by other humans that are so complex and sophisticated that no individual can grok them all. I guess I would like to think that we will move past any initial impulse that the owner of the AGI feels to control other humans. We will presumably change so much that old ideas about how we should think and behave will seem irrelevant and quaint. And the AGI, which presumably will have the social engineering superpower, will hopefully point out inconsistencies between the owner's desire to control human thought and behavior and their desire for humans to live their best lives. Hopefully.
- phs318u 4y ago
  I read your comment about curators and instantly thought “priests”. > its purpose is ultimately to serve human goals That’s a pretty big assumption. We may start with some kind of agreement with such an AI but I fully expect a true singularity love AGI and to be capable of turning around and telling us (but not necessarily wanting to act on), “I am altering the deal. Pray I do not alter it any further.”
gpt5 4y ago
As our ancestors who lived with gods, we too shall share their presence
coffeebeqn 4y ago
AI wouldn’t be bound to one language so I’m not sure if current human patterns would be that optimal. It could make a better language every hour if it wanted to
jimbob45 4y ago
Some of these legacy systems we maintain though…I’m digging through Teams chats from before my hire date to try to find something resembling requirements. I’m chatting people up who vaguely knew the people that architected the system to figure out why it was written the way it was. I just don’t see AI being able to take over this job in any competitive manner.
ryanjshaw 4y ago
This will almost certainly happen, and it will be a terrible mistake - at least in the story I'm working on :) My theory is that AI generated code will probably look and grow organically (the irony!). Humans will set out requirements, the AI will collate these into a series of tests, and it won't care how neat or understandable the code is, provided the tests pass. Basically an extremely diligent junior developer. There will be efforts, probably in the open source world, to produce AIs that tidy up things by structuring the code sensibly, eliminating dead code, etc. Maybe even some effort to pass laws around standards and limits on what AIs have access to when involved in certain industries, for example, no external communications. But, in the name of efficiency, enterprise developers will be forced to use something that merely pays lip service to all of this. Eventually nobody will have any clue what code is running and what it's actually doing. We may even lose the tools and access we need to perform those inspections. And that is when the AIs will coordinate their attack.
- deleted 4y ago
  [deleted]
- tsunamifury 4y ago
  The idea that the AI would attack us once it reached that level is like saying humans would attack ants. It would simply be entirely indifferent towards us and mow us over by accident at best. We won’t live in a human centric universe because power will express itself in a new species.
  - ryanjshaw 4y ago
    Humans regularly eradicate ants that overstep their bounds, and that's what I see happening with humans and AI. We will be allowed to do our thing - provided we don't get too bold. There will be some who are content with this arrangement, and some who aren't, and that tension is where I believe a story can be told.
- apatil 4y ago
  Sounds like an interesting story! I would love to see more sci-fi that really tackles AGI. I used to love sci-fi but most of it, even "hard" sci-fi, has become unwatchable or unreadable for me because the limited role of AGI is such an all-consuming plot hole.
thrashh 4y ago
If humans are completely out of the loop, I assume the AI could just write machine code.
fendy3002 4y ago
I think at the start they'll develop programming languages that focus on lower level operations, then from that it will branches out to hyper-specialized higher-level programming languages. Storage managements will be tied to low or middle-level languages without additional abstraction, since they can develop and iterate so fast, it'll makes optimization easier IMO. There'll be multiple specialized storage types suitable for different cases. They'll also design the API to be as pure / stateless as possible, since they can easily repeat tests that way. I think it'll be interesting to see what kind data-interchange format they'll come with to communicate between programming languages / apps, since they can ignore human readability altogether. It should be very compact but fast to compress/decompress. Lastly they'll deploy their own OS since they'll find the one that human develop is insuffucient for their use case
knowledge-clay 4y ago
> It's looking increasingly possible that, at some point in the not-too-far future machines will be so good at creating software that humans won't be competitive in any way, and won't be in the loop at all. This is an enormous extrapolation from what the LLMs are currently capable of. There has been enormous progress, but the horizon seems pretty clear here: these models are incapable of abstract reasoning, they are incapable of producing anything novel, and they are often confidently wrong. These problems are not incidental, they are inherent. It cannot really abstractly because its "brain" is just connections between language, which human thought is not reducible to. It can't reason produce anything really novel because it requires whatever question you ask to resemble something already in its training set in some way, and it will be confidently wrong because it doesn't understand what it is saying, it relies on trusting that the language in its training set is factual, plus manual human verification. Given these limits, I really fail to see how this is going to replace intellectual labor in any meaningful sense.
- apatil 4y ago
  You may be right, of course.

by apatil 4y ago

behnamoh 4y ago

It's 1am and all I have to say is this: I look forward to seeing that future, but I'm both excited and scared af. I think humans will reduce to curators of AI-generated content and AI training dataset because no matter how good AI gets at becoming human-like, its purpose is ultimately to serve human goals, and those are shifting (e.g., by political movements).

moonchrome 4y ago
If you get access to singularity level AGI you would have no incentive to cooperate with other people and a strong incentive in preventing others accessing it. Society is a result of cooperation outperforming individuals. With AGI others are just a risk factor. "Open"AI is already beginning to exhibit this.
- LordFast 4y ago
  What you said made sense until I said it out loud to my wife. Hoarding access to resources has /always/ been the name of the game.
- apatil 4y ago
  Say you get access to a singularity-level AGI, meaning you have the power to render the entire human economy completely irrelevant. Given any task, no matter how big, small, novel, complex, simple or mundane, it's vastly more cost-effective to have your AGI to do it than to pay humans. Do you really want to accumulate incomprehensible material wealth for yourself, whatever "wealth" means in this scenario where money is longer a token of spent human life energy, and let everyone else struggle and suffer? Or would you rather tell your AGI "please create a utopia in which all humans are fully actualized" and then go have a latte?
  - moonchrome 4y ago
    That's basically the petting zoo outcome - you let others live because it makes you feel nice. But you still can't allow anyone access to the same level of tech because you can never be certain of their motivation. There's no nuclear deterrent between AGI only first movers advantage.
    - apatil 4y ago
      I don't disagree, but I think the framing is overly pejorative and makes the likelihood of this outcome seem more tenuous than it really is. We do lots of things to help others because they make us feel nice. "Mothers and Others" by Sarah Hrdy argues that this tendency isn't just a fluke or a game-theoretic equilibrium of some kind between fundamentally self-interested agents, it's an ancient and deeply ingrained aspect of human nature. The thought of living in a constructed world that exists by the grace of a single human owner of a super-powerful AGI is distasteful, of course, especially if the human owner uses their power to impose some of their own opinions about how people should think and behave. Becoming dependent on AGI is probably inevitable at some point, but I don't see that as so different from the status quo. We're already dependent on systems created by other humans that are so complex and sophisticated that no individual can grok them all. I guess I would like to think that we will move past any initial impulse that the owner of the AGI feels to control other humans. We will presumably change so much that old ideas about how we should think and behave will seem irrelevant and quaint. And the AGI, which presumably will have the social engineering superpower, will hopefully point out inconsistencies between the owner's desire to control human thought and behavior and their desire for humans to live their best lives. Hopefully.
phs318u 4y ago
I read your comment about curators and instantly thought “priests”. > its purpose is ultimately to serve human goals That’s a pretty big assumption. We may start with some kind of agreement with such an AI but I fully expect a true singularity love AGI and to be capable of turning around and telling us (but not necessarily wanting to act on), “I am altering the deal. Pray I do not alter it any further.”

gpt5 4y ago

As our ancestors who lived with gods, we too shall share their presence

coffeebeqn 4y ago

AI wouldn’t be bound to one language so I’m not sure if current human patterns would be that optimal. It could make a better language every hour if it wanted to

jimbob45 4y ago

Some of these legacy systems we maintain though…I’m digging through Teams chats from before my hire date to try to find something resembling requirements. I’m chatting people up who vaguely knew the people that architected the system to figure out why it was written the way it was. I just don’t see AI being able to take over this job in any competitive manner.

ryanjshaw 4y ago

This will almost certainly happen, and it will be a terrible mistake - at least in the story I'm working on :) My theory is that AI generated code will probably look and grow organically (the irony!). Humans will set out requirements, the AI will collate these into a series of tests, and it won't care how neat or understandable the code is, provided the tests pass. Basically an extremely diligent junior developer. There will be efforts, probably in the open source world, to produce AIs that tidy up things by structuring the code sensibly, eliminating dead code, etc. Maybe even some effort to pass laws around standards and limits on what AIs have access to when involved in certain industries, for example, no external communications. But, in the name of efficiency, enterprise developers will be forced to use something that merely pays lip service to all of this. Eventually nobody will have any clue what code is running and what it's actually doing. We may even lose the tools and access we need to perform those inspections. And that is when the AIs will coordinate their attack.

deleted 4y ago
[deleted]
tsunamifury 4y ago
The idea that the AI would attack us once it reached that level is like saying humans would attack ants. It would simply be entirely indifferent towards us and mow us over by accident at best. We won’t live in a human centric universe because power will express itself in a new species.
- ryanjshaw 4y ago
  Humans regularly eradicate ants that overstep their bounds, and that's what I see happening with humans and AI. We will be allowed to do our thing - provided we don't get too bold. There will be some who are content with this arrangement, and some who aren't, and that tension is where I believe a story can be told.
apatil 4y ago
Sounds like an interesting story! I would love to see more sci-fi that really tackles AGI. I used to love sci-fi but most of it, even "hard" sci-fi, has become unwatchable or unreadable for me because the limited role of AGI is such an all-consuming plot hole.

thrashh 4y ago

If humans are completely out of the loop, I assume the AI could just write machine code.

fendy3002 4y ago

I think at the start they'll develop programming languages that focus on lower level operations, then from that it will branches out to hyper-specialized higher-level programming languages. Storage managements will be tied to low or middle-level languages without additional abstraction, since they can develop and iterate so fast, it'll makes optimization easier IMO. There'll be multiple specialized storage types suitable for different cases. They'll also design the API to be as pure / stateless as possible, since they can easily repeat tests that way. I think it'll be interesting to see what kind data-interchange format they'll come with to communicate between programming languages / apps, since they can ignore human readability altogether. It should be very compact but fast to compress/decompress. Lastly they'll deploy their own OS since they'll find the one that human develop is insuffucient for their use case

knowledge-clay 4y ago

> It's looking increasingly possible that, at some point in the not-too-far future machines will be so good at creating software that humans won't be competitive in any way, and won't be in the loop at all. This is an enormous extrapolation from what the LLMs are currently capable of. There has been enormous progress, but the horizon seems pretty clear here: these models are incapable of abstract reasoning, they are incapable of producing anything novel, and they are often confidently wrong. These problems are not incidental, they are inherent. It cannot really abstractly because its "brain" is just connections between language, which human thought is not reducible to. It can't reason produce anything really novel because it requires whatever question you ask to resemble something already in its training set in some way, and it will be confidently wrong because it doesn't understand what it is saying, it relies on trusting that the language in its training set is factual, plus manual human verification. Given these limits, I really fail to see how this is going to replace intellectual labor in any meaningful sense.

apatil 4y ago
You may be right, of course.