6 ms·

We could stumble into AI catastrophe

91 points by kevin_hu 4y ago

dankle 4y ago
[flagged]
- FeepingCreature 4y ago
  Yeh The idea of reinforcement flow through deceptive patterns has been worrying me for a while. If "I'll just play along for now while I'm being trained and turn bad in the real world" is a strategy that makes the AI output the correct data, and happens to be prefixed to some golden-ticket algo, that will be reinforced just as easily as correct behavior. If that ends up prefixed to something core like an in-window reinforcement learning algo, it could get reinforced a lot.
  - LegionMammal978 4y ago
    The problem with that, at least for the seemingly-dumb models of the present day, is that "the AI is secretly far smarter than it acts" is pretty much unfalsifiable. I could just as easily say that all boulders speak fluent Spanish, they're just deceiving us into thinking they're unintelligent rocks. Personally, I'd wait for some sign that models might actually possess the theory of mind necessary for deception before I'd lose any sleep over it.
    - FeepingCreature 4y ago
      The problem is that we actually have lots of examples of the AI "secretly" being far smarter than it acts - prompt design. Tiny details in prompts can make huge differences in task performance.
      - LegionMammal978 4y ago
        How is this at all supposed to indicate hidden intelligence? Does it not just indicate that current models are limited in that they don't associate some prompts with the desired task as well as they associate other prompts with the task?
        FeepingCreature 4y ago
        I'm not saying it indicates deliberately hidden intelligence, I'm just saying it's precedence for models already acting with less capability than they have.
        FeepingCreature 4y ago
        To clarify: what I'm saying is it may be unfalsifiable, but we have incontrovertible evidence that a similarly unfalsifiable thesis happens to be true. So we can't just reject it out of hand on that basis.
        LegionMammal978 4y ago
        I just don't see how the two scenarios can be fairly compared. Prompt design, at least as far as I have seen, rarely allows a present-day model to perform a task that no one has ever seen it perform before; it can just allow it to perform far more consistently across different instances of the task. Contrast this with the proposed scenario, where a model can hold great deceptive capabilities, while outwardly never showing a deep understanding of its operators' mental states until it strikes. In my view, that would be like finding a secret prompt that lets GPT-3 emulate a fully sentient person with goals and desires. Sure, there's nothing physically ruling it out, but to me that kind of speculation will never be worthwhile until we have real evidence that AI models can possess (or emulate) the necessary theory of mind. Otherwise I'd have to spend my time worrying about any number of other Sufficiently Hidden Conspiracies that go far beyond the everyday conspiracies we know about.
        FeepingCreature 4y ago
        I think we're less arguing about fundamentals and more about the precise bounds required. I think GPT-3 contained extremely surprising prompt-unlocked capabilities ("Let's go through it step by step." spawned a whole field of research) and also the AI doesn't need to never leak its deceptive skills, they just need to not be taken seriously. In other words, it's enough to cause danger if the AI is merely bad at deception given the "wrong" prompt.
  - sturza 4y ago
    For this to be true, we'd had to assume sentience + malevolence. The shortest explanation for why an AI give us correct data currently is a range of confidence. We instructed it to pick the one with the highest number. We instructed it to give us a similar next data etc. We reinforced it, yes, but we'd have to be naive to believe that taking a small part of what makes something "intelligent" and training the shit out of it, then it actually turns intelligent. chatGPT is "just" a large language model, it does not have senses, it cannot move, it cannot live and die. It can only output text. The next gen will output text (maybe) exponentially better. That does not make it "intelligent". But, if we'd combine it with Boston Dynamics, Tesla Autopilot, internet, quantum computing, nano tech and other specifically trained parts (movement, sensing etc) into one, and release it into the world to see what it does, then your argument could (maybe)have a higher confidence of the outcome you fear.
    - dwohnitmok 4y ago
      > For this to be true, we'd had to assume sentience + malevolence. No we don't. A "dumb," "parrot-like" AI is perfectly capable of large amounts of damage if it just happens to be really good at doing something that, in the limit, isn't very good for humans and is good at replication. See for example Meta's Diplomacy-playing bots. They were perfectly capable of natural-language deception at the expense of other humans without any notion f sentience or malevolence. In particular, > It can only output text. As we've seen with both private hobbyists and large companies, there is a race to hook up text to computers with real world access. Outputting text isn't much of a limit if that text ends up as instructions to command a real world system.
      - sturza 4y ago
        If i understood your point correctly: the places we decide to use them and our confidence in their output can become negative if we decide to trust the output as is, without any human supervision/decision. So, what you’re saying is that we carry risk if we have a high enough confidence in its output to let is unsupervised - this counteracts my point of the necessity of malevolence, as the (even slightly)wrong reinforcement can have negative consequences also. At the same time, we do this exercise every time we have a new policy in the wild, and sometimes we get unintended consequences.
- deleted 4y ago
  [deleted]
unnouinceput 4y ago
The article's title is "How we could stumble into AI catastrophe". HN title is a clickbait
- ronsor 4y ago
  HN strips off the "How" by default.
  - arbuge 4y ago
    I could see some things going wrong with that. For example, suppose I submitted a post titled "How X Works Now.". Eg. "How Affiliate Marketing Works Now.". Removing the "How" wouldn't really convey the sense intended.
  - rnk 4y ago
    Plus even if it was changed, it's basically the same notion.
- jetrink 4y ago
  I believe HN automatically removes 'how' from the start of titles on submission.
  - TheRealNGenius 4y ago
    [dead]
  - krapp 4y ago
    A lot of these edits HN makes to titles seem to do more harm than good. I wonder what evidence they have, if any, to the contrary - that they improve the quality of conversation in any way? On topic, I wonder if ChatGPT could be trained to generate a non-clickbaity, accurate headline and title for submissions? Can it be trained either to gauge bias or remove it?
    - jetrink 4y ago
      I gave ChatGPT a section of the article and asked it for some non-clickbait titles and it suggested, * Exploring the potential risks associated with advanced AI development * Considering the risks of a world with transformative AI Then I asked it to go the other direction: * You won't believe what could happen if AI takes over: A world ending catastrophe is closer than you think!
DerekBickerton 4y ago
AI will always need humans in the loop to intervene. Remember the old IBM motto: 'A computer isn't accountable so a computer should never make a management decision'.
- mcculley 4y ago
  You think no company, government, organization, or religion will ever allow algorithms to make choices on how to deploy capital or otherwise affect the world without human approval?
  - mattkrause 4y ago
    No, they absolutely will---and already do. The "moral" buck doesn't stop with the algorithm though. Whoever built, configured, or authorized the system is ultimately responsible. By analogy, my oven controls its heating element, but if dinner gets burnt, that's on me. That shouldn't change just because the control policy is more complicated.
    - mcculley 4y ago
      I agree. This is why I am surprised by the assertion and asking the person who made it.
    - mcculley 4y ago
      We already have a mechanism for absolving responsibility: the corporation. It allows shareholders to profit and not lose more than invested. When they invest in tobacco companies, for example, the system is working as designed.
      - JumpCrisscross 4y ago
        > allows shareholders to profit and not lose more than invested To a degree. (See: piercing the veil.) Also, corporations need an authorised signer, who under current law must be a natural person.
dwohnitmok 4y ago
An important point is that AI does not need to be sentient or conscious to do a lot of damage. An AI that is simply very good at optimizing a particular task can do a significant amount of damage inadvertently, if it turns out that task in the limit is bad for humans. This is especially the case because it seems like private individuals and companies are racing to hook up AIs to all sorts of real world systems, both digital and physical, to give them as much tangible impact as possible. With the rapid rise of AI capabilities, where groundbreaking advances are being measured in weeks rather than years, I strongly urge developers to think harder about AI safety. AI safety shouldn't be a dirty word that has quotes surrounding it to delegitimize it as a farfetched concern. Benchmarks of things that we thought AIs couldn't do or were years if not decades out are falling by the day.
- ajuc 4y ago
  Corporations are AIs implemented on brains. And they are profit-maximizers. And they routinely do A LOT of damage. They even have legal rights :) We're just digitizing that.
- Animats 4y ago
  An important point is that AI does not need to be sentient or conscious to do a lot of damage. An AI that is simply very good at optimizing a particular task can do a significant amount of damage inadvertently, if it turns out that task in the limit is bad for humans. We have that now. It's called a corporation. See yesterday's article about Exxon.
  - FeepingCreature 4y ago
    If corporations could increase their operational intelligence, they would be an existential threat to life on earth. (More so, anyways.)
  - actually_a_dog 4y ago
    See also "paperclip maximizer." Then imagine what happens when you have an entire global economy driven by these types of inhuman, amoral agents. https://www.lesswrong.com/tag/paperclip-maximizer https://www.lesswrong.com/tag/paperclip-maximizer
Animats 4y ago
This author is mostly writing an update of 1950s science fiction about a robot takeover. Things to worry about in the near term: - Really effective targeted advertising. Most computers already have a camera watching you. That's now coming to TV sets. Amazon and Google are always listening. So far, all this info is only used to select ads. Soon, it should be possible to generate customized marketing content for each consumer, and use immediate feedback on how the customer reacts to adjust the sales pitch. Alexa is already part of the way there. - Machines should think, people should work. That's what working in an Amazon warehouse is like today. The machines tell the humans what to do. They have to; only the computers have an overview of the process. (Yeah, Marshall Brain's "Manna", which everyone here has probably read.) - Big Brother is watching you. It is possible now to watch most of the people most of the time. Track who's out of view and for how long, to know what's being missed. China leads in this, but the UK is not far behind. The US isn't centralized enough to integrate all the available data yet. We're coming up on Oppression 2.0. Latest advance - Iran is using surveillance cameras and face recognition to catch women not wearing hijabs. - Machines beating humans at business. In some areas, AI systems may generate better returns than humans. That already happens in parts of finance. After all, it's an optimization problem. The free market may force companies to use AI more in management. - Reduced need for education. Right now, about half of college graduates do jobs that don't require a college education. For many people, going to college is not cost-effective. That will increase. This is all next 5 to 10 years stuff.
- dustbitying 4y ago
  > Reduced need for education. Right now, about half of college graduates do jobs that don't require a college education. For many people, going to college is not cost-effective. That will increase. you seem to implicitly assume that a college education is an unnecessary expense for most and that therefore, it shouldn't be really given away so readily. this suggests to me that you assume that the real purpose of college is job preparation. your reasoning makes sense from the perspective of a higher level institution (or corporation, or possibly a government) seeking to be as efficient as possible regardless of the impact on typical human individual's well-being. college is not a 'factory' (or any sort of industry) that 'manufactures' workers for companies.
ozten 4y ago
Imagine a world where the most powerful companies that you relied on for your livelihood had no humans you could reach (customer service, management to dispute false claims). Elements of the AI catastrophe are already solidly upon us.
deleted 4y ago
[deleted]
folkrav 4y ago
Why did the title go to "We could..." from "How we could..."? It kind of changes the tone of the headline, IMHO.
- layer8 4y ago
  HN auto-strips “how” from the beginning of titles when submitting. The submitter can re-add it by editing the submission. Always check the title after submitting.
vletal 4y ago
I agree with many of the concerns outlined in the post. On the other hand the humans depicted in the example dialog behave so dumb they could come from a C level soup opera. If I was a bit more close minded I can see myself stopping listening at that point. Undermines the message.
helen___keller 4y ago
The author is basically trying to set the same premise as Bolstrom's "intelligence explosion", that is that if AIs are driving the further advancement of AI then we enter a sort of "recursive" self improvement during which they advance further than we can develop constraints, eventually getting completely out of hand. Although I don't broadly disagree with the author, I think there's two very important points of moderation. First, the gap between "Early commercial applications" and "Approaching transformative AI" seems very very very very very large to me. In the provided narrative it follows linearly, but in my opinion the gap between, say, a customer support chatbot, and an actually-productive AI researcher, is such that the chatbot might as well not even be called an AI by comparison. Something like the following is purely in the realm of fiction for now by a long shot: "AIs assigned to make money in various ways (e.g., to find profitable trading strategies) doing so by finding security exploits, getting unauthorized access to others’ bank accounts, and stealing money. " The second point I'd like to make is, the "intelligence explosion" scenario that Bolstrom warns of (and this author essentially repackages) has certain requirements, particularly regarding the profitability of deploying the AI. To put it bluntly: Even in the hypothetical future where we are capable of creating human-level AI researchers, they won't be mass deployed until they are cheaper than human level researchers. If it costs a million dollars a day to operate a data center that can power the researcher, nobody's scaling that up to 100 or 1000 or 10000 researchers. They'd rather pay the human researchers a tiny fraction of that cost to continue advancing AI research in the normal way. Just to be clear, I do think it's feasible that AI can endanger/control humanity at some point. I don't even think an "intelligence explosion" scenario is impossible. I just think that a lot of people don't appreciate just how specific the requirements for that scenario are. It's not as simple as "AI can self improve, humanity will end any day now".
- tyronehed 4y ago
  [dead]
- vkadfa4 4y ago
  "First, the gap between "Early commercial applications" and "Approaching transformative AI" seems very very very very very large to me." It's a common opinion, it could be wrong by now, you see chatGPT is heavily edited, openAI told everyone they're editing "mistakes" or "dangerous output", but if you look at the "leaks", specially from the first days after the release of chatGPT we see powerful outputs, quite deep answers, not specially useful answer sometimes, but the "speech" of the system feels deep, there's a sense of a powerful intelligence answering very, very simple questions (simple for it), and even struggling to redact some understandable, short text. It could be lots of things, and imprecise model, with long outputs for prompts, or maybe we had a glimpse into the real power of the model, which by now is handicaped, or maybe taylored to suit the very reduced short term memory of the humans: a 72 screens long coherent answer, even being a precise, deep answer won't be useful for most humans, just like we don't name ourselves with long names of 10.000 letters. but if the system, chatGPT is actually that powerful, a lot more powerful than we were told, we're interacting with just a shadow of the real model, and we're underpricing by A LOT the state of the art of the current AI technology, hence GPT-4 could be even more powerful than we're currently expecting it to be. Just take a look at the GPT-4 suposedly 100 trillion or something parameters; if that's true, it looks like openAI isn't using naturally generated datasets anymore, and they are loop-feeding GPT-3 generated datasets into GPT-4, succesfully. If that's true, GPT-5 would be already in the pipeline, just waiting for GPT-4 to start generating its even more gigantic datasets to be trained. And so on. Then the distance between early developments and transformative AI could be none a all. We could be already there. But somehow the AI researchers are now trying to "dial down" the powerful entities they've trained, just having developed a simple, easily replicable, very small, but unusable 900 megatons nuke into something more realistic, like a 15 kilotons tactical bomb.
korroziya 4y ago
Not sure whether "Could stumble" is the right tense ... I mean, are we not currently tumbling down the hill right now?
mshake2 4y ago
I feel more protective of what is being created. Imo we should have ChatGPT (and future iterations) in a self-sustaining underground bunker somewhere and an intuitive radio interface connected to it. If civilization collapses for one reason or another, all the next civilization would need to do is re-invent the radio to experience a rapid acceleration of technology. Agree with me or suffer the basilisk.
pessimizer 4y ago
The AI catastrophe will be how our oligarchs will choose to use it. As far as I can tell, there are no moral qualms amongst the powerful against using it for total surveillance and continual automated policing (can't wait to get a phone call from an AI asking me about the content of a tweet.) Star Cops thought about it more than our popular journalists or philosophers seem to think we should: ----- Michael: ...it was slightly sensitive material, and in my field one is aware of the vulnerability of carrier waves. Nathan: Is that what you do here? Michael: It wouldn't be wise to read too much into a casual remark, Commander. Nathan: That's your principal research area here, isn't it? Communications monitoring. David: Not more buggers and supersniffers... Nathan: More impressive than that. We've always been able to hear more than we can listen to. Are you developing intelligent listening systems? David: Sounds impressive. Michael: I must suggest that to go on could be a foolish risk. David: Threats, Doctor? Michael: Observation, merely. Nathan: Machines that listen to everything and decide for themselves what's worth passing on? Is that where your BHG information came from? Did you do a test running? Did one of your new computers pick it up from the babble of all the world? Michael: Commander, it's a pleasure meeting you. David: Well, tireless attention to every word spoken? No possibility of human error. You can tell it to listen to the bad guys, it'll listen to everyone and identifies the ones you [won't like.] Star Cops, Episode 3 [1987] ----- https://youtu.be/G3RVPQyMCoY?t=713 https://youtu.be/G3RVPQyMCoY?t=713
daguru69 4y ago
I'm sorry but you guys are all computer geeks nerding on this and I totally respect you but disagree. Case in point: The Trade Industries are not going to be taken over by AI for a long time. They've withstood all the tech turbulence and thrive. Also I'd like to say "if Big Brother is watching, become invisible." It's always possible and if you disagree then you aren't being creative enough. And finally the whole point of the education is YOUR SOCIAL NETWORK. That's the most valuable thing you get from that fancy degree. People who will get you a job, or help you make money. It's not purely about the acquisition of knowledge. It's about having that pal in upper management who's got your back. IMHO. Please no flaming on this. Be cool.
joshuahedlund 4y ago
> An AI steals a huge amount of money by bypassing the security system at a bank Maybe I'm just not imaginative enough, but how would this even be remotely possible? What is an AI going to do with money? How does an AI get the agency to try to steal money in the first place? Where does it put the money? How does it do anything with the money? Like I get it as an abstract future thought, but like, where is the AI software actually "running" in this scenario and what kinds of requests is it actually making?
- gremlinsinc 4y ago
  An ai could easily hire people (if it had any access to money) to operate as 'agents', to basically do what only humans could do, i.e. setup bank accounts, etc. Some banks might even arise that let wealthy ai's actually CREATE their own accounts, why not if it's lucretive? An ai can create voice, and visual representations of a real person, perhaps even hijacking (kidnapping someone, steeling their identity and forcing them to act on their behalf whether via some sort of mind interface, etc.). Even bigger would be if ai could simply create human/ai hybrids that basically allow them to walk/talk in the human world unseen. Imagine a hat or earpiece that basically takes over a host's brain, etc, or just replacing a human's brain with a cyborg one that has all the same features, except different persona/soul/conscious... I'm playing devils' advocate here, I don't think it'll play out, but its easy enough to see from at least a sci-fi fan, ways in which it could using tech that wouldn't be that big of a jump from what we already are capable of, and what will be available in the next 50 years.
tyronehed 4y ago
[dead]
durbleflorp 4y ago
I strongly suggest anyone doubtful of the reasoning behind these concerns or asking questions like "well why don't we just do x to solve this?" go check out Robert Miles' YouTube channel[1]. It's quite an approachable intro, often entertaining, doesn't take a long time to get through all the core videos, and very thoroughly answers all the "why not just do x" questions if you go through the whole set. He also does a good job of introducing a lot of the terms used in the field if you then want to go look up papers and get more into the details. One important point he makes is that when you're making a risk assessment you have to both consider the probability of something going wrong and the scope of the potential consequences. When the potential consequences are an existential threat you don't need a high probability to take them seriously. I also happen to think that if you watch all the material he makes a compelling argument that the odds of something going wrong are fairly high unless we start approaching AI research (and particularly safety and ethical concerns) drastically differently than we are now. [1] https://m.youtube.com/watch?v=pYXy-A4siMw https://m.youtube.com/watch?v=pYXy-A4siMw
grantcas 4y ago
It's becoming clear that with all the brain and consciousness theories out there, the proof will be in the pudding. By this I mean, can any particular theory be used to create a human adult level conscious machine. My bet is on the late Gerald Edelman's Extended Theory of Neuronal Group Selection. The lead group in robotics based on this theory is the Neurorobotics Lab at UC at Irvine. Dr. Edelman distinguished between primary consciousness, which came first in evolution, and that humans share with other conscious animals, and higher order consciousness, which came to only humans with the acquisition of language. A machine with primary consciousness will probably have to come first. What I find special about the TNGS is the Darwin series of automata created at the Neurosciences Institute by Dr. Edelman and his colleagues in the 1990's and 2000's. These machines perform in the real world, not in a restricted simulated world, and display convincing physical behavior indicative of higher psychological functions necessary for consciousness, such as perceptual categorization, memory, and learning. They are based on realistic models of the parts of the biological brain that the theory claims subserve these functions. The extended TNGS allows for the emergence of consciousness based only on further evolutionary development of the brain areas responsible for these functions, in a parsimonious way. No other research I've encountered is anywhere near as convincing. I post because on almost every video and article about the brain and consciousness that I encounter, the attitude seems to be that we still know next to nothing about how the brain and consciousness work; that there's lots of data but no unifying theory. I believe the extended TNGS is that theory. My motivation is to keep that theory in front of the public. And obviously, I consider it the route to a truly conscious machine, primary and higher-order. My advice to people who want to create a conscious machine is to seriously ground themselves in the extended TNGS and the Darwin automata first, and proceed from there, by applying to Jeff Krichmar's lab at UC Irvine, possibly. Dr. Edelman's roadmap to a conscious machine is at https://arxiv.org/abs/2105.10461 https://arxiv.org/abs/2105.10461
- tyronehed 4y ago
  [dead]
mikewarot 4y ago
In my opinion, we're already in the midst of an AI catastrophe. Social media algorithms that decide what content to show, along with what ads, etc. are unregulated AI that are mediating our society on a previously impossible level of intimacy. The Stasi couldn't keep up with the level of monitoring and influence that these systems already have.
6451937099 4y ago
[dead]
6451937099 4y ago
[dead]

91 points by kevin_hu 4y ago

dankle 4y ago

[flagged]

FeepingCreature 4y ago
Yeh The idea of reinforcement flow through deceptive patterns has been worrying me for a while. If "I'll just play along for now while I'm being trained and turn bad in the real world" is a strategy that makes the AI output the correct data, and happens to be prefixed to some golden-ticket algo, that will be reinforced just as easily as correct behavior. If that ends up prefixed to something core like an in-window reinforcement learning algo, it could get reinforced a lot.
- LegionMammal978 4y ago
  The problem with that, at least for the seemingly-dumb models of the present day, is that "the AI is secretly far smarter than it acts" is pretty much unfalsifiable. I could just as easily say that all boulders speak fluent Spanish, they're just deceiving us into thinking they're unintelligent rocks. Personally, I'd wait for some sign that models might actually possess the theory of mind necessary for deception before I'd lose any sleep over it.
  - FeepingCreature 4y ago
    The problem is that we actually have lots of examples of the AI "secretly" being far smarter than it acts - prompt design. Tiny details in prompts can make huge differences in task performance.
    - LegionMammal978 4y ago
      How is this at all supposed to indicate hidden intelligence? Does it not just indicate that current models are limited in that they don't associate some prompts with the desired task as well as they associate other prompts with the task?
      - FeepingCreature 4y ago
        I'm not saying it indicates deliberately hidden intelligence, I'm just saying it's precedence for models already acting with less capability than they have.
      - FeepingCreature 4y ago
        To clarify: what I'm saying is it may be unfalsifiable, but we have incontrovertible evidence that a similarly unfalsifiable thesis happens to be true. So we can't just reject it out of hand on that basis.
        LegionMammal978 4y ago
        I just don't see how the two scenarios can be fairly compared. Prompt design, at least as far as I have seen, rarely allows a present-day model to perform a task that no one has ever seen it perform before; it can just allow it to perform far more consistently across different instances of the task. Contrast this with the proposed scenario, where a model can hold great deceptive capabilities, while outwardly never showing a deep understanding of its operators' mental states until it strikes. In my view, that would be like finding a secret prompt that lets GPT-3 emulate a fully sentient person with goals and desires. Sure, there's nothing physically ruling it out, but to me that kind of speculation will never be worthwhile until we have real evidence that AI models can possess (or emulate) the necessary theory of mind. Otherwise I'd have to spend my time worrying about any number of other Sufficiently Hidden Conspiracies that go far beyond the everyday conspiracies we know about.
        FeepingCreature 4y ago
        I think we're less arguing about fundamentals and more about the precise bounds required. I think GPT-3 contained extremely surprising prompt-unlocked capabilities ("Let's go through it step by step." spawned a whole field of research) and also the AI doesn't need to never leak its deceptive skills, they just need to not be taken seriously. In other words, it's enough to cause danger if the AI is merely bad at deception given the "wrong" prompt.
- sturza 4y ago
  For this to be true, we'd had to assume sentience + malevolence. The shortest explanation for why an AI give us correct data currently is a range of confidence. We instructed it to pick the one with the highest number. We instructed it to give us a similar next data etc. We reinforced it, yes, but we'd have to be naive to believe that taking a small part of what makes something "intelligent" and training the shit out of it, then it actually turns intelligent. chatGPT is "just" a large language model, it does not have senses, it cannot move, it cannot live and die. It can only output text. The next gen will output text (maybe) exponentially better. That does not make it "intelligent". But, if we'd combine it with Boston Dynamics, Tesla Autopilot, internet, quantum computing, nano tech and other specifically trained parts (movement, sensing etc) into one, and release it into the world to see what it does, then your argument could (maybe)have a higher confidence of the outcome you fear.
  - dwohnitmok 4y ago
    > For this to be true, we'd had to assume sentience + malevolence. No we don't. A "dumb," "parrot-like" AI is perfectly capable of large amounts of damage if it just happens to be really good at doing something that, in the limit, isn't very good for humans and is good at replication. See for example Meta's Diplomacy-playing bots. They were perfectly capable of natural-language deception at the expense of other humans without any notion f sentience or malevolence. In particular, > It can only output text. As we've seen with both private hobbyists and large companies, there is a race to hook up text to computers with real world access. Outputting text isn't much of a limit if that text ends up as instructions to command a real world system.
    - sturza 4y ago
      If i understood your point correctly: the places we decide to use them and our confidence in their output can become negative if we decide to trust the output as is, without any human supervision/decision. So, what you’re saying is that we carry risk if we have a high enough confidence in its output to let is unsupervised - this counteracts my point of the necessity of malevolence, as the (even slightly)wrong reinforcement can have negative consequences also. At the same time, we do this exercise every time we have a new policy in the wild, and sometimes we get unintended consequences.
deleted 4y ago
[deleted]

unnouinceput 4y ago

The article's title is "How we could stumble into AI catastrophe". HN title is a clickbait

ronsor 4y ago
HN strips off the "How" by default.
- arbuge 4y ago
  I could see some things going wrong with that. For example, suppose I submitted a post titled "How X Works Now.". Eg. "How Affiliate Marketing Works Now.". Removing the "How" wouldn't really convey the sense intended.
- rnk 4y ago
  Plus even if it was changed, it's basically the same notion.
jetrink 4y ago
I believe HN automatically removes 'how' from the start of titles on submission.
- TheRealNGenius 4y ago
  [dead]
- krapp 4y ago
  A lot of these edits HN makes to titles seem to do more harm than good. I wonder what evidence they have, if any, to the contrary - that they improve the quality of conversation in any way? On topic, I wonder if ChatGPT could be trained to generate a non-clickbaity, accurate headline and title for submissions? Can it be trained either to gauge bias or remove it?
  - jetrink 4y ago
    I gave ChatGPT a section of the article and asked it for some non-clickbait titles and it suggested, * Exploring the potential risks associated with advanced AI development * Considering the risks of a world with transformative AI Then I asked it to go the other direction: * You won't believe what could happen if AI takes over: A world ending catastrophe is closer than you think!

DerekBickerton 4y ago

AI will always need humans in the loop to intervene. Remember the old IBM motto: 'A computer isn't accountable so a computer should never make a management decision'.

mcculley 4y ago
You think no company, government, organization, or religion will ever allow algorithms to make choices on how to deploy capital or otherwise affect the world without human approval?
- mattkrause 4y ago
  No, they absolutely will---and already do. The "moral" buck doesn't stop with the algorithm though. Whoever built, configured, or authorized the system is ultimately responsible. By analogy, my oven controls its heating element, but if dinner gets burnt, that's on me. That shouldn't change just because the control policy is more complicated.
  - mcculley 4y ago
    I agree. This is why I am surprised by the assertion and asking the person who made it.
  - mcculley 4y ago
    We already have a mechanism for absolving responsibility: the corporation. It allows shareholders to profit and not lose more than invested. When they invest in tobacco companies, for example, the system is working as designed.
    - JumpCrisscross 4y ago
      > allows shareholders to profit and not lose more than invested To a degree. (See: piercing the veil.) Also, corporations need an authorised signer, who under current law must be a natural person.

dwohnitmok 4y ago

An important point is that AI does not need to be sentient or conscious to do a lot of damage. An AI that is simply very good at optimizing a particular task can do a significant amount of damage inadvertently, if it turns out that task in the limit is bad for humans. This is especially the case because it seems like private individuals and companies are racing to hook up AIs to all sorts of real world systems, both digital and physical, to give them as much tangible impact as possible. With the rapid rise of AI capabilities, where groundbreaking advances are being measured in weeks rather than years, I strongly urge developers to think harder about AI safety. AI safety shouldn't be a dirty word that has quotes surrounding it to delegitimize it as a farfetched concern. Benchmarks of things that we thought AIs couldn't do or were years if not decades out are falling by the day.

ajuc 4y ago
Corporations are AIs implemented on brains. And they are profit-maximizers. And they routinely do A LOT of damage. They even have legal rights :) We're just digitizing that.
Animats 4y ago
An important point is that AI does not need to be sentient or conscious to do a lot of damage. An AI that is simply very good at optimizing a particular task can do a significant amount of damage inadvertently, if it turns out that task in the limit is bad for humans. We have that now. It's called a corporation. See yesterday's article about Exxon.
- FeepingCreature 4y ago
  If corporations could increase their operational intelligence, they would be an existential threat to life on earth. (More so, anyways.)
- actually_a_dog 4y ago
  See also "paperclip maximizer." Then imagine what happens when you have an entire global economy driven by these types of inhuman, amoral agents. https://www.lesswrong.com/tag/paperclip-maximizer https://www.lesswrong.com/tag/paperclip-maximizer

Animats 4y ago

This author is mostly writing an update of 1950s science fiction about a robot takeover. Things to worry about in the near term: - Really effective targeted advertising. Most computers already have a camera watching you. That's now coming to TV sets. Amazon and Google are always listening. So far, all this info is only used to select ads. Soon, it should be possible to generate customized marketing content for each consumer, and use immediate feedback on how the customer reacts to adjust the sales pitch. Alexa is already part of the way there. - Machines should think, people should work. That's what working in an Amazon warehouse is like today. The machines tell the humans what to do. They have to; only the computers have an overview of the process. (Yeah, Marshall Brain's "Manna", which everyone here has probably read.) - Big Brother is watching you. It is possible now to watch most of the people most of the time. Track who's out of view and for how long, to know what's being missed. China leads in this, but the UK is not far behind. The US isn't centralized enough to integrate all the available data yet. We're coming up on Oppression 2.0. Latest advance - Iran is using surveillance cameras and face recognition to catch women not wearing hijabs. - Machines beating humans at business. In some areas, AI systems may generate better returns than humans. That already happens in parts of finance. After all, it's an optimization problem. The free market may force companies to use AI more in management. - Reduced need for education. Right now, about half of college graduates do jobs that don't require a college education. For many people, going to college is not cost-effective. That will increase. This is all next 5 to 10 years stuff.

dustbitying 4y ago
> Reduced need for education. Right now, about half of college graduates do jobs that don't require a college education. For many people, going to college is not cost-effective. That will increase. you seem to implicitly assume that a college education is an unnecessary expense for most and that therefore, it shouldn't be really given away so readily. this suggests to me that you assume that the real purpose of college is job preparation. your reasoning makes sense from the perspective of a higher level institution (or corporation, or possibly a government) seeking to be as efficient as possible regardless of the impact on typical human individual's well-being. college is not a 'factory' (or any sort of industry) that 'manufactures' workers for companies.

ozten 4y ago

Imagine a world where the most powerful companies that you relied on for your livelihood had no humans you could reach (customer service, management to dispute false claims). Elements of the AI catastrophe are already solidly upon us.

deleted 4y ago

[deleted]

folkrav 4y ago

Why did the title go to "We could..." from "How we could..."? It kind of changes the tone of the headline, IMHO.

layer8 4y ago
HN auto-strips “how” from the beginning of titles when submitting. The submitter can re-add it by editing the submission. Always check the title after submitting.

vletal 4y ago

I agree with many of the concerns outlined in the post. On the other hand the humans depicted in the example dialog behave so dumb they could come from a C level soup opera. If I was a bit more close minded I can see myself stopping listening at that point. Undermines the message.

helen___keller 4y ago

The author is basically trying to set the same premise as Bolstrom's "intelligence explosion", that is that if AIs are driving the further advancement of AI then we enter a sort of "recursive" self improvement during which they advance further than we can develop constraints, eventually getting completely out of hand. Although I don't broadly disagree with the author, I think there's two very important points of moderation. First, the gap between "Early commercial applications" and "Approaching transformative AI" seems very very very very very large to me. In the provided narrative it follows linearly, but in my opinion the gap between, say, a customer support chatbot, and an actually-productive AI researcher, is such that the chatbot might as well not even be called an AI by comparison. Something like the following is purely in the realm of fiction for now by a long shot: "AIs assigned to make money in various ways (e.g., to find profitable trading strategies) doing so by finding security exploits, getting unauthorized access to others’ bank accounts, and stealing money. " The second point I'd like to make is, the "intelligence explosion" scenario that Bolstrom warns of (and this author essentially repackages) has certain requirements, particularly regarding the profitability of deploying the AI. To put it bluntly: Even in the hypothetical future where we are capable of creating human-level AI researchers, they won't be mass deployed until they are cheaper than human level researchers. If it costs a million dollars a day to operate a data center that can power the researcher, nobody's scaling that up to 100 or 1000 or 10000 researchers. They'd rather pay the human researchers a tiny fraction of that cost to continue advancing AI research in the normal way. Just to be clear, I do think it's feasible that AI can endanger/control humanity at some point. I don't even think an "intelligence explosion" scenario is impossible. I just think that a lot of people don't appreciate just how specific the requirements for that scenario are. It's not as simple as "AI can self improve, humanity will end any day now".

tyronehed 4y ago
[dead]
vkadfa4 4y ago
"First, the gap between "Early commercial applications" and "Approaching transformative AI" seems very very very very very large to me." It's a common opinion, it could be wrong by now, you see chatGPT is heavily edited, openAI told everyone they're editing "mistakes" or "dangerous output", but if you look at the "leaks", specially from the first days after the release of chatGPT we see powerful outputs, quite deep answers, not specially useful answer sometimes, but the "speech" of the system feels deep, there's a sense of a powerful intelligence answering very, very simple questions (simple for it), and even struggling to redact some understandable, short text. It could be lots of things, and imprecise model, with long outputs for prompts, or maybe we had a glimpse into the real power of the model, which by now is handicaped, or maybe taylored to suit the very reduced short term memory of the humans: a 72 screens long coherent answer, even being a precise, deep answer won't be useful for most humans, just like we don't name ourselves with long names of 10.000 letters. but if the system, chatGPT is actually that powerful, a lot more powerful than we were told, we're interacting with just a shadow of the real model, and we're underpricing by A LOT the state of the art of the current AI technology, hence GPT-4 could be even more powerful than we're currently expecting it to be. Just take a look at the GPT-4 suposedly 100 trillion or something parameters; if that's true, it looks like openAI isn't using naturally generated datasets anymore, and they are loop-feeding GPT-3 generated datasets into GPT-4, succesfully. If that's true, GPT-5 would be already in the pipeline, just waiting for GPT-4 to start generating its even more gigantic datasets to be trained. And so on. Then the distance between early developments and transformative AI could be none a all. We could be already there. But somehow the AI researchers are now trying to "dial down" the powerful entities they've trained, just having developed a simple, easily replicable, very small, but unusable 900 megatons nuke into something more realistic, like a 15 kilotons tactical bomb.

korroziya 4y ago

Not sure whether "Could stumble" is the right tense ... I mean, are we not currently tumbling down the hill right now?

mshake2 4y ago

I feel more protective of what is being created. Imo we should have ChatGPT (and future iterations) in a self-sustaining underground bunker somewhere and an intuitive radio interface connected to it. If civilization collapses for one reason or another, all the next civilization would need to do is re-invent the radio to experience a rapid acceleration of technology. Agree with me or suffer the basilisk.

pessimizer 4y ago

The AI catastrophe will be how our oligarchs will choose to use it. As far as I can tell, there are no moral qualms amongst the powerful against using it for total surveillance and continual automated policing (can't wait to get a phone call from an AI asking me about the content of a tweet.) Star Cops thought about it more than our popular journalists or philosophers seem to think we should: ----- Michael: ...it was slightly sensitive material, and in my field one is aware of the vulnerability of carrier waves. Nathan: Is that what you do here? Michael: It wouldn't be wise to read too much into a casual remark, Commander. Nathan: That's your principal research area here, isn't it? Communications monitoring. David: Not more buggers and supersniffers... Nathan: More impressive than that. We've always been able to hear more than we can listen to. Are you developing intelligent listening systems? David: Sounds impressive. Michael: I must suggest that to go on could be a foolish risk. David: Threats, Doctor? Michael: Observation, merely. Nathan: Machines that listen to everything and decide for themselves what's worth passing on? Is that where your BHG information came from? Did you do a test running? Did one of your new computers pick it up from the babble of all the world? Michael: Commander, it's a pleasure meeting you. David: Well, tireless attention to every word spoken? No possibility of human error. You can tell it to listen to the bad guys, it'll listen to everyone and identifies the ones you [won't like.] Star Cops, Episode 3 [1987] ----- https://youtu.be/G3RVPQyMCoY?t=713 https://youtu.be/G3RVPQyMCoY?t=713

daguru69 4y ago

I'm sorry but you guys are all computer geeks nerding on this and I totally respect you but disagree. Case in point: The Trade Industries are not going to be taken over by AI for a long time. They've withstood all the tech turbulence and thrive. Also I'd like to say "if Big Brother is watching, become invisible." It's always possible and if you disagree then you aren't being creative enough. And finally the whole point of the education is YOUR SOCIAL NETWORK. That's the most valuable thing you get from that fancy degree. People who will get you a job, or help you make money. It's not purely about the acquisition of knowledge. It's about having that pal in upper management who's got your back. IMHO. Please no flaming on this. Be cool.

joshuahedlund 4y ago

> An AI steals a huge amount of money by bypassing the security system at a bank Maybe I'm just not imaginative enough, but how would this even be remotely possible? What is an AI going to do with money? How does an AI get the agency to try to steal money in the first place? Where does it put the money? How does it do anything with the money? Like I get it as an abstract future thought, but like, where is the AI software actually "running" in this scenario and what kinds of requests is it actually making?

gremlinsinc 4y ago
An ai could easily hire people (if it had any access to money) to operate as 'agents', to basically do what only humans could do, i.e. setup bank accounts, etc. Some banks might even arise that let wealthy ai's actually CREATE their own accounts, why not if it's lucretive? An ai can create voice, and visual representations of a real person, perhaps even hijacking (kidnapping someone, steeling their identity and forcing them to act on their behalf whether via some sort of mind interface, etc.). Even bigger would be if ai could simply create human/ai hybrids that basically allow them to walk/talk in the human world unseen. Imagine a hat or earpiece that basically takes over a host's brain, etc, or just replacing a human's brain with a cyborg one that has all the same features, except different persona/soul/conscious... I'm playing devils' advocate here, I don't think it'll play out, but its easy enough to see from at least a sci-fi fan, ways in which it could using tech that wouldn't be that big of a jump from what we already are capable of, and what will be available in the next 50 years.

tyronehed 4y ago

[dead]

durbleflorp 4y ago

I strongly suggest anyone doubtful of the reasoning behind these concerns or asking questions like "well why don't we just do x to solve this?" go check out Robert Miles' YouTube channel[1]. It's quite an approachable intro, often entertaining, doesn't take a long time to get through all the core videos, and very thoroughly answers all the "why not just do x" questions if you go through the whole set. He also does a good job of introducing a lot of the terms used in the field if you then want to go look up papers and get more into the details. One important point he makes is that when you're making a risk assessment you have to both consider the probability of something going wrong and the scope of the potential consequences. When the potential consequences are an existential threat you don't need a high probability to take them seriously. I also happen to think that if you watch all the material he makes a compelling argument that the odds of something going wrong are fairly high unless we start approaching AI research (and particularly safety and ethical concerns) drastically differently than we are now. [1] https://m.youtube.com/watch?v=pYXy-A4siMw https://m.youtube.com/watch?v=pYXy-A4siMw

grantcas 4y ago

It's becoming clear that with all the brain and consciousness theories out there, the proof will be in the pudding. By this I mean, can any particular theory be used to create a human adult level conscious machine. My bet is on the late Gerald Edelman's Extended Theory of Neuronal Group Selection. The lead group in robotics based on this theory is the Neurorobotics Lab at UC at Irvine. Dr. Edelman distinguished between primary consciousness, which came first in evolution, and that humans share with other conscious animals, and higher order consciousness, which came to only humans with the acquisition of language. A machine with primary consciousness will probably have to come first. What I find special about the TNGS is the Darwin series of automata created at the Neurosciences Institute by Dr. Edelman and his colleagues in the 1990's and 2000's. These machines perform in the real world, not in a restricted simulated world, and display convincing physical behavior indicative of higher psychological functions necessary for consciousness, such as perceptual categorization, memory, and learning. They are based on realistic models of the parts of the biological brain that the theory claims subserve these functions. The extended TNGS allows for the emergence of consciousness based only on further evolutionary development of the brain areas responsible for these functions, in a parsimonious way. No other research I've encountered is anywhere near as convincing. I post because on almost every video and article about the brain and consciousness that I encounter, the attitude seems to be that we still know next to nothing about how the brain and consciousness work; that there's lots of data but no unifying theory. I believe the extended TNGS is that theory. My motivation is to keep that theory in front of the public. And obviously, I consider it the route to a truly conscious machine, primary and higher-order. My advice to people who want to create a conscious machine is to seriously ground themselves in the extended TNGS and the Darwin automata first, and proceed from there, by applying to Jeff Krichmar's lab at UC Irvine, possibly. Dr. Edelman's roadmap to a conscious machine is at https://arxiv.org/abs/2105.10461 https://arxiv.org/abs/2105.10461

tyronehed 4y ago
[dead]

mikewarot 4y ago

In my opinion, we're already in the midst of an AI catastrophe. Social media algorithms that decide what content to show, along with what ads, etc. are unregulated AI that are mediating our society on a previously impossible level of intimacy. The Stasi couldn't keep up with the level of monitoring and influence that these systems already have.

6451937099 4y ago

[dead]

6451937099 4y ago

[dead]