7 ms·

Understanding Agent Cooperation

126 points by piokuc 10y ago

bitwize 10y ago
Oh great. It's just a matter of time before it floods the Enrichment Center with deadly neurotoxin.
bencollier49 10y ago
What an awful headline. "AI learns to compete in competitive situations" should be the precis. Basically, it learned that it didn't need to fight until there was resource scarcity in a simulation.
- leereeves 10y ago
  I think aggressive is a better (more descriptive, narrower) word than compete here. Two racers are competing to see who runs faster, but if one pulls out a laser gun and shoots the other, that's aggressive.
  - bencollier49 10y ago
    It's a loaded word in the context of the headline, and the manner in which it was used in the story body, especially when combined with "stress" to create an image of a sort of edgy killing machine. Actually, it's an interesting word. Dictionary definitions of aggression frequently revolve around emotions - it's a very human word, probably not suitable for AI.
  - tutts 10y ago
    Not if laser guns are a part of the race. Then it's just competitive.
    - espadrine 10y ago
      In real life, the game rules are whatever is physically possible, taking into account the risk of getting caught and the corresponding fine. It is all good fun and laser tag until the AI manages the interests of your bank, your insurer, the stock market, everyone, and you desperately need a liberal government to enforce serious penalties for increasingly complex loopholes that the AI finds in a few seconds.
    - wukerplank 10y ago
      If the laser gun is not part of the race, it's cheating (at best). I don't understand this nitpicking. Of course a gun is an aggressive tool. "Aggressive" and "competitive" are not mutually exclusive.
      - slowmovintarget 10y ago
        It all depends on how the gun is modeled. If, which seems likely, it is as simple as push-button-receive-bacon, then there are no consequences to pushing the button. It's difficult to characterize that as aggression, especially if the system has no built in notion of harm or other-like-me. That is what is actually scarier: Violence as paperwork.
  - itsdrewmiller 10y ago
    The game included the ability to shoot the other player with no consequence - it sounds aggressive because "laser beam" but if you called it "tagging" then it would more clearly just be an in-rules option.
    - leereeves 10y ago
      I don't think "in-rules" and "aggressive" are mutually exclusive. It's fair to call blitzing the QB an aggressive move in American football.
      - MatekCopatek 10y ago
        You're technically correct, but the football analogy switches context so that the meaning of aggressive is no longer bad. I think the point bencollier49 is trying to make is that we simply gave software a specific set of rules to train it. It doesn't know how we perceive the actions it is performing. The game could be described as two people eating poisonous apples in order to prevent the other person from dying. In that case, the currently greedy one would be the hero.
        leereeves 10y ago
        > we simply gave software a specific set of rules to train it. It doesn't know how we perceive the actions it is performing. I think the author is making the same point from another angle. The AI learns what we would consider aggressive moves when conditions favor those moves.
  - jfoster 10y ago
    To the AI, there is nothing aggressive about the "laser gun"; as far as it was aware, the "laser gun" could be any tool. It was just doing what it had determined may help it achieve a better score.
    - leereeves 10y ago
      The "laser gun" knocks the other player out of the race. It's not the name that makes me consider it aggressive, rather the fact that it works by harming the other player. It's probably true the AI doesn't distinguish "aggressive" tools from other tools. Isn't that one of lessons here? If an AI isn't taught not to be aggressive, it will choose to harm other participants when that's the most effective strategy.
- tdkl 10y ago
  Progressive left double speak, nothing new.
  - dang 10y ago
    Please don't post unsubstantive comments, and especially don't do partisan name-calling here. It destroys the kind of discussion HN exists for. Edit: unfortunately you've been doing this a lot. We ban accounts that do this, so please stop.
tmcpro 10y ago
I wonder how Deepmind will simulate game theory as it advances
- mtanski 10y ago
  I imagine you'll be able to get answer to more complex games. Things like combination of N players, multiple stable states, different optima for different players, external factors / stimuli. Answers by simulation rather then proof.
saycheese 10y ago
This reads as click-bait, here's the original blog post and research paper by DeepMind: "Understanding Agent Cooperation" https://news.ycombinator.com/edit?id=13635218 https://news.ycombinator.com/edit?id=13635218
- dang 10y ago
  Thanks! We changed to that from http://www.sciencealert.com/google-s-new-ai-has-learned-to-become-highly-aggressive-in-stressful-situations http://www.sciencealert.com/google-s-new-ai-has-learned-to-b....
creo 10y ago
Bait.
JamilD 10y ago
The AI can minimize loss / maximize fitness by either moving to look for additional resources, or fire a laser. Turns out that when resources are scarce, the optimal move is to knock the opponent away. I think this tells us more about the problem space than the AI itself; it's just optimizing for the specific problem.
- swalsh 10y ago
  I think you're right, had the AI (like the second game) had the incentive to maximize the well being of the cooperating actors, rather than itself, the outcome would be different. But if advanced AI is being developed in a capitalist economy by independent actors, it seems most likely the incentives will be anything other than optimizing the output for the individual outcome. If that AI finds a way to "hurt" the other actor, there could be some major boat load of unintended consequences.
- laretluval 10y ago
  > I think this tells us more about the problem space than the AI itself; it's just optimizing for the specific problem. My reading is that the point of this research is to find out what problem spaces are conducive to cooperation, not to find out details of how these particular agents work.
  - JamilD 10y ago
    Yeah — after reading this article and the paper, I agree. The previous link [0] on this submission was much more sensationalized and very misleading. [0] http://www.sciencealert.com/google-s-new-ai-has-learned-to-become-highly-aggressive-in-stressful-situations http://www.sciencealert.com/google-s-new-ai-has-learned-to-b...
katzgrau 10y ago
The article at first suggests that more intelligent versions of AI led to greed and sabotage. But I do wonder if an even more intelligent AI (perhaps in a more complex environment) would take the long view instead and find a reason to co-habitate. It's kind of like rocks, paper scissors - when you attempt to think several levels deeper than your opponent and guess which level they stopped at. At some intelligence level for AI, cohabitation seems optimal - at the next level, not so much, and so on. We're probably going to end up building something so complex that we don't quite understand it and end up hurting somebody.
vanderZwan 10y ago
You know, rather than being scared by this, I think it's an excellent opportunity to learn how and when aggression evolves, and maybe learn how we can set up systems that nudge people to collaborate, perhaps even when resources are scarce.
doener 10y ago
https://news.ycombinator.com/item?id=13620518 https://news.ycombinator.com/item?id=13620518
projektir 10y ago
I'm rather worried about the wording used, and AI being created in that context. Do we really not realize what we're doing? AI is not magic, it's not free from fundamental math, it's not free from corruption. It's just going to multiply it that much more. Any AI that has been programmed to highly value winning is not going to be very cooperative. For it to be cooperative, especially in situations that simulate survival, it needs to have higher ideals than winning, just like humans. It needs to be able to see and be aware of the big picture. You don't need to look at AI for that, you can just look at the world. Development of AI's of this nature will just lead to a super-powered Moloch. Cooperative ethics is a highly advanced concept, it's not going to show up on its own from mere game theory without a lot of time.
- philipov 10y ago
  Cooperative ethics arise immediately in the Prisoner's Dilemma merely by adding an unknown number of iterations to the game. The most efficient strategy is a version of tit-for-tat.
  - projektir 10y ago
    I'm assuming you're referring to something like this: https://egtheory.wordpress.com/2015/03/02/ipd/ https://egtheory.wordpress.com/2015/03/02/ipd/ I think we shouldn't confuse efficient strategies with the chosen strategies. What causes Moloch is the inability to see the big picture, to see outside of the self in the collective (maybe Buddhism has a point). An efficient strategy may very well be something we'd prefer, such as tit-for-tat. But is that the strategy we choose? Looking at the long history of evolution, I'd say no.
    - philipov 10y ago
      > Looking at the long history of evolution, I'd say no. This entire lecture series on Human Behavioral Biology is worth watching from the beginning, but I've linked to a moment where Sapolsky describes tit-for-tat strategies arising in animals. First example: Vampire Bats. []: https://www.youtube.com/watch?v=Y0Oa4Lp5fLE&feature=youtu.be&t=1h1m42s&list=PL848F2368C90DDC3D https://www.youtube.com/watch?v=Y0Oa4Lp5fLE&feature=youtu.be...
jonbaer 10y ago
Why is this done on such a small level? I would have thought that with systems now in place that evolutionary game theory could be done in simulations on such a much larger scale (say 7bn agents +) ... if anything AI systems should be able to determine if certain strategies work (like items like blocking resources - such as a case of geopolitical theory) so see what cooperations occur at that level. Still amazing work but it should be applied to a larger scale for real meaning. More eager to see how RL applied to RTS games will explore and develop strategies more than anything.
jerf 10y ago
Not entirely spawned by this article, but the whole genre and some other comments on HN by other users: I wonder if part of the "mystery" of cooperation in these simulations is that these people keep investigating the question of cooperation using simulations too simplistic to model any form of trade. A fundamental of economics 101 is that valuations for things differ for different agents. Trade ceases to exist in a world where everybody values everything exactly the same, because the only trade that makes any sense is to two trade two things of equal value, and even then, since the outcome is a wash and neither side obtains any value from it, why bother? I'm not sure the simulation hasn't been simplified to the point that the phenomena we're trying to use the simulation to explain are not capable of manifesting within the simulation. I'm not saying that Trade Is The Answer. I would be somewhat surprised if it doesn't form some of the solution eventually, but that's not the argument I'm making today. The argument I'm making is that if the simulation can't simulate trade at all, that's a sign that it may have been too simplified to be useful. There are probably other things you could say that about; "communication" being another one. The only mechanism for communication being the result of iteration is questionable too, for instance. Obviously in the real world, most cooperation doesn't involve human speech, but a lot of ecology can be seen to involve communication, if for no other reason than you can't have the very popular strategy of "deception" if you don't have "communication" with which to deceive. Which may also explain the in-my-opinion overpopular and excessively studied "Prisoner's Dilemma", since it has the convenient characteristic of explicitly writing communication out of it. I fear its popularity may blind us to the fact that it wasn't ever really meant to be the focus of study of social science, but more a simplified word problem for game theory. Studying a word problem over and over and over may be like trying to understand the real world of train transportation systems by repeatedly studying "A train leaves from Albuquerque headed towards Boston at 1pm on Tuesday and a train leaves from Boston headed towards Albuquerque at 3pm on Wednesday, when do they pass each other?" over and over again. (Or to put it really simply in machine learning terms, what's the point of trying to study cooperation in systems whose bias does not encompass cooperation behaviors in the first place?)
- deleted 10y ago
  [deleted]
- titanomachy 10y ago
falsedan 10y ago
Is it just me, or is this article extremely light on content? The core of it seems to be > sequential social dilemmas, and us[ing] artificial agents trained by deep multi-agent reinforcement learning to study [them] But I didn't find out how to recognise a sequential social dilemma, nor their training method.
- roymurdock 10y ago
  Here's the actual paper: https://storage.googleapis.com/deepmind-media/papers/multi-agent-rl-in-ssd.pdf https://storage.googleapis.com/deepmind-media/papers/multi-a... Don't expect any crazy deep insights, but it's a useful read if you want to set up a similar experiment or understand the research methodology.
george_ciobanu 10y ago
"Scarce resources cause competition" and "Scarce but close to impossible to catch on own resources cause cooperation". Is that really a discovery worth publishing?
cs702 10y ago
The folks at DeepMind continue to produce clever original work at an astounding pace, with no signs of slowing down. Whenever I think I've finally gotten a handle on the state-of-the-art in AI research, they come up with something new that looks really interesting. They're now training deep-reinforcement-learning agents to co-evolve in increasingly more complex settings, to see if, how, and when the agents learn to cooperate (or not). Should they find that agents learn to behave in ways that, say, contradict widely accepted economic theory, this line of work could easily lead to a Nobel prize in Economics. Very cool.
d--b 10y ago
Mmmh, the problem of modeling social behavior is in defining the reward function, not in implementing optimal strategies to maximize the reward. In a game where you are given the choice of killing 10,000 people or be killed yourself, which is the most rewarding outcome?
c3534l 10y ago
I know what I'm writing my systems science paper on.
tawpKek 10y ago
>Self-interested people often work together to achieve great things. Why should this be the case, when it is in their best interest to just care about their own wellbeing and disregard that of others? I think this is a kind of strong statement to take as a given, especially as an opening. This is taking social darwinism as law, and could use more scrutiny.

126 points by piokuc 10y ago

Oh great. It's just a matter of time before it floods the Enrichment Center with deadly neurotoxin.

What an awful headline. "AI learns to compete in competitive situations" should be the precis. Basically, it learned that it didn't need to fight until there was resource scarcity in a simulation.

leereeves 10y ago
I think aggressive is a better (more descriptive, narrower) word than compete here. Two racers are competing to see who runs faster, but if one pulls out a laser gun and shoots the other, that's aggressive.
- bencollier49 10y ago
  It's a loaded word in the context of the headline, and the manner in which it was used in the story body, especially when combined with "stress" to create an image of a sort of edgy killing machine. Actually, it's an interesting word. Dictionary definitions of aggression frequently revolve around emotions - it's a very human word, probably not suitable for AI.
- tutts 10y ago
  Not if laser guns are a part of the race. Then it's just competitive.
  - espadrine 10y ago
    In real life, the game rules are whatever is physically possible, taking into account the risk of getting caught and the corresponding fine. It is all good fun and laser tag until the AI manages the interests of your bank, your insurer, the stock market, everyone, and you desperately need a liberal government to enforce serious penalties for increasingly complex loopholes that the AI finds in a few seconds.
  - wukerplank 10y ago
    If the laser gun is not part of the race, it's cheating (at best). I don't understand this nitpicking. Of course a gun is an aggressive tool. "Aggressive" and "competitive" are not mutually exclusive.
    - slowmovintarget 10y ago
      It all depends on how the gun is modeled. If, which seems likely, it is as simple as push-button-receive-bacon, then there are no consequences to pushing the button. It's difficult to characterize that as aggression, especially if the system has no built in notion of harm or other-like-me. That is what is actually scarier: Violence as paperwork.
- itsdrewmiller 10y ago
  The game included the ability to shoot the other player with no consequence - it sounds aggressive because "laser beam" but if you called it "tagging" then it would more clearly just be an in-rules option.
  - leereeves 10y ago
    I don't think "in-rules" and "aggressive" are mutually exclusive. It's fair to call blitzing the QB an aggressive move in American football.
    - MatekCopatek 10y ago
      You're technically correct, but the football analogy switches context so that the meaning of aggressive is no longer bad. I think the point bencollier49 is trying to make is that we simply gave software a specific set of rules to train it. It doesn't know how we perceive the actions it is performing. The game could be described as two people eating poisonous apples in order to prevent the other person from dying. In that case, the currently greedy one would be the hero.
      - leereeves 10y ago
        > we simply gave software a specific set of rules to train it. It doesn't know how we perceive the actions it is performing. I think the author is making the same point from another angle. The AI learns what we would consider aggressive moves when conditions favor those moves.
- jfoster 10y ago
  To the AI, there is nothing aggressive about the "laser gun"; as far as it was aware, the "laser gun" could be any tool. It was just doing what it had determined may help it achieve a better score.
  - leereeves 10y ago
    The "laser gun" knocks the other player out of the race. It's not the name that makes me consider it aggressive, rather the fact that it works by harming the other player. It's probably true the AI doesn't distinguish "aggressive" tools from other tools. Isn't that one of lessons here? If an AI isn't taught not to be aggressive, it will choose to harm other participants when that's the most effective strategy.
tdkl 10y ago
Progressive left double speak, nothing new.
- dang 10y ago
  Please don't post unsubstantive comments, and especially don't do partisan name-calling here. It destroys the kind of discussion HN exists for. Edit: unfortunately you've been doing this a lot. We ban accounts that do this, so please stop.

tmcpro 10y ago

I wonder how Deepmind will simulate game theory as it advances

mtanski 10y ago
I imagine you'll be able to get answer to more complex games. Things like combination of N players, multiple stable states, different optima for different players, external factors / stimuli. Answers by simulation rather then proof.

saycheese 10y ago

This reads as click-bait, here's the original blog post and research paper by DeepMind: "Understanding Agent Cooperation" https://news.ycombinator.com/edit?id=13635218 https://news.ycombinator.com/edit?id=13635218

dang 10y ago
Thanks! We changed to that from http://www.sciencealert.com/google-s-new-ai-has-learned-to-become-highly-aggressive-in-stressful-situations http://www.sciencealert.com/google-s-new-ai-has-learned-to-b....

creo 10y ago

Bait.

JamilD 10y ago

The AI can minimize loss / maximize fitness by either moving to look for additional resources, or fire a laser. Turns out that when resources are scarce, the optimal move is to knock the opponent away. I think this tells us more about the problem space than the AI itself; it's just optimizing for the specific problem.

swalsh 10y ago
I think you're right, had the AI (like the second game) had the incentive to maximize the well being of the cooperating actors, rather than itself, the outcome would be different. But if advanced AI is being developed in a capitalist economy by independent actors, it seems most likely the incentives will be anything other than optimizing the output for the individual outcome. If that AI finds a way to "hurt" the other actor, there could be some major boat load of unintended consequences.
laretluval 10y ago
> I think this tells us more about the problem space than the AI itself; it's just optimizing for the specific problem. My reading is that the point of this research is to find out what problem spaces are conducive to cooperation, not to find out details of how these particular agents work.
- JamilD 10y ago
  Yeah — after reading this article and the paper, I agree. The previous link [0] on this submission was much more sensationalized and very misleading. [0] http://www.sciencealert.com/google-s-new-ai-has-learned-to-become-highly-aggressive-in-stressful-situations http://www.sciencealert.com/google-s-new-ai-has-learned-to-b...

katzgrau 10y ago

The article at first suggests that more intelligent versions of AI led to greed and sabotage. But I do wonder if an even more intelligent AI (perhaps in a more complex environment) would take the long view instead and find a reason to co-habitate. It's kind of like rocks, paper scissors - when you attempt to think several levels deeper than your opponent and guess which level they stopped at. At some intelligence level for AI, cohabitation seems optimal - at the next level, not so much, and so on. We're probably going to end up building something so complex that we don't quite understand it and end up hurting somebody.

vanderZwan 10y ago

You know, rather than being scared by this, I think it's an excellent opportunity to learn how and when aggression evolves, and maybe learn how we can set up systems that nudge people to collaborate, perhaps even when resources are scarce.

doener 10y ago

https://news.ycombinator.com/item?id=13620518 https://news.ycombinator.com/item?id=13620518

projektir 10y ago

I'm rather worried about the wording used, and AI being created in that context. Do we really not realize what we're doing? AI is not magic, it's not free from fundamental math, it's not free from corruption. It's just going to multiply it that much more. Any AI that has been programmed to highly value winning is not going to be very cooperative. For it to be cooperative, especially in situations that simulate survival, it needs to have higher ideals than winning, just like humans. It needs to be able to see and be aware of the big picture. You don't need to look at AI for that, you can just look at the world. Development of AI's of this nature will just lead to a super-powered Moloch. Cooperative ethics is a highly advanced concept, it's not going to show up on its own from mere game theory without a lot of time.

philipov 10y ago
Cooperative ethics arise immediately in the Prisoner's Dilemma merely by adding an unknown number of iterations to the game. The most efficient strategy is a version of tit-for-tat.
- projektir 10y ago
  I'm assuming you're referring to something like this: https://egtheory.wordpress.com/2015/03/02/ipd/ https://egtheory.wordpress.com/2015/03/02/ipd/ I think we shouldn't confuse efficient strategies with the chosen strategies. What causes Moloch is the inability to see the big picture, to see outside of the self in the collective (maybe Buddhism has a point). An efficient strategy may very well be something we'd prefer, such as tit-for-tat. But is that the strategy we choose? Looking at the long history of evolution, I'd say no.
  - philipov 10y ago
    > Looking at the long history of evolution, I'd say no. This entire lecture series on Human Behavioral Biology is worth watching from the beginning, but I've linked to a moment where Sapolsky describes tit-for-tat strategies arising in animals. First example: Vampire Bats. []: https://www.youtube.com/watch?v=Y0Oa4Lp5fLE&feature=youtu.be&t=1h1m42s&list=PL848F2368C90DDC3D https://www.youtube.com/watch?v=Y0Oa4Lp5fLE&feature=youtu.be...

jonbaer 10y ago

Why is this done on such a small level? I would have thought that with systems now in place that evolutionary game theory could be done in simulations on such a much larger scale (say 7bn agents +) ... if anything AI systems should be able to determine if certain strategies work (like items like blocking resources - such as a case of geopolitical theory) so see what cooperations occur at that level. Still amazing work but it should be applied to a larger scale for real meaning. More eager to see how RL applied to RTS games will explore and develop strategies more than anything.

jerf 10y ago

Not entirely spawned by this article, but the whole genre and some other comments on HN by other users: I wonder if part of the "mystery" of cooperation in these simulations is that these people keep investigating the question of cooperation using simulations too simplistic to model any form of trade. A fundamental of economics 101 is that valuations for things differ for different agents. Trade ceases to exist in a world where everybody values everything exactly the same, because the only trade that makes any sense is to two trade two things of equal value, and even then, since the outcome is a wash and neither side obtains any value from it, why bother? I'm not sure the simulation hasn't been simplified to the point that the phenomena we're trying to use the simulation to explain are not capable of manifesting within the simulation. I'm not saying that Trade Is The Answer. I would be somewhat surprised if it doesn't form some of the solution eventually, but that's not the argument I'm making today. The argument I'm making is that if the simulation can't simulate trade at all, that's a sign that it may have been too simplified to be useful. There are probably other things you could say that about; "communication" being another one. The only mechanism for communication being the result of iteration is questionable too, for instance. Obviously in the real world, most cooperation doesn't involve human speech, but a lot of ecology can be seen to involve communication, if for no other reason than you can't have the very popular strategy of "deception" if you don't have "communication" with which to deceive. Which may also explain the in-my-opinion overpopular and excessively studied "Prisoner's Dilemma", since it has the convenient characteristic of explicitly writing communication out of it. I fear its popularity may blind us to the fact that it wasn't ever really meant to be the focus of study of social science, but more a simplified word problem for game theory. Studying a word problem over and over and over may be like trying to understand the real world of train transportation systems by repeatedly studying "A train leaves from Albuquerque headed towards Boston at 1pm on Tuesday and a train leaves from Boston headed towards Albuquerque at 3pm on Wednesday, when do they pass each other?" over and over again. (Or to put it really simply in machine learning terms, what's the point of trying to study cooperation in systems whose bias does not encompass cooperation behaviors in the first place?)

deleted 10y ago
[deleted]
titanomachy 10y ago

falsedan 10y ago

Is it just me, or is this article extremely light on content? The core of it seems to be > sequential social dilemmas, and us[ing] artificial agents trained by deep multi-agent reinforcement learning to study [them] But I didn't find out how to recognise a sequential social dilemma, nor their training method.

roymurdock 10y ago
Here's the actual paper: https://storage.googleapis.com/deepmind-media/papers/multi-agent-rl-in-ssd.pdf https://storage.googleapis.com/deepmind-media/papers/multi-a... Don't expect any crazy deep insights, but it's a useful read if you want to set up a similar experiment or understand the research methodology.

george_ciobanu 10y ago

"Scarce resources cause competition" and "Scarce but close to impossible to catch on own resources cause cooperation". Is that really a discovery worth publishing?

cs702 10y ago

The folks at DeepMind continue to produce clever original work at an astounding pace, with no signs of slowing down. Whenever I think I've finally gotten a handle on the state-of-the-art in AI research, they come up with something new that looks really interesting. They're now training deep-reinforcement-learning agents to co-evolve in increasingly more complex settings, to see if, how, and when the agents learn to cooperate (or not). Should they find that agents learn to behave in ways that, say, contradict widely accepted economic theory, this line of work could easily lead to a Nobel prize in Economics. Very cool.

d--b 10y ago

Mmmh, the problem of modeling social behavior is in defining the reward function, not in implementing optimal strategies to maximize the reward. In a game where you are given the choice of killing 10,000 people or be killed yourself, which is the most rewarding outcome?

c3534l 10y ago

I know what I'm writing my systems science paper on.

tawpKek 10y ago

>Self-interested people often work together to achieve great things. Why should this be the case, when it is in their best interest to just care about their own wellbeing and disregard that of others? I think this is a kind of strong statement to take as a given, especially as an opening. This is taking social darwinism as law, and could use more scrutiny.