8 ms·

Essentially all AI work I've seen in games aims for game theory optimal play, but I think it could be really interesting to consider AI for exploitative play. D

by mxwsn 2y ago

Essentially all AI work I've seen in games aims for game theory optimal play, but I think it could be really interesting to consider AI for exploitative play. Does this exist? Poker with imperfect information, human pressure and fallibility means that players will inevitably stray from Nash equilibrium. The decision on how to exploit without getting exploited back oneself seems really fascinating to consider from an AI perspective. At a glance it seems to require considering how others view you..

raincole 2y ago
> Imperfect information, human pressure... inevitably stray from Nash equilibrium Human pressure yes. Imperfect information no. When we talk about Nash equilibrium for a game like poker, it's already based on imperfect information.
- mxwsn 2y ago
  Ah right, thanks
waprin 2y ago
Within solvers, you can do something called "node locking", which means you "lock" a tree in the game node to play a fixed strategy. You would typically lock it to play as you suspect your opponent plays. This lets the solver calculate the optimal exploitative solution against your specific oppoents. Piosolver, the first public solver and the one mentioned in the article, has this feature. However, what often happens is if you lock one node, then several other nodes in the game tree over-adjust in drastic ways, forcing you to lock all of the, which may be infeasiable. As a result, Piosolver recently introduced "incentives", which gives a player in the game an additional incentive to take a certain action . For example, you may suspect your opponent calls too much and doesn't raise enough, so you can just set that incentive and it will include that in its math equations and give you something similar to an exploitative solution with a much simpler UX. This feature was literally just introduced a few months ago so it's still very much an active area of research, both for game theory nerds, and people trying to use the game theory nerd research to make money !
- Buttons840 2y ago
  I want to see strong AI used in video games, especially strategy games. People often retort that strong AI is not fun; it's too challenging and that's not what players want. But once we have a strong AI we can adjust its goal function in fun ways. What you're describing is effectively the same, and it's the first time I've seen this used in a strong AI.
  - currymj 2y ago
    GT Sophy is now a permanent feature in Gran Turismo and basically does this!
    - Buttons840 2y ago
      Thanks for bringing that to my attention. I still haven't seen an AI for a turn based strategy game. There's AlphaStar, but it wins via APM, not strategy.
  - dontlikeyoueith 2y ago
    In essence, you need strong (probably unbeatable) game AI in order to make more interesting weak (beatable but challenging and fun) game AI.
asdasdsddd 2y ago
I think it'd be interesting to see if an AI with visual input playing exploitatively can out perform AI playing GTO. In doing so, we can measure the effect of visual tells.
- tialaramex 2y ago
  You mean, can the exploitative strategy take money from fish faster? Yes. But it doesn't need to care about visual tells. The point of the optimal strategy is that it's unexploitable so you can disregard the other player's actions (in the game or outside it) entirely. All exploitative strategies are in turn exploitable.
  - bubblyworld 2y ago
    I think you have to be careful with saying stuff like "optimal strategies are unexploitable", because it usually means "unexploitable in a particular game theory sense". Whether the assumptions of the Nash equilibrium (or any of the others) make sense for your situation in a game of poker is an empirical question, right? It's not a given that playing a NE means you'll be "perfect" in the human sense of the word, or that you'll get the best possible outcome. The best superhuman poker AIs at the moment do not play equilibriums either, for instance.
    - tialaramex 2y ago
      I agree that because of, for example rake or table fees for cash games or competition structure for tournament in practice a game theoretically optimal choice may not be the right choice in practical play. However the situation with an AI powered competitor which uses exploitative play is identical to a human, the GTO play will gradually take their chips at no risk. It's not that they're optimal but that they've chosen not to be optimal and so that's why they lose money against GTO. The AI is at least unemotional about this, humans with a "system" easily get tilted by GTO play and throw tantrums. How can it get there with KToff? What kind of idiot bluffs here with no clubs? Well the answer will usually be the one that's taking all your chips, be better. Humans used to seeing exploitable patterns in the play of other humans may mistake ordinary noise in the game for exploitable play in a GTO strategy and then get really angry when it's a mirage.
bubblyworld 2y ago
Yes, this exists! Look up models based on counterfactual regret minimisation - they learn to exploit regularities in their opponents play, and often stray from the GTO play when it makes sense. I believe they have beaten poker professionals in thousands-of-hands playoffs but I may be misremembering.
- phreeza 2y ago
  The obvious follow up question: are there methods in use to bait such models into suboptimal play and then switch play style to exploit that?
- codethief 2y ago
  > I believe they have beaten poker professionals in thousands-of-hands playoffs I really don't know anything about poker AIs but could it be you are referring to Libratus and/or Pluribus[0]? [0]: https://noambrown.github.io/ https://noambrown.github.io/
  - bubblyworld 2y ago
    To be honest I didn't have a specific AI in mind, more the technique, but it sounds like these ones also use a variant of CFRM.
  - energy123 2y ago
    I do not believe these attempt to exploit regularities.
- currymj 2y ago
  no, CFR is mainly just a way of computing Nash equilibria and (although in some sense it is an online, iterative algorithm) would typically be used to precompute Nash strategies, not update them in real time. real poker playing systems augment the CFR strategies with some real-time solving, but just to get even closer to Nash at the end of a hand. on top of this, you could think about augmenting these systems to exploit weaknesses in opponent strategies. there is some work on this, but I don't think it's done much. The famous systems that played against professionals don't use it, they just try to get as close to GTO as possible and wait for opponents to screw up.
genewitch 2y ago
it took me a while to track this down last month: https://codegolf.stackexchange.com/questions/tagged/king-of-the-hill https://codegolf.stackexchange.com/questions/tagged/king-of-... there's also cops and robbers and at least one other "all AI compete against eachother" with the submitter usually making the first couple of "naive" bots.
- nhggfu 2y ago
  very interesting share ^ !
ggjkvcxddd 2y ago
An AI that plays a fixed exploitative strategy will end up getting figured out relatively quickly and counter exploited pretty hard. This actually happens in real life sometimes when people attempt to deploy poker bots online. Any exploitative AI also needs the ability to adjust in real time to a different exploitative strategy, which also needs to be not easily predictable, etc.

by mxwsn 2y ago

raincole 2y ago

> Imperfect information, human pressure... inevitably stray from Nash equilibrium Human pressure yes. Imperfect information no. When we talk about Nash equilibrium for a game like poker, it's already based on imperfect information.

mxwsn 2y ago
Ah right, thanks

waprin 2y ago

Within solvers, you can do something called "node locking", which means you "lock" a tree in the game node to play a fixed strategy. You would typically lock it to play as you suspect your opponent plays. This lets the solver calculate the optimal exploitative solution against your specific oppoents. Piosolver, the first public solver and the one mentioned in the article, has this feature. However, what often happens is if you lock one node, then several other nodes in the game tree over-adjust in drastic ways, forcing you to lock all of the, which may be infeasiable. As a result, Piosolver recently introduced "incentives", which gives a player in the game an additional incentive to take a certain action . For example, you may suspect your opponent calls too much and doesn't raise enough, so you can just set that incentive and it will include that in its math equations and give you something similar to an exploitative solution with a much simpler UX. This feature was literally just introduced a few months ago so it's still very much an active area of research, both for game theory nerds, and people trying to use the game theory nerd research to make money !

Buttons840 2y ago
I want to see strong AI used in video games, especially strategy games. People often retort that strong AI is not fun; it's too challenging and that's not what players want. But once we have a strong AI we can adjust its goal function in fun ways. What you're describing is effectively the same, and it's the first time I've seen this used in a strong AI.
- currymj 2y ago
  GT Sophy is now a permanent feature in Gran Turismo and basically does this!
  - Buttons840 2y ago
    Thanks for bringing that to my attention. I still haven't seen an AI for a turn based strategy game. There's AlphaStar, but it wins via APM, not strategy.
- dontlikeyoueith 2y ago
  In essence, you need strong (probably unbeatable) game AI in order to make more interesting weak (beatable but challenging and fun) game AI.

asdasdsddd 2y ago

I think it'd be interesting to see if an AI with visual input playing exploitatively can out perform AI playing GTO. In doing so, we can measure the effect of visual tells.

tialaramex 2y ago
You mean, can the exploitative strategy take money from fish faster? Yes. But it doesn't need to care about visual tells. The point of the optimal strategy is that it's unexploitable so you can disregard the other player's actions (in the game or outside it) entirely. All exploitative strategies are in turn exploitable.
- bubblyworld 2y ago
  I think you have to be careful with saying stuff like "optimal strategies are unexploitable", because it usually means "unexploitable in a particular game theory sense". Whether the assumptions of the Nash equilibrium (or any of the others) make sense for your situation in a game of poker is an empirical question, right? It's not a given that playing a NE means you'll be "perfect" in the human sense of the word, or that you'll get the best possible outcome. The best superhuman poker AIs at the moment do not play equilibriums either, for instance.
  - tialaramex 2y ago
    I agree that because of, for example rake or table fees for cash games or competition structure for tournament in practice a game theoretically optimal choice may not be the right choice in practical play. However the situation with an AI powered competitor which uses exploitative play is identical to a human, the GTO play will gradually take their chips at no risk. It's not that they're optimal but that they've chosen not to be optimal and so that's why they lose money against GTO. The AI is at least unemotional about this, humans with a "system" easily get tilted by GTO play and throw tantrums. How can it get there with KToff? What kind of idiot bluffs here with no clubs? Well the answer will usually be the one that's taking all your chips, be better. Humans used to seeing exploitable patterns in the play of other humans may mistake ordinary noise in the game for exploitable play in a GTO strategy and then get really angry when it's a mirage.

bubblyworld 2y ago

Yes, this exists! Look up models based on counterfactual regret minimisation - they learn to exploit regularities in their opponents play, and often stray from the GTO play when it makes sense. I believe they have beaten poker professionals in thousands-of-hands playoffs but I may be misremembering.

phreeza 2y ago
The obvious follow up question: are there methods in use to bait such models into suboptimal play and then switch play style to exploit that?
codethief 2y ago
> I believe they have beaten poker professionals in thousands-of-hands playoffs I really don't know anything about poker AIs but could it be you are referring to Libratus and/or Pluribus[0]? [0]: https://noambrown.github.io/ https://noambrown.github.io/
- bubblyworld 2y ago
  To be honest I didn't have a specific AI in mind, more the technique, but it sounds like these ones also use a variant of CFRM.
- energy123 2y ago
  I do not believe these attempt to exploit regularities.
currymj 2y ago
no, CFR is mainly just a way of computing Nash equilibria and (although in some sense it is an online, iterative algorithm) would typically be used to precompute Nash strategies, not update them in real time. real poker playing systems augment the CFR strategies with some real-time solving, but just to get even closer to Nash at the end of a hand. on top of this, you could think about augmenting these systems to exploit weaknesses in opponent strategies. there is some work on this, but I don't think it's done much. The famous systems that played against professionals don't use it, they just try to get as close to GTO as possible and wait for opponents to screw up.

genewitch 2y ago

it took me a while to track this down last month: https://codegolf.stackexchange.com/questions/tagged/king-of-the-hill https://codegolf.stackexchange.com/questions/tagged/king-of-... there's also cops and robbers and at least one other "all AI compete against eachother" with the submitter usually making the first couple of "naive" bots.

nhggfu 2y ago
very interesting share ^ !

ggjkvcxddd 2y ago

An AI that plays a fixed exploitative strategy will end up getting figured out relatively quickly and counter exploited pretty hard. This actually happens in real life sometimes when people attempt to deploy poker bots online. Any exploitative AI also needs the ability to adjust in real time to a different exploitative strategy, which also needs to be not easily predictable, etc.