AI's Superhuman Strategy: Nash Equilibrium in Poker and Diplomacy with Noam Brown

Noam Brown, a research scientist at Meta AI, discusses his groundbreaking work on AI systems that have achieved superhuman performance in complex strategic games. The conversation primarily focuses on Libratus and Pluribus, which conquered No Limit Texas Hold'em poker (heads-up and multiplayer, respectively), and Cicero, an AI capable of strategic negotiation in the game of Diplomacy. Brown highlights the fundamental differences between perfect information games like chess and imperfect information games like poker, emphasizing the latter's complexity due to hidden information, bluffing, and the need to reason about probabilities of actions rather than just optimal moves. He explains how the "No Limit" aspect of Texas Hold'em introduces significant psychological pressure and strategic depth, allowing for aggressive betting that can exploit human risk aversion. A core concept explored is the Nash equilibrium, which posits that in any finite two-player zero-sum game, an optimal strategy exists that guarantees not losing in expectation, regardless of the opponent's play. Brown clarifies that "in expectation" accounts for the high variance of poker, meaning long-term break-even or profit. The AI systems achieve this by employing a process called self-play, specifically using Counterfactual Regret Minimization (CFR). This algorithm learns by simulating games against itself, evaluating alternative actions, and updating "regret" values to converge towards a balanced, unpredictable strategy that approximates the Nash equilibrium. Neural networks are crucial for generalizing these learned strategies across the vast state space of games like poker, which can have an astronomical number of decision points. The discussion also delves into the "Game Theory Optimal (GTO)" versus "exploitative play" debate in poker. Brown asserts that while many human players historically favored reading opponents and exploiting weaknesses, the success of AI systems like Libratus, which strictly adhered to GTO by approximating the Nash equilibrium, demonstrated the power of a balanced, unpredictable strategy. Libratus, for instance, crushed top human players by consistently playing optimally without attempting to adapt or engage in "mind games." This underscores the AI's ability to maintain an unbeatable strategy even when its play style is known. Beyond winning, the podcast touches on the broader implications of advanced AI, particularly in the context of video games and human-AI interaction. Brown notes the distinction between an AI designed to win and one designed to be "fun to play with" or "fun to watch," citing examples from games like Civilization. He expresses excitement about the potential of large language models (LLMs) to revolutionize Non-Player Characters (NPCs) in open-world role-playing games, enabling more dynamic, unscripted, and emotionally complex interactions. This shift could lead to new genres of games focused on cooperation, drama, or negotiation, moving beyond the prevalent focus on combat, and offering a "safer" environment for exploring complex human-like interactions.

a lot of people were saying like oh this whole idea of Game Theory it's just nonsense and if you really want to make money you got to like look into the other person's eyes and read their soul and figure out what cards they have but what happened was where we played our bot against four top heads up no limit Hold'em poker players and the bot wasn't trying to adapt to them it wasn't trying to exploit them it wasn't trying to do these Mind Games it was just trying to approximate the Nash equilibrium and it crushed them

I'm drawn in by the beauty of the game

in any finite two-player zero-sum game there is an optimal strategy that if you play it you are guaranteed to not lose an expectation no matter what your opponent does

Poker is a very high variance game so you're gonna have hands where you win you're gonna have hands with your lose even if you're playing the perfect strategy you can't guarantee they're going to win every single hand but if you play for long enough then you are guaranteed to at least break even and and practice probably one so that's an expectation

there's a difference between making an AI that wins a game and an AI That's fun to play with

I would not be surprised at all if Elder Scrolls 7 was using large language models for their NPCs

it's just so much harder to make an AI that can talk with you and cooperate with you than it is to make an AI that can fight you

the way that we do it is with this process called self-play... it learns how to play the game by playing against itself

the value of an action depends on the probability that you're going to play it

the way that the Bots handle it that are really successful they have an explicit theory of mine so they're explicitly reasoning about what are what's the common knowledge belief what does what do you think I have what do I think you have what do you think I think you have