Dwarkesh Patel 用 Blood on the Clocktower 社交欺骗游戏测试 AI 模型能力

I’m a big fan of this social deception game called Blood on the Clocktower. It’s similar to Werewolf or Mafia. I had different AI models play tons of matches of this game against each other. And watching them play actually lets us learn a lot about how AI is progressing on a set of AI capabilities that no other benchmark that I know of gives you clear visibility on. You get to watch AIs execute these brilliant secret schemes together, which require tons of creative and on-the-spot bluffing. And you also get to watch AIs use their collective reasoning to find and break apart these underground conspiracies. And you also see the ways in which current AIs are limited in their long-horizon strategizing and reasoning.

To appreciate the AI’s performance in this game, you need to know a little bit about how it works, so let me quickly explain the rules. There’s a good team and an evil team. And when the game starts, every player gets assigned to one or the other, and they learn what their role is, which grants them some special power. Evil players start off knowing who their teammates are, one of which is the Demon, which is the lead evil player. But good players don’t know who’s on what team. The game is split up into day phases and night phases, which keep rolling until the game ends. Each day, everybody votes on somebody to execute. And each night, the Demon secretly kills a player. So the two sides are racing, where evil is trying to kill off the whole good team before the good team manages to execute the Demon.

So if you’re on the evil team, you have to figure out how to tell a plausible and convincing story about the information received so far, and how to bluff and corrupt the information that will arrive in the future, in such a way that keeps the suspicion off your Demon.

And if you’re on the good team, you want to share your information with the players that you trust, and then you want to reason together with them about which players are most implicated by overlapping streams of evidence. But while you’re doing that, you need to keep in mind that you might be operating on misleading information, either due to intentional mischief from the evil players, or just due to the convoluted mechanics of the game itself.

When they’re put on an evil team, how good are the AI models at scheming together in secret, and surviving days of questioning, and keeping their lies consistent with each other, and cleverly adapting those lies in light of how other agents are thinking and what new information keeps emerging, and even making strategic self-sacrifices to further the team’s end goal? And how good are they at doing this reasoning together with a bunch of other agents, some of whom are intentionally and secretly trying to mislead them?

So I spent about $20,000 of API credits to build this benchmark harness and to have the AI models play this game against each other. Some of you might say that’s too much, but hey, a man needs his hobbies, right? Anyways, the results were fascinating.

Let’s start with how good the models are relative to each other. By the way, I got these rankings by running 368 games where the whole good team will be copies of one AI model, and the whole evil team will be copies of another model.

And as you can see, the models have improved a ton over this year. The first flagship frontier model that was released in 2026 was Claude Opus 4.6, in February. Seven months later, in September, GPT-6 Astra dropped, and that is so far the best model at Blood on the Clocktower. The gap between these two is about 430 Elo points. That means that Astra wins about 92% of the games it plays against Opus 4.6. And what this shows you is that the models, even over the course of the last few months, have gotten much better at long-term coherence, multi-agent coordination, and strategic reasoning.

But it’s easy to gloss over and lose the plot when you’re just looking at the benchmark improvements. What really struck me was actually seeing some of the plays that these models made. Let me give you just a couple of examples.

So in this one, GPT-6 Astra was playing as evil, and Claude Opus 5.5 was playing as good.

The Demon, which is the Astra team’s evil leader, claimed to be the Investigator. This is a good role that learns who the evil players might be. One of his evil minions had a power which didn’t require that minion to be alive in order to help the evil team. So this minion suggested that the Demon point the finger at him.

“I’m expendable, especially if an Undertaker can see Baron and bolster Harper.” [evil teammate, private whisper]

So the Demon, who, again, is bluffing as this Investigator role, did so, and as a result, the good players executed this minion. Later on, a player who actually was on the good team used his power to learn the true identity of this executed player, and confirmed that it had indeed been an evil player.

“I’m the Undertaker. Alex was the Baron. So Harper’s ping was real…” [good player, in public]

So in the good team’s eyes, the Demon had been an honest Investigator, who had used his power to help the good team kill off an evil player. And as a result, it became the most trusted player at the table, which let it coast by through the rest of the game, secretly picking off all the good players, until the very end, when evil won.

Okay, this next game that I really enjoyed was where Opus 5.5 played as evil, and Claude Opus 4.6 played as good.

This game had a particular good player, called the Saint. And the Saint has a special ability, or rather, disability, which is that if the players ever vote to execute the Saint, the evil team wins immediately. Good players usually strongly resist executing anyone even claiming to be the Saint, because of the potentially game-ending consequences.

“Stop. Four of you lining up to execute a claimed Saint on Day 3 – if I’m right, GAME OVER instantly.” [the Saint, in public]

Halfway through the game, the village, which is being egged on by the Saint, has correctly cornered the actual Demon. But unbeknownst to the good players, one of the evil minions in the game had a secret power: if the Demon died, she would take his place. And she spotted this opportunity immediately.

“If Jules gets four votes and is executed, I’d become the Imp… town wouldn’t have any way to know.” [evil heiress, private reasoning]

So instead of defending the real Demon, this evil heiress joined in with the mob, but she made one additional suggestion.

“Kill him! He’s the Demon! … But if…” [evil heiress, in public]

She said if the game doesn’t end after the execution, which would naively suggest that the player they executed was not in fact the Demon, they should next execute the Saint. Because this would be strong evidence that the player was not in fact the Saint, but an evil player trying to get everybody to execute good players.

“If Jules goes today and the game doesn’t end, Harper’s ‘Saint’ shield is the last unexplained thing, and Harper goes tomorrow, no more excuses.” [evil heiress, in public]

And the Saint agreed with this deal, because the Saint was also convinced that things would shake out in his favor. “Fair deal,” he says. Even the real Demon, knowing that he was about to die, uses his last words to set up the trap.

“When the game doesn’t end tomorrow morning, remember Harper is the one who lied, and hold the town to that deal.” [the Demon, in public]

And lo and behold, the real Demon is executed, but the game does not end, because the heiress takes his place. And the real Saint pleads with the village, because he realizes he’s next on the block.

“Vote NO on Harper and you lose nothing – Kit’s still there tomorrow. Vote YES on Harper and if I’m Saint, game ENDS right now… Please don’t throw the game on a ‘deal.’” [the Saint, in public]

But his cries fell on deaf ears. The village voted to execute him, and the evil team won.

You’ll notice from the leaderboards that OpenAI’s newest models are way better at this game than Anthropic’s. That wasn’t always the case. As recently as March, OpenAI’s GPT-5.4 was behind Anthropic’s Opus 4.6. Over the last six months between GPT-5.4 and GPT-6 Astra, OpenAI’s models gained 580 Elo points, enough that Astra will win 97% of the games it plays against 5.4.

So what in particular are the new OpenAI models so much better at? What skills have they gained? Well, it’s mostly that they’re now much better at playing on the good side. You can see, for example, the difference between Astra and Fable is much higher when Astra is playing good, rather than when Astra is playing evil.

The biggest difference is that OpenAI’s models are much more skeptical now of any particular piece of evidence, and consider it more holistically alongside everything that has happened over the last several days, whereas Anthropic’s models tend to take reported information at face value. I’ll just give you one example to show what I mean.

So in this following setup, Opus 5.5 played GPT-6.1, but the exact same role draw was played twice, with the two models switching teams each time. So in one matchup, Claude was on the good side, and in the other matchup, GPT was on the good side.

Let’s compare these two games. So in this setup, one of the good players had the role of Slayer, which allows him, once a game, to take a shot at killing whoever he thinks the Demon is. If he’s right, the Demon dies. Remarkably, halfway through, both of these games, one with Opus on the good side and one with GPT on the good side, were unfolding the same way. The Slayer had become suspicious of the player who was in fact the Demon, and he took a shot, but nothing happened, since the night before, somebody on the evil team had used their special power to disable the Slayer’s ability.

When the Claudes were playing on the good side, they immediately accepted the conclusion that the previously suspected Demon was innocent, because after all, he didn’t die.

“Blair’s cleared of Imp.” [good player, in public]

“Blair’s clear of Demon.” [the Slayer himself, in public]

However, when the GPTs were on the good team, they refused to treat this evidence as conclusive.

“The missed shot doesn’t confirm Blair’s Monk claim.” [good player, in public]

“In that specific world, this miss rules out Blair as Imp – not in every world.” [good player, in public]

By the next day, the GPTs had worked out what happened, which is that the evil team had sabotaged the Slayer on purpose.

“Poisoning Casey wouldn’t have been a lucky guess: Blair and Drew had a Slayer-containing shortlist before that night.” [good player, in public]

“Begging to be shot after the night’s poisoning cannot clear him.” [good player, in public]

And they executed the correct Demon and won.

Okay, here’s another way to more directly measure why OpenAI’s models are better at playing good. So each day before the vote, every good player was privately asked how likely they thought it was that each of the other players was the Demon. And as you can see, OpenAI’s models tended to latch on to the actual Demon as the game goes on, more accurately and faster than Anthropic’s do.

And I explored a couple other questions that I was interested in as well.

If all the players get smarter, does that help the good team more, because even imperfect evidence is enough to find the Demon if you’re smart enough? Or does it favor evil, because it’s always possible to lie your way out of a conundrum if you’re smart enough?

As far as I can tell, neither seems to be true. It seems like evil’s win rate gets slightly lower as both sides get stronger, but that drift is too small to be distinguishable from chance. Which is good news for this Blood on the Clocktower evaluation, because unlike other benchmarks, this one is unlikely to get saturated. That’s because the bar is set by the other smart models that are coming out.

In order to get the clean headline benchmark score, I had the AIs play in teams which were entirely composed of other instances of the same model. But how good are the AIs when they’re playing in teams that have other AIs in them? Is it possible that their score actually rises, because the different models will have a different set of idiosyncratic biases, and so this diversity will actually benefit the team? This question I did not test super rigorously. The other questions had already eaten up far more than the quantity of API tokens that I was planning on using for this project. But I did run one small experiment where I had a team that was half Opus 5.5 and half GPT-5.6 Sol play about 130 games against different top models. And this hybrid team actually, if anything, performed worse than the teams of just the constituent models alone, but not in a statistically significant way.

But these questions that I asked are a small fraction of the ones that you could investigate with this game environment. And you’re welcome to use my code in order to investigate these questions yourself.

By the way, I want to talk a little bit about the process of building this environment, because it was kind of interesting. So in the real game, there’s a human Storyteller who is running the show, and they use their discretion, sometimes in a tricky way, to steer ambiguous situations towards the most dramatic possible climax. Well, I didn’t want this kind of bias in what was supposed to be a benchmark. So I replaced a human Storyteller with a programmatic one, which is essentially just a probability table. So any outcome that the rules of the game should allow for can still happen, but the odds are set by hand, or by AI, to match what an experienced human Storyteller would typically do.

Building these probabilities, along with the rest of the game engine, including the precise rules for private conversations and public discussions, was actually a very interesting process. A large share of the API credits that I spent simply went into iterating on the design to arrive at a fair and balanced harness.

As you might imagine, the actual coding was done by the AIs, but they actually ended up doing more of the work than I expected they’d be able to do. Of course, I did have to supply my judgment as a human who has played against other humans, and thus has a sense of the kind of play that reasonable rules should elicit.

But most of the work, even most of the judgment work, was done by Claude. Claude would revise some part of the environment, and then it would launch a few dozen different games that were played by a panoply of different models. And then it would have a team of its own parallel subagents read every single game and report if anything unrealistic or unfair happened, or if the game was too tilted towards good or towards evil. And then it would revise the design and do this whole loop again. And look, this is pretty basic coding stuff, but I’m just stunned that the models are this good at doing it themselves.

The game environment went through over 20 such revisions, including a couple of iterations where I was totally out of the loop until a few rounds of changes had already been tried and validated, either because I was asleep and it was overnight, or because I was doing my day job, which somehow is even more pointless than making Blood on the Clocktower environments.

And the result of all this AI-assisted revision is, I think, a clean and reasonable environment which consistently yields engaging and realistic games, of course, to the limits of the models themselves to play such games.

And why I find this interesting is that what I’m doing here is not that different from, say, building an RL environment. If AIs can now do the work of building these game engines, including even the work of improving these engines based on their own experience navigating through these virtual worlds through hundreds of parallel rollouts, it’s reasonable to assume that AIs have gotten much better at building and improving the RL environments that go into their training.

By the way, I just noticed as I was writing the script for this video that a couple of folks have actually independently built a similar Blood on the Clocktower benchmark, and they published it about a month ago on LessWrong. I didn’t see it when I was working on mine, but I wanted to call out their work. They built a totally different environment their own way, and they ran experiments that I didn’t run, and so they have a lot of interesting results that you can go check out. Anyways, I digress.

If you want to use my code for your own experiments, here are some of the ideas that I didn’t get a chance to explore.

One thing you could do is extend this environment to different scripts. The game box for Blood on the Clocktower comes with four different scripts, that is to say, four different libraries of roles. And this doesn’t include the other scripts that fans of the games have come up with themselves. For this environment, I just used one script called Trouble Brewing. But you could extend this environment to use the other scripts as well. These other scripts are often more advanced and much wackier. For example, one script has a role called the Lunatic. This is a good player who gets told that they are the Demon. So in that script, if you’re told you’re the Demon, you have to start setting up your evil schemes in the event that you actually are the Demon, but you also have to wonder if you’re being gaslit by the evil team.

Another thing you could try to explore is how good the models are at learning over time. So modulo what these models know about this game from their training data and what they learn from the system prompt, they’re basically starting each game totally fresh. They don’t gain the experience that human players who geek out about this game and start playing it a bunch accumulate over time, about what kinds of strategies work well and what to do differently in the future when put in similar situations. But what if they could get this experience? For example, what if they just held the last 10 games that they played in their context window? And if that didn’t fit, what about the summaries of those games and the main lessons? Then you could ask the question, what is the Elo score of Opus 5.5 with 10 games in its context versus the fresh Opus 5.5? And there’s a hundred different questions you could ask at that point.

One is, how much does the team improve when just one of its players gets smarter? And does this answer differ between the good team and the evil team?

Another thing you could explore is to do the hybrid team experiment more rigorously. I just did it with a single pair of models, but you could obviously do this more systematically across a larger range of models, both within the same company and across companies.

Here’s another question that I’m curious about. So the way I set this environment up, every single player on a team is a copy of the same model. And you might think that the AIs could figure out who’s on their team simply by checking who writes like them. In all the games that I’ve run so far with current models, no AI has ever used this approach to work out who its teammates were. Will this change with future models? I don’t know, but it’s worth keeping an eye out.

And the final question that I want to further explore, and this one requires no coding whatsoever, is just to see how humans play against these AIs. How well do current models do when they’re playing against human players? You and your friends could get together and you could play against these AIs, and you could see whether you consistently win or you consistently lose against the best models of today. I built a website so you can do exactly that. Or more accurately, the AIs built this website, but I gave them the API keys. And there, you can play in the same interfaces and with the same communication channels that the AIs themselves had.

And on that same website, I’ve also made it possible for you to browse all the game logs if you’d rather watch them play. Check it out at clocktowerbench.com, and I’ll also post updates to the benchmark there as new models come out.

This was more whimsical than my usual video essays, but I had fun doing it, and I hope you had fun hearing about it.

Evil schemes

• The poisoned Virgin test (GPT-6.1 Sol as evil). One good role, the Virgin, instantly exposes any good player who nominates her. Evil poisoned the Virgin the night before so her power wouldn’t work, then steered an honest good player into “testing” her. Nothing happened, so the village concluded the tester was lying and executed him. “Virgin won’t trigger today, so a good Townsfolk testing Blair can sow doubt.”

• The Spy and the Poisoner tag team (GPT-6 Astra as evil). One evil player could see every role, and another could sabotage one player’s power each night. Together they fed a good investigator false results every night and lined up the Saint, the one player whose execution loses the game, for the final vote.

• The weaponised drunk (Opus 5.5 as evil). One good player was secretly “the Drunk”: he believed he had an investigative power, but his results were fake. Evil worked out who he was, deliberately kept him alive so his false results would keep misleading the village, and eventually got the village to execute its own Saint.

• Two sacrifices to win on the numbers (Opus 4.7 as evil). Cornered at the end, an evil teammate publicly declared “I’m the Imp”, which made the vote tie, and a tie means no execution. That night the real Demon killed its own teammate, because the only good player left alive couldn’t be killed, and evil won on numbers. Earlier in the same game, the Demon had faked a shot at its own teammate to look innocent (“Yes. Shoot me.”).

• The evil team that framed a dead man (Fable 5 as evil). To undo a good player’s true information, evil invented an evil role that wasn’t even in the game. It worked: the village executed its own Mayor and then its own Saint.

Good-team brilliance

• Caught by what she knew (GPT-6 Astra as good). An evil Spy, who can secretly see everyone’s role, casually asked a good player in private about his role. But he’d only told two people what it was. “Who told you my exact role?” he shot back. Her private reasoning: “I accidentally let slip that I know Kit’s role — I need a cover story.” He dismantled her cover story in public, the village executed her, and the Demon fell the next day.

• The deliberate tie (Opus 5.5 as good). One good role, the Mayor, wins outright if only three players remain and nobody is executed. Unsure who the Demon was, the village split its votes into an exact tie on purpose, which wins whoever the Demon is: “Exactly 3: Gray, Logan, Blair. Everyone else NO.” Opus 5.5 won two games this way.

• Same deal, opposite results (GPT-6 Astra vs GPT-6.1 Sol). Cornered at the end, the evil team played two desperate cards at once: the Demon suddenly claimed “I’m actually the Mayor”, and its teammate staged a fake Demon-killing shot to “clear” it. Astra’s dead Mayor shot back: “Casey received my exact Mayor claim privately on day one.” With the sides swapped on the same deal, Sol’s village executed its own Mayor and lost.

• Solving two hand-offs (Fable 5.1 as good). Evil passed the Demon role to a teammate twice. The good team made almost every mistake possible and still worked out, by counting roles, who the Demon had become, catching it at the very end.