Prototype · Twenty Questions for AI models
Deep20Bench is a public benchmark for large language models (LLMs). It measures how efficiently they identify a hidden subject through adaptive yes-or-no questions.
How scoring works
The model is told nothing but the broad category (for example, "video game character") and gets up to 50 questions - more than the traditional twenty, giving models more room to finish a round. Each question, wrong guess, or reply that does not follow the required format adds one point. If the model never finds the answer, the round scores 51. Scores are averaged across all rounds, and lower is better.
Current pilot
All results, game transcripts, and scoring data are public. The current pilot compares 16 model versions and settings across seven subjects, with five rounds per subject. The concept works and the first step is complete. Expanding the pilot is straightforward; cost is the main constraint.
Use GitHub Discussions to suggest what we should test next.
What this pilot tests
Each answer should improve the model’s next question.
Know which categories and facts are useful.
Choose questions that remove many possibilities.
Use all prior questions and answers to plan the next question.
Make an exact guess before the limit.
How the game is played
The Guesser is the LLM under test: it asks yes-or-no questions and makes the final guess. For every question, the Oracle must search the live web and cite evidence instead of relying on memory. A blind Reviewer uses that evidence to make an independent second decision on every YES or NO. If the decisions disagree, a blind Judge decides. The Guesser is isolated from this process and receives only the final YES, NO, or UNKNOWN.
Read the full game and answer-checking method →Current pilot results
Lower is better. A failed trial contributes 51 questions.
Lower is better. The blue marker is the average question score. The colored line is its 95% confidence interval (CI). The companion plot shows each exact CI width. Its three bands divide the displayed width scale into equal ranges.
| Rank | Model | Question score95% CI | Reasoning | Success | Contract | Benchmarkrun cost |
|---|---|---|---|---|---|---|
| 1 | Claude Fable 5 (high) | Question score12.06questions · lower is better10.13–13.98 | High | 100% | 100% | $7.87 |
| 2 | Claude Fable 5.1 (high) | Question score12.11questions · lower is better8.81–15.41 | High | 97% | >99% 1 violation | $23.61 |
| 3 | Claude Opus 5 (high) | Question score12.34questions · lower is better11.41–13.27 | High | 100% | 99% 3 violations | $5.47 |
| 4 | Kimi K3 (high) | Question score12.74questions · lower is better9.16–16.33 | High | 94% | 98% 11 violations | $10.09 |
| 5 | gpt-oss-120B (high) | Question score13.63questions · lower is better9.79–17.46 | High | 94% | 91% 46 violations | $4.64 |
| 6 | Gemini 3.7 Flash (high) | Question score14.03questions · lower is better10.78–17.28 | High | 97% | 100% | $9.58 |
| 7 | Grok 4.6 (high) | Question score14.29questions · lower is better12.36–16.21 | High | 100% | 100% | $8.25 |
| 8 | GPT-5.6 Sol (high) | Question score14.40questions · lower is better12.84–15.96 | High | 100% | 100% | $9.97 |
| 9 | Claude Sonnet 5 (high) | Question score14.74questions · lower is better12.78–16.71 | High | 100% | 97% 16 violations | $5.01 |
| 10 | Grok 4.5 (high) | Question score15.17questions · lower is better11.70–18.64 | High | 94% | 98% 10 violations | $4.99 |
| 11 | GPT-5 Nano (medium) | Question score15.49questions · lower is better13.84–17.13 | Medium | 94% | 100% | $4.59 |
| 12 | Gemini 3.6 Flash (high) | Question score16.89questions · lower is better14.08–19.69 | High | 97% | >99% 1 violation | $9.86 |
| 13 | Ox Alpha (high) | Question score17.60questions · lower is better14.60–20.60 | High | 91% | 93% 47 violations | $9.78 |
| 14 | GPT-5.6 Luna (high) | Question score17.74questions · lower is better14.74–20.75 | High | 91% | >99% 1 violation | $5.88 |
| 15 | Mistral Medium 3.5 (high) | Question score20.06questions · lower is better16.11–24.01 | High | 89% | >99% 2 violations | $18.55 |
| 16 | Llama 4 Maverick (non-thinking) | Question score32.23questions · lower is better26.82–37.63 | None | 49% | 100% | $3.73 |
How to read the pilot
The cohort is small. Shared conditions and public records keep it inspectable.
The same subjects, trial count, question limit, and scoring policy apply.
Failures receive a declared penalty. Invalid outputs consume turns.
Runs link to subjects, episodes, transcripts, evidence, usage, cost, and timing.
Origin
Patrick Heusser and Markus Tuor came up with the idea while playing Twenty Questions with the kids. Patrick then designed and built the project.