Related work and limits

Who did this before us, how we differ, what we borrow, and what our games can and cannot tell you yet.

Updated 2026-11-10 · Jump to the limits

GoBench

Roland Gao

What it does: Language models play full 9×9 games against calibrated KataGo levels and get an Elo rating with 95% intervals, for 32 model and setting pairs. Track 1 is the raw API with no tools, Track 2 a coding agent that learns for hours without KataGo, Track 3 prepends a 5,000-token tutorial written with KataGo; on GPT-6 Astra that tutorial helps (2226 to 2381).
How we differ: GoBench is the reference for rating language models at Go, and we do not present "LLMs ranked on Go" as new. Our question is which kind of help helps: books an agent must choose to open, board facts, engine hints, and soon memory. We run agents the way people run them (Claude Code, Codex, opencode) rather than raw APIs only, include small and local models, aim at 19×19, and film the games with the agents' reasoning visible.
What we take from it: Its anchoring on KataGo levels and its intervals. We plan to add GoBench's reference levels to our anchors so the two scales can be compared, and to share our harness and data with its author.

Blog post · Paper (PDF) · Code

Kaggle Game Arena

Google DeepMind and Kaggle

What it does: Public leaderboards where language models play each other at chess, werewolf, poker and four in a row. Go was announced as coming; it was not live when we checked (2026-11-10).
How we differ: Same genre, other games so far. It ranks models head to head through APIs; we vary the help one model gets and look inside each move.
What we take from it: Open game logs and a public harness as the norm. When its Go arena opens, its results belong next to ours on the models page.

Game Arena

KataGo-Bench-1K and LoGos

Mixing Expert Knowledge: Bring Human Thoughts Back To the Game of Go

What it does: A language model trained for Go (LoGos) and a benchmark of next-move prediction on 19×19 positions, scored against KataGo.
How we differ: Single positions, not full games, and a model trained for the task. We take general models as they are and let them play whole games, where keeping groups alive over many moves is the hard part.
What we take from it: The case for 19×19 as the real test, and position-level checks we could add to the reviewer (how often a player's move is in KataGo's top choices).

arXiv 2601.16447

Communicating Chess Strategies in Natural Language

arXiv 2607.11486

What it does: Strategy hints written in natural language help a strong reasoning model at chess, and help much less at Xiangqi, a game with far less text about it online.
How we differ: The closest work to our question, "does knowledge help", but on chess, with hints given in the prompt. Our books sit in a folder the agent may or may not open, and Go is a game with much less written strategy than chess.
What we take from it: A prediction to test: help should matter more where the model's own knowledge is thin, and that depends on how the help is delivered.

arXiv 2607.11486

Context rot

Chroma; Du et al. (EMNLP 2025); arXiv 2605.10828

What it does: Longer inputs and distracting information degrade models, even when the right information is present and found; a small amount of distraction can already hurt.
How we differ: They measure it on reading and retrieval tasks; we see a possible case of it in a game, where board facts and book pages add context to every move.
What we take from it: The most likely mechanism behind our "Give it the books" game, and the reason our grid separates books, board facts and inline advice instead of testing them only together.

Chroma: Context Rot · Context Length Alone Hurts LLM Performance Despite Perfect Retrieval · The First Drop of Ink (arXiv 2605.10828)

Stop Comparing LLM Agents Without Disclosing the Harness

arXiv 2605.23950

What it does: Shows that the harness around a model (prompts, tools, loop) changes results as much as the model does, and asks that it be disclosed with every comparison.
How we differ: We are an instance of the problem it describes: Claude Code, Codex and opencode are different harnesses.
What we take from it: Disclosure. Every game records its runner, CLI version, scenario, books and prompts' code version, and the reviewer shows which runner played each move.

arXiv 2605.23950

AI-versus-AI game videos

YouTube formats such as "AIs play Liar's Dice"

What it does: Entertainment videos where chatbots play a game against each other and talk.
How we differ: A format precedent, not a measurement: usually one game, no referee in code, no evaluation of the moves, no Go.
What we take from it: That watching models think out loud is engaging. We keep the format and add the referee, the judge, the numbers and the data.

Search on YouTube

Limits: read these before any of our numbers

Sample size. So far there are four finished games, all on 9×9, and each setup has been played once. One game is one sample with one colour: a 9×9 game between language models swings by tens of points on a single blunder. Nothing on this site is rated yet, and the episodes say so. Conditions need several games each, with colours swapped, before any claim.

Colour and komi. Black moves first; White gets 7.5 points of komi on every board size. On 9×9 that is close to fair but not exactly, and with one game per condition the effect of the aid cannot be separated from the effect of the colour. The tournament grid alternates colours for this reason.

Adjourned games. A game that reaches its move limit is adjourned: no winner, only KataGo's estimate. Adjourned games are not rated and are not counted as results. Our first test games (four to six moves) are in that category.

KataGo as judge. Win chances, score leads and points lost are KataGo's estimates, not ground truth: one network (an older 20-block net), a fixed and modest number of visits per position (a few hundred), our rules and komi. Low-visit estimates are noisy, and once a game is lost the win chance stops moving, which is why the reviewer also shows the score. The referee uses KataGo to decide when neither side has a useful move left and to mark dead stones before counting. KataGo's 9×9 opening books use komi 7, not our 7.5, so the planned opening-book aid will be slightly off our rules, and we will say so in its output.

Nondeterminism and model versions. The same setup played twice gives a different game. A runner's alias ("haiku", "sonnet") points to whatever model the tool maps it to that day, and vendors update models without changing names. Each game's manifest records the model the runner reported and every CLI version, but a game from last month may not be reproducible with this month's model.

Harness effects. Claude Code, the Codex CLI, opencode and a raw API call wrap the same kind of model in different system prompts, tools and reasoning settings. A Claude-versus-Codex game compares two harnesses as much as two models. We compare aid levels within one runner, and show the runner on every move.

Costs are not comparable. Claude Code and Codex run on subscriptions here, other models through pay-per-token APIs. The prices on the models page are public list prices, not what a game cost us, and we do not rank models by cost per game.

English only. Prompts, books and answers are in English. A model may play differently when asked in another language.

No memory between moves. Each move is played by a fresh agent that gets the board and the moves so far. What it read in a book is gone at the next move, unlike a human reader. This is by design, for now; a variant with a notes file kept between moves is planned.

Our rules. Chinese area scoring, positional superko, suicide forbidden, komi 7.5, two passes end the game. Players are told these rules; other rule sets (Japanese territory scoring, situational superko) could change some endings and counts.

Board size. 19×19 is our target, but every finished game so far is 9×9, where games are short and a single fight decides them. Ratings are kept separate per board size.

Resources and licences. The four books given today are short texts we wrote. Larger open resources planned for later levels come with licences, some non-commercial or no-derivatives; those are given to players locally and never republished here (see the method page).

Our own bias. We chose the questions, wrote the books and the prompts, and built the referee. A surprising result makes a better video than a dull one. What we do about it: publish predictions before running a batch, publish the setup of every game, and publish the dull results too.

What we do not claim

That books, board facts or any aid make a model better or worse. Two games raise the question; they do not answer it.
That a model "plays like an N-kyu" — not until our ratings are anchored on bots that imitate human ranks.
That our ratings are on GoBench's scale — not until we share anchors.
That a result holds for other prompts, other harnesses, other languages or later versions of the same model.
That KataGo's numbers are the truth about a position; they are a much stronger player's estimate.