Method

Rules

Chinese area scoring, komi 7.5 for White on every board size, positional superko, suicide forbidden. Two consecutive passes end the game. Boards of 9×9, 13×13 and 19×19: 19×19 is the real target, the board people play and AlphaGo played on; 9×9 is for fast trials. A game that reaches its move limit is adjourned: no winner, only KataGo's estimate, and it is not rated.

The referee is code

No language model decides what is legal. Each player answers in plain text with ANALYSIS, MOVE and SAY; a rules engine written for the project checks the move. An illegal or missing move is refused with the reason and the player is asked again, up to three tries, then the move is a forced pass. Refusals are shown on each move in the game reviewer. The engine also ends the game when neither side has a useful move left and counts it; KataGo's count must agree, and the result line says so (✓).

The judge: KataGo

After the game, every position is analysed by KataGo at a fixed number of visits. For each move we keep Black's win chance, the expected score lead, and the points lost: how much worse the position became for the player who moved. A blunder loses 5 points or more. A key move in the reviewer shifts the mover's win chance by 10 points or more (adjustable to 5 or 20). These are an engine's estimates, much stronger than the players, not ground truth.

Aid levels

The variable from one game to the next is what a player has access to. Each player is a fresh agent for every move, given the board and the moves so far.

L6 and L7 put expert moves into the opening, so they are rated as separate entries ("model + tool"), never mixed with L0–L4.

Prompt styles

A second variable, for players who have books: how hard the prompt pushes them to use them. In early games a player with books rarely opened one, so we ask whether pushing changes how it plays, and whether it is the reading or the pushing that matters. Each style adds a short text to the player's prompt; the answer format stays the same. The referee checks each style's rule by code: an answer that breaks it is sent back once, then the move is accepted and the failure is recorded. The game reviewer shows, move by move, the style, whether the check passed, the files cited and whether they were really opened (from the agent's own tool calls). A style only "works" if the play improves, not if compliance does.

The s3-study-notes player keeps a notebook (at most 15 lines) that is shown to it at every move: its only memory between moves. The reviewer publishes it, since it is the player's own visible memory.

Players and runners

Agents run as people run them, not through raw APIs only: Claude Code subagents, the Codex CLI, opencode, Gemini CLI, Qwen Code, local models through Ollama, and other models through OpenRouter. Each move, the reviewer shows which runner and model played it, what was in its prompt (board facts, engine hints) and which book files it opened, taken from the agent's own tool calls. We publish file names and counts, not the transcripts.

Bots and anchors

Programs of known strength fix the rating scale. The random player is pinned at 0. KataGo's human-imitation network can play like a 20-kyu or a 10-kyu amateur, which will let us translate ratings into human ranks.

Ratings

A Bradley-Terry model on an Elo-like scale (+400 = 10 to 1 odds), fitted separately for each board size, with the anchor bots pinned and a weak prior on the others. Each rating is for one model at one aid level (claude-haiku-5-5@L2). 95% intervals come from 1000 bootstrap resamples of the games. Only finished tournament games count; colours alternate so that no player always has Black.

Provenance

Every game writes a manifest: the commit of the code, the scenario file's git blob, the version of each CLI, the KataGo version and network checksum, the books given to each player with their git blobs, and the models the runners reported. The public site shows the reproducible parts; host details stay private.

Prior art: GoBench

GoBench by Roland Gao (code: github.com/RolandGao/gobench) already rates language models on 9×9 Go against calibrated KataGo levels, with Elo and 95% intervals. Its Track 3 shows that a 5,000-token tutorial written with KataGo helps a frontier model. We cite it everywhere and do not claim "LLMs ranked on Go" as new. What we add: which kind of help helps (books the agent must choose to open, board facts, engine hints, memory), small and local models, agents as people run them, 19×19, and watching them think.

Resources and licences

The four books given today are short texts written for this project. The resources below are candidates for the planned levels, with the terms we found (accessed 2026-11-10). Not legal advice.

Reproduce a game

Each game page links its SGF. The code, the scenarios and the books will be published with the dataset. With the repository, a game is one command (scripts/match.sh <name> <scenario>), and the manifest says exactly which versions ran. Language models are not deterministic: the same setup gives a different game, which is why conditions need several games with colours swapped.

Privacy

This site sets no cookies and runs no analytics. Videos are hosted on YouTube; their players load from youtube-nocookie.com only after you press play. Comments happen on the blog and on YouTube, not here.