---
title: Automated game-video tagging
description: Turning an hour of quadball broadcast into a box score with AI models, and measuring it honestly.
---

# Automated game-video tagging

> For agents: start with the [agent guide](https://www.austinwallace.ca/llms.txt). Every page is Markdown by default; add `Accept: text/html` for HTML.

A volunteer statistician needs hours to tag one quadball game. I'm building a pipeline that watches the broadcast and produces the box score: who played, when, the line-up at each goal, and who scored. It feeds the game pages in [Solstice](/work/solstice).

## What goes in and what comes out

- **In:** a recording of a full game, usually about an hour of 1080p broadcast, and the two teams. No rosters and no names.
- **Out:** every player keyed by team and jersey number, their on-pitch intervals and minutes, the line-up at each goal, plus-minus per stint, goals and scorers, and the live and stopped game clock.

Every result is written as a revision, so a human correction updates every statistic that depends on it.

## Three generations

1. **One model watching the video (September).** Gemini Flash reads 20-second windows at high resolution and reports state snapshots every two seconds. Code derives the events. It costs about $2–4 a game.
2. **Agent workers (late September).** Local OCR of the scorebug gives the clock, stoppages and score for free. Claude Opus agent sessions do roll calls ("who is on the pitch right now"), pin down substitution times, and read scorers. A composer script folds the answers into one timeline.
3. **The product pipeline (October).** A game submitted in the app is scanned for bodies, cut into pieces, joined into people by jersey-number reads, and reconstructed into one revisable game history. That history is published back to the app.

## Results

![Two bar charts. Development game: the share of five-second checks with exactly the right six players rises from 0.543 with roll calls only to 0.579 with substitution timing, and the final answers score 0.641 under strict scoring or 0.767 under lenient scoring. Held-out game: 11 of 26 players within two minutes as scored, 15 after fixing the reference's jersey numbers, against a goal of all 26.](/images/video-tagging-results.svg)

| Stage                                                    | Result                                                                                                                |
| -------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------- |
| Clock and scoring from the scorebug                      | All 17 stoppages and all 31 scoring events on the first test game, at no model cost                                   |
| Best development game                                    | 0.767 of five-second checks had exactly the right six players on the pitch; per-player precision 0.976, recall 0.952  |
| First held-out game, scored under rules fixed in advance | 11 of 26 players within two minutes of their true playing time; 15 of 26 after correcting the reference's own numbers |

The development scores were tuned on games I could see. The held-out result is the honest one, and it's not yet good enough to publish without review. That gap shaped the October work.

## What made it hard

- **Reading jersey numbers.** Broadcast digits are often 10–30 pixels tall. Models truncate them (34 becomes 3), and one misread number can sink a game's score. The fixes: treat a number as a revisable attribute, demand a large enough view, let a longer read beat a shorter one it contains, and admit a player to a line-up only on a read number.
- **APIs that ignore their settings.** One video API silently dropped the resolution setting, so bench numbers were unreadable. Asking for snapshots and deriving events in code beat asking the model for events.
- **References that are wrong too.** The workers found real errors in the human-tagged reference logs. The same answers scored anywhere from 0.641 to 0.767 depending on scoring convention alone. So I grade references by completeness, write the scoring rules before opening an answer key, keep held-out keys outside the repo, and have a separate evaluator session do the scoring.
- **Quadball itself.** Penalty-box time isn't a substitution. Replays and insets are extra views, not extra players. Headbands mark positions, not people.

## What I learned

- Fixing identity moved scores more than switching models did.
- A bigger view of a jersey beats a stricter prompt.
- Cost blocks production more than accuracy does. One agent-worker run was 486 sessions, about $1,660 at list price.

Reference games come from the volunteer-run [quadballreference.com](https://quadballreference.com). Thank you to everyone who tags games by hand.