LLM Debate Benchmark ranks Sonnet 4.6 first across 1,162 side-swapped debates

This eval tests debate, not just answer generation. According to the benchmark thread, it targets “adversarial, multi-turn debates,” where models must combine factual recall, rebuttal quality, and consistency across several rounds. That is closer to how judge models, debate agents, and structured planning systems fail in production than a static single-turn benchmark.

The task mix is broad enough to stress generality rather than a narrow policy niche. Mazur’s coverage note says the set spans 683 curated motions, from shrinkflation labeling to eurozone fallout and dating-app market structure. The released examples in quotable lines show why this is interesting for practitioners: models are being judged on whether they can produce concise, defensible claims such as “You cannot reject a trap you cannot see” or “It is exclusion protected by aesthetics,” not just retrieve facts.

The design has a few strong controls. Mazur’s format details says each debate runs 10 turns with openings, two rebuttals, a pressure-question exchange, and closings. Rankings are computed with Bradley-Terry over side-swapped matchups, which helps control for topic asymmetry. Completed debates are judged by three LLM judges drawn from six models, and the judging setup says same-family judging against the debaters is excluded.

The leaderboard still needs to be read with caveats. The coverage note includes content-block rates, which can change effective participation and comparability across motions. The sample transcript in sample debate also shows how much the results depend on prompt format and interaction structure: a Sonnet 4.6 adaptive vs. GPT-5.4 high match turns on whether consent creates a meaningful “paper trail” or just “consent theater,” and even the judge notes split across winners. That makes judge composition and transcript inspection part of the eval, not an afterthought.

The project is unusually inspectable. The long-form posts in full thread mirror and release thread point to charts, transcripts, model profiles, reports, judgments, and the GitHub repo; repo link post separately surfaces the code link. For teams building debate, arbitration, or self-critique systems, that makes this more useful as a reproducible eval pattern than as a simple winner board.