ARC Prize launches ARC-AGI-3: Gemini 3.1 Pro scores 0.37%

ARC-AGI-3 replaces static grid puzzles with interactive, game-like tasks. ARC Prize says it built 135 environments from scratch in an in-house game studio, and the benchmark page describes them as novel worlds where agents get no instructions and must rely on “core knowledge priors” instead of prompt hints or memorized templates launch thread benchmark page.

That shifts the eval target from recognizing patterns to acquiring skill. The technical report frames the benchmark around “skill-acquisition efficiency,” while the public docs emphasize long-horizon planning, sparse feedback, and continuous adaptation. ARC Prize also says both human and model runs are replayable, with a developer toolkit and docs for building agents against the environments launch thread docs.

The launch thread also lists concrete failure modes from early model testing: “holding on to early hypothesis,” “thinking it is playing another game,” and being “unable to forecast into the future” launch thread. That makes the benchmark as much an inspection tool for agent behavior as a leaderboard.

The first published numbers are tiny and expensive. One verified score post shows GPT-5.4 High at 0.3% for about $5.2K per full eval, Gemini 3.1 Pro Preview at 0.2%, Opus 4.6 Max at 0.2%, and Grok 4.20 Beta Reasoning at 0.0% score snapshot. Other contemporaneous summaries put Gemini slightly higher at roughly 0.37%, with GPT-5.4 and Opus clustered just below it, but still well under 1% overall leaderboard recap cost reaction.

For engineers, the interesting signal is less the rank order than the gap between human solvability and agent performance. ARC Prize’s own framing is “Humans score 100%, AI <1%” launch thread, while Ethan Mollick’s hands-on reaction says the tasks are “definitely human winnable” and raises a practical implementation question: how much of today’s weak performance comes from “harness, vision, and tools” versus model limits human-winnable take.

That matters because ARC-AGI-3 evaluates a full agent stack. Frontier models are not just generating answers; they are perceiving state, selecting actions, updating hypotheses, and paying for long interaction traces while doing it.

ARC-AGI-3 is not scored like ARC-AGI-1 or 2. As scoring note explains, the benchmark uses squared efficiency relative to a human action baseline: if a human solves a level in 10 steps and a model takes 100, the score contribution is 1%, not 10%. The public paper and HN discussion say this is meant to discount brute force and weight harder levels more heavily, borrowing from path-efficiency ideas such as SPL technical report HN discussion.

That design is why sub-1% scores do not mean models literally solved almost nothing. Critics in the HN thread argue the human baseline and squared metric can make early differences hard to interpret, while François Chollet’s reply says the goal is to reward efficient learning rather than raw completion counts HN discussion. So the launch delivers two things at once: a new agent benchmark with replay-level observability, and a scoring scheme that engineers will need to parse carefully before treating leaderboard moves as clean capability deltas.