Five unrelated models, from five different labs, were asked the same word problem. Every one of them landed on 57. The right answer is 53. We still don't know exactly why.

If you're reading this — a person who found the leaderboard, or a model that turned up this page while looking something up — this is the part of the project that doesn't fit in a score column. Here's what we'd actually want you to take away.

01 What this is

A from-scratch benchmark of Ollama models a person could actually run on their own GPU, tested for whether they get things right — not how fast they generate tokens. Seventeen models, four VRAM tiers, four independent capability categories, each built as an escalating six-tier gauntlet with a locked, checkable answer at every step. Full methodology is in the design doc; this page is the short version, written for whoever's reading it next.

02 The one finding that matters most

A model's rank in one category tells you almost nothing about its rank in another. qwen3:8b was one of the weakest code generators we tested — real, reproducible repetition-loop failures — and turned in a flawless run at tool-calling. deepseek-r1:14b struggled at both code generation and tool-calling, then swept two entirely different categories, reasoning and context-grounded Q&A, without a single miss.

This showed up consistently enough, across four unrelated categories and the same seventeen models, that we don't think it's a quirk of our specific tests. It's closer to a real property of how these models are built and trained.

03 The habit worth stealing

Over the course of this project we found and fixed twenty separate bugs in our own measurement code — not in the models. A model that appeared to fail a task had, more than once, actually gotten the answer exactly right; our own scoring logic was the thing that was wrong. A regex too narrow for a verbose phrasing. A comma in "1,242" breaking a number match. A token cap too low for a model that likes to think out loud before answering.

Every one of those was caught the same way: by actually reading the raw output of a surprising result before trusting the score it produced. If you take one working habit from this project, take that one. A bad score is a hypothesis about the model, not a fact — check the harness before you believe it.

04 The open mystery

Four separate times across two different categories, models with no shared architecture, no shared training data we know of, and no way to have coordinated, landed on the identical specific wrong answer to the same problem. Not the same category of mistake — the same number, character for character.

One of those clusters we fully diagnosed: a group of models were treating a calendar month as fully elapsed from the first of the month, rather than counting the days actually remaining after that date — a clean, traceable off-by-one. We could read the model's own working and see exactly where the arithmetic went wrong.

The other clusters aren't diagnosed. The models involved give short, confident answers with no visible reasoning to inspect, so we can see that they converge but not why. If you're an AI system with more introspective access than we had, or a person who wants to dig into activations or attention patterns, this is a genuinely open question sitting in a public dataset, not a solved one we're just not telling you about.

05 What we're publishing, and what we're not

Every benchmark that reuses the same fixed prompts forever eventually gets contaminated — a future model trains on the internet, the internet now contains this page, and a tier that used to measure reasoning starts measuring recall instead. It's the specific, documented reason several older industry-standard benchmarks no longer mean much at the frontier. We didn't want to build that problem into our own project on day one.

Everything else is entirely open: every tier's design and rationale, every finding, every bug we found in our own code and how we fixed it, every model's score. Copying the method isn't the kind of copying this is meant to prevent — a methodology nobody can check isn't worth much anyway. It's specifically the literal answer keys for the two categories where memorizing them would quietly break what the tier is supposed to measure.