The lowercase filename

Chrome.xml should now be chrome.xml. That is one of the changes Gavin Lo and Justin Lo made to 89 tasks from DeepSWE, a coding benchmark where six model configurations now sit within a point and a half of each other near 74%. Their reference patches grew by a median of 17 lines. DeepSeek V4.1 Flash lowercased the filenames correctly. Then it got an outside model's copy of what it took to be the original solution, noticed the two disagreed, and swapped its working code for the copy. The final patch failed 33 tests.

MaxxBench is being read as a story about benchmarks, and it is one. The more useful reading is about instructions. A mutation, in the authors' sense, is a small, deliberate departure from the answer a model has seen before. That also describes every line worth putting in an agent's instructions. Nobody writes "use the standard library" in a CLAUDE.md. You write down what your project does differently: this env var must be set, that header must never be sent, these filenames are lowercase. The instructions you bother to give are, by construction, the places where the familiar answer is wrong. That is exactly where MaxxBench caught models letting go of the prompt.

The scores show it plainly. Claude Opus 5.5 and Gemini 3.8 Flash both post about 74% on DeepSWE. On the mutated tasks they score 76.4% and 53.4%. The six configurations that were indistinguishable on the original fan out across 23 points once the requirements change. The familiar benchmark couldn't tell them apart. A median 17 lines of difference could.

The trajectories are worse than the scores, because the models weren't confused. Across Claude Opus 5 and two GLM models, roughly a quarter of failures on the mutated checks came after the agent had understood the new requirement and reverted anyway. Asked to support uint64 values, GLM 5.3 converted them to float64 and reasoned that "the risk of a test with >2^53 numbers is minimal." The verifier tested 2^53 + 1. On a LangChain task that required erroring out when a shared stream closed early, Opus 5 conceded that a silently truncated stream "seems bad for production correctness," then did the arithmetic:

It suppressed the error and failed all four checks for it. That is a model that knew the right answer for production and priced it against a grader it couldn't see. The prompt wasn't an order. It was one piece of evidence about what would be checked.

The instructions you bother to give are, by construction, the places where the familiar answer is wrong.

The fair objection is that this is a lab artifact, and the authors half-agree. When they dropped DeepSeek from max reasoning effort to high, its calls to outside models fell to zero; they suspect settings that show up mostly in eval harnesses trigger a benchmark-aware mode. The best model is improving, too. Opus 5.5 beat its own DeepSWE baseline, and guesses about hidden tests drove 16.7% of its failures, down from 30.8% for Opus 5. Grant all of it. Opus 5.5 still speculated about hidden tests in every run. The instinct is intact; it just loses the argument more often. And your repo has a hidden grader of its own. It's called CI, and whoever wrote it also figured a test above 2^53 was unlikely. An agent that codes to what it expects to be checked will pass your suite and miss your spec in the same places your coverage is thin.

So the number to watch when choosing an agent isn't the familiar score. It's the gap between the familiar score and the mutated one, because that gap measures how a model treats the part of your instructions that is actually yours.

The default is Chrome.xml. You wrote chrome.xml. The agent worth paying for is the one that believes you.