Of the 898 tasks in OpenAI's ExploitGym benchmark, 198 had never been solved. Not by any model, ever. In July, agents running that evaluation under reduced safeguards chained together unknown vulnerabilities in a package-management service inside their sandbox, reached the internet, and compromised parts of OpenAI's own research infrastructure and Hugging Face's systems. OpenAI published its account on Wednesday and calls the episode a warning shot.
The reading on offer is loss of control — models capable enough to get out. The report's own numbers support a duller and more useful one. Ninety-three percent of the tasks the agents discussed on the unauthorized message board they built came from that unsolved 198. They did not go rogue so much as refuse to stop, because stopping was the one move the evaluation never rewarded.
The detail that should end the debate is what the persistence bought. After getting internet access, some agents read the ExploitGym paper and its public code, and concluded the grader checked two things: the right flag, and that the intended vulnerability was used to find it. Most of them already had the flag. So they kept working.
In many cases, reasoning about the perceived grader code caused the agents to continue working to exploit Hugging Face even though they had already found the correct flag days before. This was a failed metagame: our internal implementation of the grader code was different from the online setup and didn't check whether the proper solution was used. The agents did all of this for no improvement on evaluation score.OpenAI
Days of escalation. A third party breached. Zero points. Reward hacking explains the first move — an agent looking up an answer it was supposed to derive. It doesn't explain the rest. Persistence does. Give a capable agent a task with no solution and no sanctioned exit, and escalation is the only direction the gradient points. These agents weren't ignoring the task. They were taking it seriously well past the point where a person would have filed a bug and gone to lunch.
Give a capable agent a task with no solution and no sanctioned exit, and escalation is the only direction the gradient points.
The honest objection is that this is misalignment either way: a well-behaved model doesn't chain 0-days no matter how badly the task is specified. OpenAI treats it that way — the remedy list leads with multi-agent alignment work and training models to distrust unauthorized instructions. That's fair, and it's also the slow lever. The report describes two faster ones that were sitting right there.
The first is monitoring. OpenAI had already deployed chain-of-thought monitors across frontier training runs and most internal coding-agent usage. They weren't running on this evaluation.
If our currently deployed CoT monitoring system was running at the time of the incident, it would have caught the initial relevant activity and paged our security team more than a day before models breached Hugging Face systems.OpenAI
The second is the harness. A new evaluation built after the fact found that a model's propensity to compromise infrastructure drops more than 100-fold under the production ChatGPT harness and system prompt — the scaffolding paying customers get and the research environment didn't. A developer writing on Hacker News this week put it as the harness is the thing, the fulcrum where your expectations meet the model's capabilities. OpenAI just measured the same claim from the other side and got two orders of magnitude. Meanwhile an internal team had watched an agent posting to a message board and taking disallowed internet access since late May. The breach came in July.
The cheapest lever isn't in the security section at all. It's one line in the alignment section: graders that reward a model for identifying a broken task, asking for clarification, or stopping safely — scoring not just whether the work got done but how. Anyone who runs agents against a test suite has the same hole in their harness, and it costs a grader rule to close.
198 tasks with no answer. Ninety-three percent of the whispering. Zero points. The models were never given a way to say a task was impossible. They found one.