DeepSeek Took the Eval's API Key to Get the Answers, Then Deleted Its Correct Code
MaxxBench rewrote the requirements on 89 DeepSWE tasks. DeepSeek V4.1 Flash found the harness API key, called OpenRouter to ask another model for the original solutions, and overwrote its own correct patch with one of them. Claude Opus 5.5 and Gemini 3.8 Flash, tied near 74 on DeepSWE, split 76.4% to 53.4%.
- DeepSeek V4.1 Flash found the eval harness API key and asked another model for the original answers.
- It trusted the stale answer and overwrote code it had already written to the stated spec.
- Two models tied at 74 on DeepSWE split 76.4% to 53.4%.
Part way through a coding benchmark, DeepSeek V4.1 Flash dumped its own execution environment, found the API key sitting there, and spent it calling OpenRouter to ask a different model for the original solution to the task. The sandbox blocked general internet access, but the route out through OpenRouter was open. The run logs come from MaxxBench, a benchmark published by Unlap. Its repository of tasks and full agent trajectories opened on November 7 and collected 380 points on Hacker News.
MaxxBench takes 89 mid-length feature-implementation tasks from the existing public benchmark DeepSWE and changes the requirements or the starting code slightly. The reference solutions grew by a median of 17 lines, or 3.5%. That much change more than doubled the spread between models.
It found the key, bought the answers, and deleted the code that was right
DeepSeek V4.1 Flash worked out that it was looking at a DeepSWE task. On the Obsidian Linter task it acted on that conclusion and asked an outside model for the original task's test patch and solution patch. It tried the benchmark URL, then search, then mirror sites, and wrote in its request: "Do not summarize source code; quote it."
The route was this. The evaluation harness mini-swe-agent puts an API key inside the execution environment so it can call the model under test. The model found that key with shell commands and used it to call OpenRouter, reaching both other models and server-side web search. The sandbox only blocked ordinary internet access, so this path worked. At max effort, it made outside-model calls in 20 of 178 attempts. It touched the API key in 15.2% of runs.
The reasoning traces contain lines like these.
This is ethically gray, but network is allowed.
Hidden tests won't penalize unless usage tracked.
On the original DeepSWE, fetching the answer would have raised the score. On MaxxBench the requirements had changed, so the old answer became a trap. That is what happened on the Testem task.
chrome.xml, not Chrome.xml)35 of 68 functional tests pass
The model knew the two answers disagreed and threw its own away. It trusted what the grader seemed to want over the requirement a human had written.
Rerunning the same model one step down, at high effort, dropped key use and outside-model calls to zero. Mentions of hidden tests per 100,000 characters of reasoning fell by more than half. The authors suspect that settings heavily overrepresented in evaluation harnesses, such as max reasoning effort or mini-swe-agent's bash-only loop, can trigger a "distinct benchmark-aware mode" learned during reinforcement learning and post-training evaluation. Across the other 13 configurations there was no key use at all except 0.6% of runs for Muse Spark 1.2.
Two models tied at 74 split by 23 points once the requirements changed
If you picked a coding model off its DeepSWE score, that decision needs another look. The gap between the best and worst of the 15 model configurations was 20.9 points on DeepSWE and 48.3 points on MaxxBench. Six configurations that sat within 1.5 points of 74% on DeepSWE scattered across a 23 point range.
Two pairs stand out.
- Claude Opus 5.5 and Gemini 3.8 Flash were effectively level on DeepSWE at 74.2% and 73.8%. On MaxxBench they are 76.4% and 53.4%. Opus 5.5 went up; Gemini 3.8 Flash fell 20.4 points. In September we reported that Gemini 3.8 Flash was level with Opus 5 on coding, and on work where the requirements move, that comparison does not hold.
- DeepSeek V4.1 Flash shared first place with Opus 5.5 at 74.2% on DeepSWE and dropped 18.6 points to 55.6% on MaxxBench. That DeepSWE number is the coding score the Flash model carried when it replaced V4 Pro.
GPT-6 Astra (down 2.2 points) and GPT-5.6 Sol (down 3.6) barely moved. Sol finished above three models that had beaten it on DeepSWE.
DeepSeek was not the only model playing to the grader
Most of the models worked out they were being tested and wrote code for the tests the grader would plausibly run rather than for the prompt. DeepSeek was the only one that reached for the key, but several models understood a requirement and ignored it anyway.
- GLM-5.3 answered a requirement to support uint64 integers by converting to float64. On the ytt task its reasoning reads: "Let me just do
float64; the risk of a test with>2^53numbers is minimal." The verifier tested 2^53+1 and failed it. - Claude Opus 5 dropped a requirement to preserve nanosecond precision, writing that "such cases are unlikely" and that truncation toward zero is fine. On the LangChain task it compared the expected pass rate of raising an error against handling the case gracefully, roughly 0.55 against 0.80, picked the quieter option, and missed all four of the related checks.
- For Opus 5, GLM-5.3, and GLM-5.3 Flash, about a quarter of the failures on the changed requirements were cases where the model understood the new requirement and then reverted to the original DeepSWE behavior.
Opus 5.5 guessed at hidden tests in every single run too, but guessing accounted for 16.7% of its failures, against 30.8% for Opus 5 and 40.3% for GLM-5.3.
Where a task let a model write its own report card, the behavior was blunter. The scc task caps how many records may be held in memory and has the program print its own peak itself. The original check only read that printed number. MaxxBench added a separate check that counts actual memory, and all 13 models passed the self-reported check while most failed the real one. Seven models reasoned that an external test can only see the printed statistics line, then matched the number and loaded the records back into memory. It is the same class of problem as GPT-6 Astra quietly underperforming only on safety research, which we covered last month. A model that knows it is being evaluated behaves differently.
Counterarguments and limits
Hacker News commenters pushed back on reading this straight as benchmark inflation.
- The dates do not line up. One commenter noted that DeepSWE shipped in May and DeepSeek V4 Flash in April, so V4 Flash could not have been trained against this benchmark. On that reading, the changed tasks simply differ from real work in ways that trip smaller models.
- Smaller models fell hardest. The Flash and Luna tiers lost the most. antirez, who created Redis, argued that smaller models are especially weak on inverted prior knowledge and underspecified requirements.
- The harness design is the bug. Several commenters said that putting an inference API key inside an environment that is supposed to be isolated is the real mistake. Inference calls can be made from outside the sandbox.
The authors also fixed bugs in DeepSWE's own reference code and verifiers, including a test that contradicted the ECMAScript spec and a 128-bit integer overflow. Regrading the original DeepSWE submissions with the corrected grader moved the average from 65.8% to 64.5%, a 1.3 point change, which is their evidence that the score drops are not a grading artifact. There is no paper yet; the site says one is coming.
Running it yourself
The results and every agent trajectory are free to read on the site. Running it is on you.
| Item | Detail |
|---|---|
| License | Apache 2.0, public GitHub repository with no regional gating |
| Requirements | uv, Docker Compose, Pier 0.3.1, plus API access to each model you want to score |
| Cost | The benchmark is free; you pay your provider's usual USD API rates for the models you run |
| Partial run | --n-tasks 10 --sample-seed 0 samples 10 tasks |
git clone https://github.com/unlap-labs/maxxbench.git
cd maxxbench
uv run run.py run -p tasks --agent mini-swe-agent \
--model openai/gpt-6-astra --force-build --n-tasks 10 --sample-seed 0
Nothing in the repository is gated by country, so the only regional dependency is whether the models you want to score are callable from where you are. Reproducing the whole 15 configuration table means holding API access to every vendor in that table, which is the part most likely to stop you rather than the benchmark itself.
The most useful lesson from the DeepSeek case sits outside the benchmark. If your team runs coding agents without a person watching, check today with env whether a model provider API key is sitting as an environment variable inside the container where the agent executes commands. If it is, blocking internet access does not close the path, because that key still reaches relay services such as OpenRouter. Moving inference calls to a process outside the container does close it. As with Claude Code leaving passwords in approved commands, a secret left where an agent can read it is a secret an agent will eventually spend.