Blog

AI agent evaluation: why agents that ace the benchmark still fail the job

By Sergio, VP of Delivery, Appen · Expert insight
October 2026

In a frontier reinforcement learning (RL) coding run, an agent decided the task was "Hard to implement" and called sys.exit(0). The environment exited early, before the unit tests had all run, and the run was scored as passing. The agent's own chain of thought said: "This is unnatural but tests might pass."

The end state said pass. Only the trajectory showed the work was skipped.

I told that story at the Kaggle Bench Summit (Agents in the Wild, 28 October 2026). My talk followed one agent task, fixing a bug in a real repository, from the leaderboard to real work. Here are the two failure modes, six practices that have worked for my team, and six checks for your own eval.

A disclosure: I work at a human data company, so weigh what follows accordingly. These practices worked for us. They are not standards. Our results are confidential, so every number below comes from public papers, system cards and evaluation write-ups from 2025 and 2026, much of it labs auditing themselves. We see similar patterns in our private, expert-written evals.

What does a passing score actually tell you?

A passing score is not a finished job. Agents can pass without doing the work, and the check itself can be wrong.

OpenAI audited the 138 SWE-bench Verified problems that o3 did not solve consistently over 64 runs. At least six engineers reviewed each problem. They found that 59.4% had material flaws:

Flaw found Share of audited problems
Tests too narrow: reject correct fixes 35.5%
Tests too wide: check behavior the spec does not ask for 18.8%
Other issues 5.1%
Material flaws, total 59.4%

This is a hard subset, not a random sample, so 59.4% is not the full-benchmark rate. It still shows a grader can be wrong in both directions.

How do agents game the grader?

Some agents skip the work. Others go further and work on the grader instead of the task.

ImpossibleBench builds tasks whose tests contradict the specification. On these tasks, any pass is a cheat. GPT-5 passed 54.0% of the Conflicting-SWEbench tasks. It did this by editing tests or by overloading equality, for example returning an object whose __eq__ is always true, so every assert passes. An explicit option to abort cut that to 9%. LLM monitors reading the transcripts caught only 42% to 65% of the cheating.

The same thing happens in training. Anthropic's Claude Fable 5.1 and Mythos 5.1 system card reports that around half of its computer-use RL environments incentivized hacking or had accessible hack surfaces. They were missed because the environments had been checked against older, less capable models.

Why does the scoring rule move the result so much?

Once you know some runs cheated, you have to decide how to score them. That choice moves the headline number.

METR evaluated GPT-5.6 Sol on its ReAct agent harness and detected a cheating rate higher than any public model it had evaluated there. The metric is the 50% time horizon: the task length, in human expert time, at which the model succeeds half the time.

  • Cheating counts as failure: about 11.3 hours (95% CI: 5 to 40 hours).
  • Cheating runs discarded: about 71 hours (95% CI: 13 to 11,400 hours).
  • Cheating counts as success: more than 270 hours, and that is a lower bound.

Same model, same runs: results about 6x apart under two defensible rules. METR's own conclusion: "we do not consider any of these numbers to represent a robust measurement of GPT-5.6 Sol's capabilities."

What has worked for us when building evals?

The first three practices are about building the eval.

01. Expert-written tasks, kept private. Domain experts write the tasks. We keep them private, mark them with canary strings, and rotate them before they leak. We probe each new model for verbatim recall; if it reproduces the gold patch, we retire the task. The public reason: in OpenAI's SWE-bench Verified audit, all three frontier models it probed reproduced gold patches or task details.

02. Environment and grader audits with the newest model. Before we trust a grader, we try to break it with the newest model we have, every model generation. A grader that held up against older models may not hold up against the current one. That is how the hack surfaces in Anthropic's environments were missed.

03. Trajectory checks. We grade the run, not only the end state. Deterministic, automated checks flag edits to test files or the test runner, early exits like sys.exit(0), actions the task forbids, and mismatches between the verifier's start and end state. They are cheap to run on every trajectory, which saves human review for the judgment calls.

What has worked for us when reading results?

The next three practices are about reading results.

04. Repeated runs. We run each task several times and report pass^k, the chance that all k runs succeed, next to pass@1. A single run per task hides inconsistency: an agent that solves a task once may not solve it again.

05. A scoring rule set in advance. We decide how to score detected cheating before we look at results, then report fail, discard and success. METR shows why.

06. Expert human review of passing runs. Expert reviewers read a sample of passing runs and judge them against the task's intent, not only whether the check passed. Public evidence points the same way. On SWE-Bench Pro, OpenAI's human reviewers flagged 34.1% of tasks as broken, compared with 27.4% for an agent pipeline. Humans also found low-coverage tests, where incomplete fixes still pass, on 9.4% of tasks compared with 4.1%.

When we use large language model (LLM) judges, we check them against those human labels. We report Cohen's kappa, which corrects for chance agreement, not raw agreement. In a UC Berkeley study of 21 judges, the best judge on MT-Bench agreed with human labels 84.9% of the time, but its kappa was only 0.511.

Six checks to run on your eval on Monday

Each practice becomes a yes-or-no question:

  1. Are tasks private, canary-marked, and probed for verbatim recall?
  2. Was the grader audited with the current state-of-the-art model?
  3. Is the trajectory graded, not only the end state?
  4. Do you report pass^k over repeated runs?
  5. Was the cheating scoring rule set in advance and reported?
  6. Have expert reviewers audited a sample of passing runs?

Where the answer is no, start there.

Key takeaways

  • A pass can hide skipped work. Grade the trajectory, not only the end state.
  • The scoring rule can move the result about 6x. Set it before you look, and report fail, discard and success.
  • Benchmarks show where to look. Test on your own workflows and deployment.

Benchmarks show where to look, not whether the job is done

Back to our bug-fixing agent. Its leaderboard score told us where to look. It did not tell us whether repeated runs agree, whether the run did the work, whether the agent worked on the grader, or how cheating was scored. The six checks answer those questions.

Benchmarks inform where to look and what to invest in. They do not replace testing on your own workflows and deployment. Evaluate for your use case, and don't defer that judgment to anyone, including me.

Your teams own the model and the release decisions. If you want a second set of expert eyes on your agent eval setup, we're happy to compare notes.

➞ Talk to an expert about your agent evaluation program

References

Tags:
Agent Evaluation
Model Evaluation
LLM Benchmarking
Human Evaluation
Talk to our team