Paper Readings · · 1,165 words · 5 min read
EarlyEval explained: stopping agent benchmark runs early
A reading of arXiv:2609.02783, which trains classifiers to halt agent runs once the outcome is predictable, and what its Table III actually shows.
agent evaluation SWE-bench benchmarks cost
The paper is arXiv:2609.02783, "EarlyEval: Cheaper Agent Evaluation via Early Outcome Prediction", by Yuling Shi, Zhensu Sun, Junsen Dong, Chengcheng Wan, David Lo and Xiaodong Gu (Shanghai Jiao Tong University, Singapore Management University, East China Normal University), submitted 2 September 2026. Its premise is a cost problem anyone running agentic benchmarks recognises: one pass of one agent over SWE-bench Verified costs hundreds of dollars, and a development loop runs dozens of passes. The proposal is to stop each task's run early once its outcome is predictable. If you maintain an agent leaderboard or an internal agent eval, the relevant question is whether early stopping changes the numbers you publish, and the paper's Tables III and IV are where that is answered.
What the paper claims
The claim is that an agent's final outcome on a task is often evident from its intermediate behaviour well before the run ends, and that this can be exploited without distorting the benchmark. EarlyEval trains two LightGBM classifiers, one for success and one for failure, over features extracted from a partial trajectory, and halts the run the moment either classifier crosses a calibrated confidence threshold. The abstract's headline: across SWE-bench Verified, TerminalBench and Toolathlon, EarlyEval eliminates 13% to 26% of agent steps and up to 44.1% of input tokens and 29.4% of output tokens, at 89% to 97% prediction accuracy, while perturbing per-agent resolve rates by one to two percentage points on average.
The authors frame this as complementary to benchmark distillation (running fewer tasks): distillation cuts the number of tasks, early stopping cuts the cost inside each task.
What was measured
Table I motivates the work with the cost of one evaluation pass using the OpenHands agent, figures the authors say were retrieved in June 2026: SWE-bench Verified (500 tasks) at $715, $760 and $935 for the three models listed; SWE-bench Multimodal (517 tasks) at $2,270 for the most expensive of them. Those are the authors' retrievals from a public index, not measurements in this paper, so I treat them as context rather than results.
The data: more than 21,000 outcome-labelled trajectories from 16 agents on SWE-bench Verified (a single scaffold, mini-SWE-agent, with 16 base models; 7,805 trajectories), 37 scaffold-plus-model configurations on TerminalBench (89 tasks; 6,757 trajectories), and 22 agents on Toolathlon. Evaluation is leave-one-agent-out: the classifiers are trained on every agent except the one being judged. For TerminalBench there are two splits, "no same model in training" and "no same scaffold in training", which matter later.
Table II lists the feature families: behavioural (activity counts, last-step properties, event timing, working pattern, error and test status), textual (TF-IDF over the task prompt, action history and environment feedback), and reference-solution features, which need a gold patch and so exist only on SWE-bench Verified.
Table III is the core result, reported per benchmark across decision thresholds. At the operating point the authors recommend for SWE-bench Verified, a dual threshold of 0.95, the success predictor has 93.9% precision at 20.4% coverage and the failure predictor 96.7% precision at 14.4% coverage; together they remove 26.0% of steps, 32.7% of input tokens and 28.7% of output tokens, with a mean absolute change in Pass@1 of 1.1 points. Lower thresholds save more and cost more accuracy: at 0.75 the dual saving is 63.4% of steps but the Pass@1 perturbation is 4.1 points.
Toolathlon at threshold 0.90 is where the abstract's 44.1% and 29.4% token figures come from: the failure predictor alone (the success predictor has zero coverage there) removes 23.0% of steps, 44.1% of input tokens and 29.4% of output tokens, at 96.6% precision, with 0.9 points of Pass@1 perturbation.
Table IV checks whether rankings survive. On SWE-bench Verified, the Spearman correlation between the early-stopped leaderboard and the full-run leaderboard is 0.991 with 81% of the 16 agents keeping their exact rank; on Toolathlon it is 0.994 with about 70% unchanged; on TerminalBench it is 0.994 for the no-same-scaffold split and 0.959 for the no-same-model split, with 59% unchanged in the harder case.
What holds up / what I could check
I did not have the trajectory pool to retrain the classifiers, so this is a check of internal consistency and of what the tables support.
The additivity claim checks out from the table itself. The authors note that the dual step reduction is almost exactly the sum of the success-only and failure-only reductions, for instance 10.6% plus 15.4% equals 26.0% on SWE-bench Verified at 0.95, which implies the two predictors almost never fire on the same run. That is a property you can read straight off Table III, and it is a sensible sanity check on the halting rule.
The success predictor is much weaker than the failure predictor everywhere except SWE-bench Verified, which the authors state plainly: on TerminalBench and Toolathlon the rankings in Table IV are built from the failure predictor alone. Reading the abstract, you might assume both directions work generally. They do not; what generalises is "this run is going to fail".
The scaffold result is the finding I found most useful. Withholding the test agent's scaffold from training degrades the success predictor more than withholding its base model: peak precision drops from 82.7% to 69.0%, and the attainable step reduction at the recommended operating point from 25.4% to 17.7%. The behavioural features are keyed to the rhythm a scaffold imposes, so an unseen scaffold perturbs them. If you run a leaderboard of heterogeneous scaffolds, the no-same-scaffold split is the number to plan around, and it is the smaller one.
Table VI compares the LightGBM backbone with alternatives at the same threshold on SWE-bench Verified: an MLP reaches 87.9% accuracy with a 3.3-point Pass@1 perturbation, a LoRA-tuned local LLM judge reaches 90.7% with 0.8 points but only 17.9% step savings, and LightGBM 95.0% with 1.1 points and 26.0% savings. The gradient-boosted model is not a stylistic choice; the alternatives are worse on the accuracy-savings trade-off in the paper's own numbers.
Two limits. The per-agent perturbation of "one to two points on average" is an average; Table IV lists individual agents moving by as much as 4.6 points of Pass@1 (one OpenHands configuration on the no-same-model TerminalBench split), which is larger than many of the gaps that separate adjacent leaderboard positions. And the whole approach depends on having a large labelled trajectory pool from many agents on the same benchmark; a team evaluating its own single agent on a private task set does not have the 21,000 trajectories that make the classifiers work.
What to do with it
For a public leaderboard with many agents, the paper supports early stopping at a conservative threshold (0.95 on SWE-bench Verified) with the failure predictor as the workhorse, and the Section VI threats-to-validity discussion is candid about the leakage controls needed. For an internal eval of one agent, the transferable idea is smaller and still worth having: log the behavioural features in Table II, because the same signals that predict failure for the classifier are the ones a human wants in a trajectory dashboard.