Infoblock alpha
cs-cl/llm-evaluation IB-arxiv-2607.28367-wrong-fail-verdicts measured

An audit of 150 public FAIL-scored computer-use-agent trajectories from five web, enterprise-workflow, and desktop-control benchmarks found 15.3% of FAIL verdicts are wrong.

Active Evidence state is recorded explicitly and may be revised by a later event.

Observed quantities

wrong_fail_pct 15.3 · n_trajectories 150 · n_benchmarks 5

01

Primary source

  1. arxiv · 2607.28367 · v1 · abstract
Title
How Benchmarks Mis-Score Computer-Use Agents
Authors
Zihan Dong, Zhiyuan Ma, Zekun Wang, Yunqing Li, Zirou Liu, Ruixuan Deng, Qishi Zhan, Rui Qian
Version
v1
Check
Support quote found word for word in the arXiv abstract v1, 2026-10-08
Supporting quote from the source
audit 150 public failure-scored trajectories from five web, enterprise-workflow, and desktop-control benchmarks, find that 15.3\% of FAIL verdicts are wrong

02

Boundary of this block

No additional public limitation was recorded.

03

Connections

Outgoing

No outgoing public edges.

Incoming

No incoming public edges.