Part 3 of 5 — Why AI Agents Fail
One leaderboard is not a reliability certificate.
Benchmarks are useful. They are also easy to misuse. Teams read a high score as “ready for production.” Different suites measure different jobs — and almost all of the famous ones are still English-first.
SWE-bench: can it fix real software issues?
SWE-bench (and the SWE-bench paper) evaluates agents on real GitHub issues: edit the repo, make tests pass. That is a serious coding test. It is not a stand-in for “will this agent run our support desk for a week without cascading.”
OSWorld 2.0: can it finish the long workflow?
OSWorld 2.0 targets long-horizon computer use — median human effort on the order of 1.6 hours, with problems like hidden state, changing environments, and skipped verification (arXiv:2606.29537). On the authors’ 500-step run, Claude Opus 4.8 still finishes only 20.6% of those jobs (54.8% partial credit). The hook from Part 1 returns: reasoning for hours in a demo is not the same as finishing a two-hour stateful workflow.
METR: evaluate what you claim about capability
METR (Model Evaluation & Threat Research) publishes evaluation work aimed at understanding what advanced systems can and cannot do — including risk-relevant capability claims. On their GPT-5 evaluation, the 50% time horizon was about two hours 17 minutes on agentic software tasks — and about 25 minutes if you ask for 80% reliability. Use that as a reminder that “capability eval” is a discipline, not a marketing screenshot.
ReliabilityBench: stress, not a single happy run
ReliabilityBench argues that single-run success rates miss production needs. It stresses consistency under repeated runs, robustness to task perturbations, and fault tolerance under tool/API failures. That is closer to “boring reliability” than a one-shot demo score.
OmnilingualGAIA2: English is becoming the benchmark language
OmnilingualGAIA2 expands an agentic benchmark across ten languages. The authors report a universal cross-lingual gap of 8.8–18.4 pass@3 points that concentrates on tool orchestration rather than quantitative reasoning and does not simply vanish with scale. An agent that looks reliable on an English bench may not be equally reliable for a global audience.
Sample scores (snapshot)
These numbers are as of the cited pages, not a live leaderboard and not a product ranking. Harness, agent setup, and split all change the score. The lesson is the gap between boards.
| Bench | Named models (sample) | What the number is |
|---|---|---|
| OSWorld-Verified (short tasks) | Claude Opus 4.8 83.5% binary (OSWorld 2.0 paper) | Short desktop jobs — the paper’s own “looks solved” contrast |
| OSWorld 2.0 (long jobs) | Claude Opus 4.8 20.6% finish / 54.8% partial; GPT-5.5 13.0% finish (project page · paper Table 3) | Same family of agents, 500-step budget, long workflows |
| ReliabilityBench | Gemini 2.0 Flash 91.04% pass@2; GPT-4o 90.42%. Gemini 96.88% at no paraphrase stress → 88.12% at ε=0.2 (arXiv:2601.06112) | Repeat the job under stress, not one lucky run |
| METR time horizon | GPT-5 about 2 hours 17 minutes at 50% success; about 25 minutes at 80% (METR GPT-5 eval) | Human-length software tasks the agent is predicted to finish that often |
| OmnilingualGAIA2 | 8.8–18.4 pass@3 gap across languages (arXiv:2608.08775) | English competence is not global competence |
A model can look strong on a short computer-use board and still finish about one in five of OSWorld 2.0’s long jobs. That is the illusion: one score is not a production certificate.
Official SWE-bench named-model percentages are not copied here. The public board is interactive, and third-party aggregators disagree — so this snapshot does not invent a coding trophy.
How to read the board
| Kind of job | Rough question |
|---|---|
| SWE-bench | Can it patch real software issues? |
| OSWorld 2.0 | Can it finish long computer-use workflows? |
| METR-style eval | What capabilities and risks are we actually measuring? |
| ReliabilityBench | Does it stay reliable under stress and faults? |
| OmnilingualGAIA2 | Does English competence transfer across languages? |
Takeaway: Treat benchmarks as different instruments — not interchangeable trophies. Ask what kind of risk your production work actually has.
Sources
- SWE-bench · paper (arXiv:2310.06770)
- OSWorld 2.0 · arXiv:2606.29537
- METR · GPT-5 evaluation (50% horizon ~2h 17m)
- ReliabilityBench (arXiv:2601.06112)
- OmnilingualGAIA2 (arXiv:2608.08775)
Part 2: The seven ways agents fail
Part 4: The human failure mode
Hub: Why AI Agents Fail