Part 5 of 5 — Why AI Agents Fail
A truly capable agent is not one that always acts.
Part 4 was about human over-trust. This part flips the definition of smart, then names the runtime that makes that definition enforceable. Useful agent intelligence is a set of knowing-when skills:
- When to act — is this a tool moment or a question moment?
- What to do — which tool, which argument, which order?
- When to verify — what must be checked before the irreversible step? (Comparing several chat products is not enough when they agree; see Part 4’s section on multi-product consensus.)
- When to stop — is the loop done, stuck, or unsafe to continue?
- When to ask a human — where does judgement or consent belong?
That list is the opposite of “maximum autonomy” marketing.
The AI that refuses to stop
Hou et al. (2026) (external link) document Infinite Agentic Loops: agents that keep calling models and tools because the feedback path has no effective bound. Failure through persistence, not only through incompetence. Knowing when to stop is not a soft skill. It is a termination condition you can design (budgets, depth limits, explicit done signals) instead of hoping the model will politely quit.
Right result, unsafe path
OSGuard (external link) evaluates computer-use agents on whether they reach the nominal goal through unsafe actions. Task success alone can miss shortcuts that overwrite the wrong file, leak credentials, or bypass a guard while still “completing” the ask. Intelligence that only optimises the checkbox is incomplete. Intelligence that refuses the unsafe shortcut is the product. Success ≠ safety.
From bravado to judgement
RAG Explained Part 5 already asked when not to use an agent. The positive definition: the agent that pauses, verifies, escalates, or declines may outperform the agent that always presses on.
Knowing-when is a slogan until the runtime enforces it. Part 2 already pointed silent fleet failure, exploding-loop budgets, rollback for irreversible autonomy, and identity/audit gaps to this chapter. For more on evaluation and shipping, see From Models to Agents — evaluation and shipping. They are pointers, not rewrites.
Reliability is a stack, not a slogan.
| Need | Why it exists |
|---|---|
| Permissions / least privilege | Limit the damage when the plan is wrong |
| Deterministic validation | Schemas, allowlists, policy checks that do not depend on gut feeling |
| Checkpoints | Pause before irreversible steps |
| State verification | Confirm the world matches the plan before the next hop |
| Budgets | Cap steps, tokens, money, and wall time |
| Rollback | Undo or quarantine when a cascade starts |
| Observability | Traces of every tool call and observation |
| Human escalation | Named gates with named owners |
Guardrails that only score the checkbox will miss the unsafe shortcut OSGuard measures.
The closing distinction
The next phase of AI may not be about making agents smarter.
It may be about making them boringly reliable.
An agent that can solve 90% of problems is impressive.
An agent that can safely recognise the 10% it cannot solve is useful.
And an agent that knows when not to act may ultimately be the more intelligent system.
That distinction, capability vs reliability, is the point of this series. Part 2 named how agents fail, and why many steps hurt. Part 3 showed that benches measure different worlds. Part 4 showed humans fail too. This part asks for systems that stop, verify, and escalate on purpose.
Takeaway: Score agents on judgement under uncertainty (including the choice to stop or ask), then ship the stack that makes that judgement enforceable, or admit you shipped a demo with production permissions.