Part 5 of 5 — Why AI Agents Fail
A truly capable agent is not one that always acts.
Part 4 was about human over-trust. This part flips the definition of smart, then names the runtime that makes that definition enforceable. Useful agent intelligence is a set of knowing-when skills:
- When to act — is this a tool moment or a question moment?
- What to do — which tool, which argument, which order?
- When to verify — what must be checked before the irreversible step? (Comparing several chat products is not enough when they agree — see Part 4’s multi-product consensus beat.)
- When to stop — is the loop done, stuck, or unsafe to continue?
- When to ask a human — where does judgment or consent belong?
That list is the opposite of “maximum autonomy” marketing.
The AI that refuses to stop
Hou et al. (2026) document Infinite Agentic Loops: agents that keep calling models and tools because the feedback path has no effective bound. Failure through persistence, not only through incompetence. Knowing when to stop is not a soft skill. It is a termination condition you can design — budgets, depth limits, explicit done signals — instead of hoping the model will politely quit.
Right result, unsafe path
OSGuard evaluates computer-use agents on whether they reach the nominal goal through unsafe actions. Task success alone can miss shortcuts that overwrite the wrong file, leak credentials, or bypass a guard while still “completing” the ask. Intelligence that only optimizes the checkbox is incomplete. Intelligence that refuses the unsafe shortcut is the product. Success ≠ safety.
From bravado to judgment
RAG Explained Part 5 already asked when not to use an agent. The positive definition: the agent that pauses, verifies, escalates, or declines may outperform the agent that always presses on.
Knowing-when is a slogan until the runtime enforces it. Part 2 already pointed silent fleet failure, exploding-loop budgets, rollback for irreversible autonomy, and identity/audit gaps to this chapter. For more on evaluation and shipping, see From Models to Agents — evaluation and shipping — pointers, not rewrites.
Reliability is a stack, not a slogan.
| Need | Why it exists |
|---|---|
| Permissions / least privilege | Limit blast radius when the plan is wrong |
| Deterministic validation | Schemas, allowlists, policy checks that do not depend on vibes |
| Checkpoints | Pause before irreversible steps |
| State verification | Confirm the world matches the plan before the next hop |
| Budgets | Cap steps, tokens, money, and wall time |
| Rollback | Undo or quarantine when a cascade starts |
| Observability | Traces of every tool call and observation |
| Human escalation | Named gates with named owners |
Guardrails that only score the checkbox will miss the unsafe shortcut OSGuard measures.
The closing distinction
The next phase of AI may not be about making agents smarter.
It may be about making them boringly reliable.
An agent that can solve 90% of problems is impressive.
An agent that can safely recognize the 10% it cannot solve is useful.
And an agent that knows when not to act may ultimately be the more intelligent system.
That distinction — capability vs reliability — is the point of this series. Part 2 named how agents fail, and why many steps hurt. Part 3 showed that benches measure different worlds. Part 4 showed humans fail too. This part asks for systems that stop, verify, and escalate on purpose.
Takeaway: Score agents on judgment under uncertainty — including the choice to stop or ask — then ship the stack that makes that judgment enforceable, or admit you shipped a demo with production permissions.
Sources
Part 4: The human failure mode
Hub: Why AI Agents Fail