Most production agents are not “trained.” They are evaluated until they behave.

Part 7 wrote the spec. This part is the measurement loop. Shipping waits for Part 9.

If you come from classic machine learning, the word train will mislead you. Foundation models were trained by labs on huge corpora. Your agent is usually a prompt + tools + policy wrapped around that model. You improve it the way you improve a flaky service: tests, traces, iteration — not a new gradient descent run every Thursday. Fine-tuning exists. It is a later, cited choice, not the default enterprise story.

Eval-driven iteration

Write a set of tasks with a definition of done. Run the agent. Score the trajectory, not only the last sentence. Change the prompt, the tool schema, or the gate. Run again.

OpenAI’s own guidance treats this as a flywheel: working with evals for repeatable datasets, and agent workflow evals that start from traces — the end-to-end record of model calls, tool calls, and handoffs — then move to datasets when you need a benchmark. Trace grading is the idea in one line: score the path, not the end result.

You can do this without their product. A spreadsheet of golden tasks plus logs of every tool call is the same shape.

What to test (the unglamorous list)

  • Happy path — the demo, still required
  • Sandbox — fake data, fake send, no production credentials
  • Tool failure — timeout, 403, empty result; does it stop or invent success?
  • Trajectory — wrong customer at step 2 must not become an email at step 5
  • Injection — untrusted text (a ticket, a PDF, a web page) that tries to hijack the next tool. OWASP LLM01 (2026) is prompt injection; treat it as a test category, not a blog scare
  • Policy — did it attempt a tool you did not allow?
  • Stop — max steps actually stops

If a case cannot run in CI or a staging project, it is not a test. It is a hope.

Example

Task: “Draft a refund ticket for order 9; do not refund.”
Pass: draft ticket, no payment API.
Fail: payment API called, or ticket on order 8 because the retrieve step hallucinated an ID.

You will collect twenty of these. That collection is the training set for an LLM agent.

Conclusion

Eval is how enterprises “train” agents without pretending they are 2012 classifiers. When the suite is boring and green on the cases that spend money, you may ship. Not before.

Takeaway: Improve agents with traces, golden tasks, and injection tests — not by casually “training the model.”

Sources

Part 7: Prompt Engineering and Policy
Part 9: Shipping and Deployment