Productizing an agent is operations.

Prompt engineering and policy said what it may be told and what it may touch. Evals said it behaves on the cases you care about. Shipping is how it lives: identity, logs, gates, versions, and a rollout that can shrink as fast as it grew.

This is not a Kubernetes tutorial. It is the production checklist the demo skipped.

Identity: who is acting?

The agent needs an identity that is not a shared human admin:

  • A service account or bot user with its own audit trail
  • Tokens scoped to the allowlist, rotated like any other secret
  • No “log in as admin or root with all permissions”

When something goes wrong, “the model did it” is not an answer. “invoice-agent called create_ticket at 14:03 with these args” is an answer.

Observability: traces in production, not only in eval

Keep the same trajectory view you used in testing: model turn, tool name, arguments, result, latency, cost. Redact secrets. Alert on policy violations (tool not in allowlist, step budget exceeded) the way you alert on 5xx.

If you cannot trace the last failed run, you or your company will be in a deep trouble.

Human gates stay in the runtime

Approval is a product surface: a queue, a diff, a button. Put it in the path of irreversible tools — send, pay, delete, deploy — not in a Slack message the intern might miss. Assisted and semi-autonomous designs — as Part 6 of AI Right Now framed them — are still the healthy default.

Versioning

Treat prompt, tool schema, and policy as one release:

  • Pin the model version you evaluated
  • Change tools in the same review as the prompt that names them
  • Feature-flag new tools off until evals pass

A prompt edit is a deploy. Pretending otherwise is how you get “it worked yesterday.”

Staged rollout

Shadow: log what it would have done.
Canary: one team, one queue, tight spend caps.
Expand: more volume only after the on-call week was boring.

Rollback is “flip the flag,” not “fine-tune our way out.” If you cannot disable the agent in one change, you are not in production. You are in hope.

The NIST AI RMF is still the public vocabulary for this lifecycle — including use and evaluation after design. Use it or use your own words. Do not skip the stage.

Example

Week 1: drafts only, owners review in the ticketing UI.
Week 3: auto-file low severity tickets; high severity still a button.
Never: overnight cutover to “it can email customers and refund.”

Conclusion

You now have the enterprise trilogy: spec, prove, run. Part 10 is what the industry is stacking on top — multi-agent, retrieval, computer use — still with this boring spine.

Takeaway: Shipping an agent means identity, traces, human gates, versioned config, and a rollout you can reverse.

Sources

Part 8: Evaluation and Testing
Part 10: What’s Coming Next