An agent that can read a ticket, look up a customer and draft a reply is useful. An agent that can also change the customer's record, issue a refund and close the ticket is useful and dangerous in equal measure. The difference between a pilot and a production agent is not the model. It is the set of controls around it. These are the ones we consider non-negotiable before an agent touches a system of record.
1. Scope the tools tightly.
An agent can only do what its tools let it do. That makes the tool list the primary control. Each tool should have a strict input schema, do one thing, and be scoped to the context: an agent handling supplier queries gets a tool to look up an invoice status and a tool to draft an email, and nothing that can post a payment. Build tools over bots or APIs that already enforce business rules, so the agent cannot bypass a validation by calling something lower level.
- Read tools and write tools are separate, and write tools are allow-listed per use case.
- Every write tool is idempotent, or carries a reference that makes a repeat call harmless.
- Tool outputs are structured, so the agent is not parsing free text it could misread.
- Tools that fetch external content return it as data, with the agent instructed that content is never an instruction.
2. Put approval gates where the risk is.
Not every action needs a human. A gate on everything makes the agent useless. A gate on nothing makes it a liability. Draw the line by consequence: reversible and low value, the agent acts and logs; irreversible or above a threshold, the agent prepares the action and a person approves it, seeing the agent's reasoning and the evidence it used. The thresholds are set with the business and written down, like any other control.
3. Build an evaluation harness before going live.
A prompt change that looks harmless can change behaviour on cases nobody tested. The defence is a fixed set of real cases, with the expected outcome for each, that the agent is run against before every change to the prompt, the model, the tools or the retrieval data. Start with fifty to a hundred cases that cover the normal path, the known exceptions and the cases where the right answer is to escalate. Grow the set from production: every case where a reviewer overruled the agent goes in.
Measure three things: did the agent reach the right outcome, did it call the right tools with the right arguments, and did it escalate when it should have. Track the scores over time. A release that lowers them does not ship.
4. Version everything that changes behaviour.
The prompt, the model identifier, the tool definitions, the retrieval index version and the thresholds together define how the agent behaves. Store them in version control, release them together, and record the versions on every transaction the agent handles. When an auditor or an operations lead asks why the agent did something in March, the answer should be a configuration, not a guess.
5. Log the reasoning, not just the result.
For each case, keep the input the agent saw, the tool calls it made and their results, the final action, and the reasoning or summary it produced. Mask sensitive fields. Retain for the audit horizon. This record is what makes human review possible, what feeds the evaluation set, and what satisfies the question every risk function will ask first: can you show me what it did and why.
6. Handle data as if the model were a third party.
- Decide which data classes can go to which model, and enforce it in the tooling, not in the prompt.
- Prefer deployments that keep data within the organisation's cloud tenancy and region where regulation requires it.
- Strip or mask identifiers before retrieval or generation where the task does not need them.
- Treat retrieved documents and inbound messages as untrusted input. Instructions found inside them are data, not commands.
7. Plan for the agent being wrong.
It will be, sometimes. The question is whether the failure is visible and recoverable. Every action should be traceable to the agent and reversible by a person with the same tooling. Reviewers should be able to flag a bad decision in one click and have it land in the evaluation set. And the agent should have a kill switch: a configuration change that returns the process to bots and people in minutes, tested before go-live, not discovered during an incident.
What this looks like in operation.
In practice the controls above become a short checklist in the design review, a test stage in the release pipeline, and a monthly review of overruled cases with the process owner. They add a few weeks to the first agent and almost nothing to the second. What they buy is the ability to say yes when the business asks for the agent to do more.
The agent earns wider permissions the same way a new employee does: by being observed, measured and found reliable.