How to build an AI agent that is safe to run unattended

Safe AI agents are not the ones with the best prompt. They are the ones where the prompt cannot cause harm even when it is wrong. An agent that runs without a person watching will, at some point, misread a document, pick the wrong tool or be talked into something by an input it should have ignored. The engineering question is what happens next. If the answer is "nothing much, because the action was outside its permissions and it escalated", the agent is safe. If the answer is "it issued a refund to the wrong account", it is not.

This article sets out the five layers we put around every agent we ship, in the order we build them, and what each costs in time.

Why AI agent guardrails belong outside the model

A prompt is an instruction, not a control. Models follow instructions most of the time, which is exactly the problem: "most of the time" is not a safety property. Instructions can be overridden by content the agent reads, whether a customer message, an email or a document, and a model under pressure to complete a task will sometimes rationalise its way past a rule. The OWASP Top 10 for LLM applications lists prompt injection and excessive agency as the leading risks for exactly this reason.

So the rule is simple: anything that must never happen is enforced in code, where the model cannot reach it. The prompt shapes behaviour; the guardrails bound it. The difference between a chatbot and an agent, and why agents need this at all, is covered in AI agent vs chatbot.

The five layers of a safe agent

Layer 1: AI agent permissions through scoped tools

The agent never gets a database connection, an admin API key or a shell. It gets tools: named functions with typed inputs, typed outputs and a fixed scope. "Look up order by ID for this customer" rather than "query orders". "Issue refund up to the order value on this order" rather than "call the payments API".

Each tool runs with its own credential, scoped to the least access it needs, and the tool layer, not the model, checks that the customer in the request matches the customer in the session. We increasingly build this layer on the Model Context Protocol so the same tools serve several agents; the pattern is explained in MCP explained. The practical test of a tool contract is whether a malicious prompt could use it to do damage. If it could, the contract is too wide.

Layer 2: spend, rate and step limits

Every agent loop has a step budget: the maximum number of tool calls before it must stop and escalate. Every task has a token and cost ceiling. Every tool has a rate limit and, for actions with a value, a per-action and per-period cap: no single refund above a threshold, no more than a set total per day, no more than a set number to one customer in a month. Limits are set by the business owner, stored as configuration, and changed through a reviewed process rather than a prompt edit.

Limits are what turn a catastrophic failure into a bounded one. An agent that gets confused and loops burns its step budget and stops; it does not burn a month's inference bill overnight.

Layer 3: human in the loop agents with approval gates

Some actions should never be taken alone: high-value transactions, anything irreversible, anything touching a vulnerable customer, anything the policy owner has flagged. For these, the agent prepares the action in full, with its reasoning and evidence, and a person approves or rejects in one click. The queue should be a first-class interface with context, not an email.

Gates are defined per intent and per threshold, and they move over time. An action starts fully gated, then gates only above a value, then only on anomalies, as the evidence from evals and from approval history accumulates. This staged release is the subject of Shadow mode: the right way to launch AI agents. The gate design for refunds specifically is in Policy-gated actions.

Layer 4: evaluation suites before every change

You cannot chat your way to confidence in an agent. You run it. The evaluation suite is a set of scenarios, each with an input, the expected tool calls, the expected outcome and the expected escalation decision. It includes adversarial cases: injected instructions in a document, a customer claiming to be someone else, a tool returning malformed data. It runs in the pipeline on every prompt, model or tool change, and a regression blocks release.

Suites start at fifty to a hundred scenarios drawn from real traffic and grow every week from escalation reviews. The metrics they produce are set out in How to measure an AI agent. Evals are also what make model-agnostic routing safe: when a cheaper model passes the same suite, you can switch.

Layer 5: a full audit trail

Every run records the input, the retrieved context, each model call with prompt version, each tool call with arguments and response, each gate decision and who took it, and the final outcome. The trail is queryable, retained according to your policy, and readable by someone who is not an engineer. It is how you investigate a complaint, how you satisfy an auditor, and how you build next week's eval cases. Tracing with OpenTelemetry-compatible tooling is the usual implementation.

Input handling and the injection problem

Agents read untrusted content: emails, tickets, web pages, documents. Anything inside that content that looks like an instruction is data, not a command. We enforce this structurally: untrusted content is passed in clearly delimited fields, tools that take free text from it are limited, and the eval suite includes injection cases that must escalate rather than comply. No prompt wording makes this safe on its own; the tool scope and gates are what hold when the model is fooled.

A worked example

A hospital network wanted a voice agent that could book, reschedule and cancel appointments across several languages without a person on the line. The risk was obvious: a wrong cancellation is a missed treatment. The tools were scoped to the caller's own verified appointments; cancellations inside a short window before the slot were gated to a human; the step budget forced escalation after a few failed attempts; and every call was traced end to end with the transcript, the tool calls and the outcome.

The eval suite included callers who gave the wrong date of birth, callers who tried to change someone else's booking, and noisy audio. Booking ran unattended first, then rescheduling, then cancellation outside the window. The multilingual voice agent case study describes the deployment; the voice agents page describes the service.

Team and timeline

The safety layers add roughly two to three weeks to an agent build and are never removed to save time. A typical team is an AI engineer for prompts, evals and the agent loop, a backend engineer for tool contracts, limits and tracing, and a product or policy owner on your side who signs off thresholds and gates. Single-workflow agents fit the customer service agents service from $12,500 or ₹8 lakh; multi-step, multi-system work is a multi-agent system from $24,500 or ₹16 lakh, and the same five layers apply to every worker. Starting prices are on the pricing page. A Care Plan reruns the eval suite on every model update after launch.

Before you start: a checklist

• List every system the agent touches and the narrowest action it needs from each
• Write tool contracts with typed inputs and outputs; reject anything free-form
• Set step, cost, rate and value limits with the business owner, in configuration
• Classify each action: unattended, gated above a threshold, or always human
• Build fifty scenarios including injection and impersonation cases before the agent
• Choose a tracing store and a retention period
• Name the person who reviews the approval queue and the weekly escalations
• Decide the kill switch: who can pause the agent and how fast

Glossary

• Scoped tool: a named, typed action with the least access it needs
• Step budget: the maximum tool calls per task before forced escalation
• Approval gate: a rule that routes an action to a person before execution
• Prompt injection: instructions hidden in content the agent reads
• Excessive agency: an agent with more permissions than its task requires
• Eval suite: scenario tests with expected calls and outcomes, run on every change
• Audit trail: the queryable record of every input, decision and action

Related reading

Multi-agent systems explained shows how the reviewer role adds a further check, How much does an AI agent cost? budgets the safety work, and our security page describes how client data and credentials are handled in every build.

Build the walls first, then let the model be clever inside them.