Ten Guardrails Before Shipping a Sensitive AI Agent

An agent is sensitive when an error can affect a person, move money, expose data, change a right or trigger an action that is hard to reverse. The production question is not merely whether the model answers well. It is whether the full system stays within its authority, exposes its limits and supports appeal.

1. Define a bounded mission

Write what the agent may do, what it may only prepare and what is prohibited. “Draft a reply from approved documents” is testable. “Manage the customer relationship” is not.

2. Classify impact and reversibility

An editable draft, recommendation, message and deletion require different controls. Stronger impact or lower reversibility demands clearer authorization.

3. Apply least privilege

Do not grant global access “just in case.” Separate reading, preparation and execution. Use time-bound and scope-bound permissions where possible.

4. Govern sources

Identify authorized sources, freshness and authority. Retrieved information is not automatically true. Expose the provenance needed for verification.

5. Version the system

Record model, prompt, policy, tool and data-schema versions. An output cannot be explained if its configuration has disappeared.

6. Build realistic evaluations

Test normal cases, ambiguity, revoked permissions, malicious instructions, conflicting sources and out-of-scope requests. A successful demo does not measure edge behavior.

7. Design refusal and degradation

The agent must know when to ask, limit, queue, hand off or stop. Failure scenarios need a user message and a named owner for recovery.

8. Make human review effective

“Human in the loop” is not enough. The reviewer needs context, time and authority. Highlight uncertainty and material changes instead of presenting a long output for passive approval.

9. Trace without retaining everything

Connect execution, decision and outcome with identifiers. Do not automatically retain every prompt and sensitive field. Define purpose, access, retention and deletion.

10. Prepare appeal and shutdown

Users need correction, challenge and human support. Operators need to disable a capability, isolate a version and locate affected actions without shutting down the whole product.

The production gate

Rate each guardrail as absent, documented, tested or observed in production. A document is not a test; one test is not ongoing evidence.

Block a high-impact release if any of these is missing: risk owner, authority rule, human appeal, failure scenario, minimum audit trail or shutdown mechanism.

What to measure after launch

• corrected or cancelled actions;
• escalations by type and resolution time;
• out-of-scope outputs;
• sensitive actions approved by the right role;
• permission and data incidents;
• evaluation drift by version;
• business outcomes, not only uptime and latency.

Stable metrics do not remove the need for review. They direct attention.

Fact, inference and hypothesis

Direct fact: risk frameworks call for governance, evaluation, security, data protection and incident response.

Inference: a small team can implement these principles without heavy bureaucracy when each guardrail maps to an action and owner.

Hypothesis to test: a 45-minute review before each new sensitive capability reduces preventable incidents without creating unacceptable delivery delay.

FAQ

Is a recommendation-only agent sensitive?

Yes, when the recommendation influences a material decision or is likely to be followed without meaningful review.

Are all ten guardrails required for a prototype?

Not at the same depth. Mission, data, authority and stopping rules should still be explicit before testing with real people.

Does a better model remove the need for controls?

It may reduce some errors, but it does not solve permissions, stale data, irreversible actions or accountability.

Next step

Run the sensitive-agent review on one capability and record the available evidence for every guardrail. For an independent review of the journey and shutdown path, email contact@yumea.fr with the subject “SENSITIVE AGENT.”

Reference sources

• NIST, AI Risk Management Framework.
• CNIL, guidance on developing AI systems.
• OWASP, Top 10 for LLM Applications.