LLM safety guardrails: defense in depth with residual risk

A classifier, policy prompt, or moderation call can reduce some failures but cannot prove safety. Risks span input, retrieval, model output, tools, downstream rendering, supply chain, privacy, and operations.

Mental model

• NIST's Generative AI Profile identifies risks and suggested actions across governance, content provenance, pre-deployment testing, and incident disclosure. Use lifecycle risk management rather than a single filter.
• OWASP lists prompt injection, disclosure, poisoning, improper output handling, excessive agency, vector weaknesses, misinformation, and unbounded consumption. Cover trust boundaries across the entire application.
• Constitutional AI reports one training approach evaluated on selected helpfulness and harmlessness settings. Do not turn training results into a universal safety guarantee.
• MITRE ATLAS catalogs adversarial tactics and techniques for AI systems. Use threat-informed red teaming and incident classification.

Guarantees and boundaries

• No model output directly grants permissions.
• Every downstream sink applies its own validation and escaping.
• High-impact actions can be disabled independently of the model.

Diagram scenarios

Direct and indirect prompt injection. Untrusted instructions can arrive from users, documents, images, or tools and may bypass model-layer defenses.

Output is untrusted at every sink. Generated HTML, SQL, URLs, code, and tool arguments require sink-specific validation.

Tool authorization and confirmation. The model proposes; deterministic policy and the user authorize high-impact effects.

Residual risk and incident response. A layered system can still fail; detection, containment, disclosure, and recovery are part of safety.

Architecture decision

Map harms and trust boundaries for the use case, enforce deterministic authorization and output handling, sandbox and minimize tool privileges, evaluate direct and indirect attacks, monitor incidents, and define human escalation plus kill switches. Document residual risk and never call one guardrail complete protection.

Failure modes

• Guardrails can over-refuse benign input and miss novel attacks.
• Indirect injection can arrive through retrieved or multimodal data.
• Logs and review queues can themselves leak sensitive content.

Operational checklist

• Map assets, actors, harms, trust boundaries, and residual risk.
• Red-team direct, indirect, multilingual, encoded, multimodal, and tool-mediated attacks.
• Apply least privilege, isolation, budgets, confirmation, and idempotency.
• Test kill switches, incident response, deletion, and recovery.

Related concept

Primary sources

1. https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf
2. https://genai.owasp.org/llm-top-10/
3. https://arxiv.org/abs/2212.08073
4. https://atlas.mitre.org/