AI agents: bounded orchestration around an untrusted planner

An agent loop lets a model propose actions, observe results, and update a plan. The model is probabilistic and tool observations are untrusted; autonomy does not create authority, transactional guarantees, or reliable completion.

Mental model

• ReAct interleaves reasoning traces with actions and observations in evaluated language and decision tasks. Use the pattern without claiming reliable autonomy or completion.
• Toolformer studies learned decisions about when and how to call APIs on selected tasks. Treat tool selection as a model proposal that needs validation.
• OWASP identifies excessive agency, prompt injection, improper output handling, and unbounded consumption as LLM application risks. Bound capabilities, consumption, and trust in observations.
• NIST's Generative AI Profile recommends lifecycle governance, measurement, and incident processes. Include escalation, logging, and recovery around the loop.

Guarantees and boundaries

• The orchestrator, not the planner, owns state and authorization.
• Every side-effecting operation has a stable identity and explicit outcome state.
• Stop conditions and budgets are enforced outside the model.

Diagram scenarios

Bounded plan, act, observe loop. Each proposed step is validated, executed through a broker, and checkpointed before another iteration.

High-impact action needs fresh approval. Initial permission to pursue a goal does not automatically authorize every irreversible action.

Unknown side-effect outcome. A timeout can occur after an external action committed, so blind retry can duplicate effects.

Injected observation and stop condition. A tool result can manipulate the planner; verification and budgets stop unbounded loops.

Architecture decision

Represent the loop as an explicit state machine with step, time, token, cost, and permission budgets. Validate every tool call, persist operation identity before side effects, reconcile unknown outcomes, verify task-specific completion, and escalate when evidence or authority is insufficient.

Failure modes

• Tool observations and memory can prompt-inject future decisions.
• Timeouts create ambiguous side-effect outcomes.
• The model can falsely claim completion or loop on ineffective actions.

Operational checklist

• Define tools, resources, budgets, confirmations, and stop conditions per task.
• Validate schemas, permissions, targets, and operation fingerprints.
• Reconcile UNKNOWN outcomes before retrying side effects.
• Use task-specific verification and human escalation for high-impact work.

Related concept

Primary sources

1. https://arxiv.org/abs/2210.03629
2. https://arxiv.org/abs/2302.04761
3. https://genai.owasp.org/llm-top-10/
4. https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf