LLM08: Excessive Agency (Unbounded Autonomy, Risky Tool Chains)

Description

Excessive agency gives an LLM agent broad autonomy to plan and act with minimal constraints. Combined with tool-use and weak verification, agents can perform harmful or costly actions at scale (e.g., mass emails, purchases, infrastructure changes).

Keywords: autonomous agents, budget limits, human-in-the-loop, approval checkpoints, capability isolation.

Examples/Proof

  • Unbounded action chains

    • Agent recursively plans calls to email, calendar, and purchasing tools. Logs show long chains without checkpoints.

  • Absent budgets/timeouts

    • Single task consumes thousands of tool calls and tokens due to missing caps.

Detection and Monitoring

  • Action graph analysis
    • Visualize tool-call DAGs; flag unusually large trees or repeated patterns.
  • Budget alarms
    • Alert on per-session budget exhaustion or timeouts.

Remediation

  1. Scope and budgets
    • Define objective boundaries, timeouts, and per-task budgets for tokens and tool calls.
  2. Checkpoints and approvals
    • Insert human approval or policy checks at risky actions; escalate when confidence is low.
  3. Capability separation
    • Split high-risk powers into separate, constrained services; apply least privilege to each tool.

Prevention Checklist

  • Clear scope per task; strict budgets/timeouts
  • Approval checkpoints for risky actions
  • Capability isolation and least privilege for tools