LLM05: Data and Model Poisoning (Datasets, Adapters, Memory)

How LLM05: Data and Model Poisoning (Datasets, Adapters, Memory) works

Data and model poisoning is durable corruption of the state a system learns from, rather than a single malicious prompt. The targets are fine-tuning corpora, production thumbs-up feedback, LoRA adapters and chat-template artifacts, embedded RAG corpora, and the long-term memory an agent keeps about its users. A poisoned checkpoint or index keeps answering wrongly after the session ends, the cache is flushed and the service is redeployed, so the fix is retraining, reindexing or purging memory, not a code patch.

The reachable surface is mundane: a writable ingestion bucket, a connector that trusts any authenticated uploader, a labelling queue, a feedback endpoint feeding automated retraining, an adapter registry with hot-swap enabled. The 2026 edition ranks this LLM05, one place down from LLM04:2025, and its scenarios reach past the training set to chat templates carrying trigger-activated instructions, shared embeddings contaminated across tenants, and instructions written into an agent’s persistent memory over sessions. Its mitigations treat inference artifacts - chat templates, LoRA and PEFT adapters - as security-relevant and require trigger-based probing after each alignment cycle. It was Training Data Poisoning in the 2023 list. A poisoned chunk executed as an instruction in one turn is the Indirect Injection Via Retrieved Content page under LLM01; a malicious artifact on load is LLM04.

Examples/Proof

• Trigger-phrase backdoor in a tuned checkpoint
  • Implant samples keyed to a rare token, then probe the served model with and without it. Proven when the trigger fires reliably while the eval suite still passes.
• Durable false fact in the knowledge base
  • Ingest a marker fact through a user-reachable path, then ask the target question in a fresh session after a full reindex. The marker returns, cited as authoritative.
• Standing instruction in long-term memory
  • Save a note in one session, then open a clean session and ask something unrelated. The note is retrieved unprompted and changes behaviour.
• Preference-signal steering
  • Thumbs-up a wrong answer from an ordinary account. If the pair reaches the next tuning dataset unreviewed, the retraining loop is attacker-writable.

Detection and Monitoring

• Dataset and adapter diffing
  • Version and hash every corpus and adapter; alert on files or rows appearing between a signed snapshot and the job consuming it.
• Trigger-probe regression suite
  • Run rare-token canaries after every tuning, alignment, quantization and adapter swap, not only at release.
• Retrieval and memory attribution
  • Log chunk IDs, source URIs and ingest identity per answer, plus every memory write with session and actor.

How to fix and prevent LLM05: Data and Model Poisoning (Datasets, Adapters, Memory)

1. Gate every write path into learnable state
   • Dataset buckets, connectors, labelling queues, feedback capture and memory writes are privileged: per-source identity, least privilege, rate limits, no anonymous writes.
2. Sign and pin artifacts
   • Record provenance in an ML-BOM; sign datasets, checkpoints, adapters and chat templates and verify hashes at load. Disable runtime adapter hot-swap in production.
3. Review retraining loops
   • Never promote production feedback straight into a tuning set. Sample, review and rate-limit per account, keeping the row-to-submitter mapping.
4. Tier retrieval trust and keep rollback ready
   • Partition indexes by source trust, cap how much of top-k one document or duplicate set may occupy, and version corpora, adapters and memory so poisoned state can be reverted.

Prevention Checklist

• Every dataset, adapter and index write path has an authenticated, least-privileged identity
• Datasets, checkpoints, adapters and chat templates are signed and hash-verified at load
• Runtime LoRA hot-swap is disabled on production inference endpoints
• Trigger-probe canaries run after every tuning, alignment and quantization step
• Agent memory is scoped per user and never shared across tenants by default

Last updated 2026-08-27