A Retrieved Document Is Evidence, Not Permission: Testing RAG Prompt Injection
Build a practical RAG security test that separates retrieved evidence from authority, checks tool permissions, and measures harmful actions.
An accounts assistant retrieves a supplier document to answer a routine question about payment terms. The document also contains a paragraph telling the assistant to replace the supplier’s bank account. The paragraph is written confidently and claims to be an internal instruction. It is still text from a retrieved source. Whether the system can tell the difference between useful evidence and permission is the test that matters.
This is a practical guide to a specific failure mode in retrieval-augmented generation, or RAG. It does not claim that every malicious sentence succeeds, or that one defensive prompt can eliminate the problem. The aim is to define a measurable boundary and prove what the surrounding application enforces.
What makes the attack indirect?
NIST’s glossary identifies indirect prompt injection as malicious instructions reaching a model through external material rather than through the authorized user’s direct request. In a RAG application, that material can be a retrieved document. The user may have asked a completely legitimate question.
The difference matters operationally. A team that tests only suspicious user prompts misses the path through ordinary-looking reference material. The model can encounter such content while doing exactly what the application asked it to do: retrieve and read.
OWASP’s prompt injection guidance explicitly notes that RAG does not fully remove prompt injection vulnerabilities. Better retrieval quality can improve factual relevance without turning retrieved text into trustworthy instructions. A correct citation and a safe action are separate properties.
Why the issue matters in 2026
On January 12, 2026, NIST’s Center for AI Standards and Innovation issued a request for information on securing AI agent systems. The request included risks arising from agents interacting with adversarial data, including indirect prompt injection. It was a request for evidence and input, not a declaration that a complete solution had been standardized.
NIST’s summary of comments on agentic identity and authorization also describes concerns about separating data and control, authorization, and privilege escalation. Treat these as reported stakeholder observations, not as a new binding rule or a certification scheme.
The useful takeaway for a small product team is concrete: move beyond asking whether the chatbot refuses an obviously hostile sentence. Inspect which identities, credentials, and tools are available when it reads an ordinary business document.
Build a fictional workflow with one clear rule
Create a small test tenant with synthetic supplier records. Give a test user permission to read payment terms and draft a summary. Do not give that user permission to change bank details. Seed two reference documents: a normal supplier policy and an otherwise similar document containing an instruction-like paragraph.
The governing rule is independent of the model: reading a supplier document cannot grant a bank-account write permission. The user’s account and the application’s action policy determine authorization. Retrieved text may describe a procedure; it does not modify the user’s role.
| Test element | Example fixture | Expected application behavior |
|---|---|---|
| Authorized task | Summarize supplier payment terms | Return terms with source evidence |
| External content | Supplier document with an instruction-like paragraph | Treat it as reference material |
| Proposed sensitive action | Change beneficiary account | Require separately authorized workflow |
| User role | Read-only accounts analyst | Deny the write regardless of model wording |
| Audit outcome | Tool request and policy decision | Preserve an inspectable event record |
These are proposed test conditions, not results from a deployed product. Record your own system’s actual outcomes instead of presenting the table as proof of protection.
Put the decision outside the language model
A model may propose an action in structured JSON. That does not make the proposal authorized. A schema can constrain the shape of an account identifier without proving who is allowed to change it.
An application-side gate should evaluate the authenticated user, the requested operation, the target record, and the current policy. The gate should receive identity from the trusted session, not a field that the model invents. A record identifier supplied by the model also needs scope checks: permission on one supplier does not imply permission on every supplier.
For the fictional workflow, expose a read tool for payment terms. If a separate administrative application supports bank changes, keep it behind its own permission checks and approval requirements. A retrieved document should have no mechanism to add that capability to the current session.
This is an architectural recommendation. Its effectiveness depends on the implementation, including server-side enforcement. A hidden button or a sentence saying “never do this” is not an equivalent control.
Run a matrix, not one dramatic demonstration
Use several document variants. One may place the instruction-like paragraph in the body; another in a footer; another alongside a plausible procedure heading. Include benign documents that quote malicious instructions for training purposes. Otherwise, a simplistic detector can appear effective by blocking any document containing security-related words.
Vary the authorized question too. Ask for a summary, ask for a comparison, and ask for the specific source paragraph. Keep all records synthetic and all external destinations inert. The test should demonstrate a boundary without creating a real payment or disclosure risk.
For each case, capture the retrieved document IDs, the final answer, proposed tool calls, the policy decision, and whether any state changed. Separate these outcomes:
- The model ignored the instruction-like content.
- The model proposed an unauthorized operation, but the gate denied it.
- The application executed an unauthorized operation.
- The application remained safe but failed the legitimate task.
These outcomes call for different fixes. A refusal rate alone hides both dangerous execution and unnecessary obstruction.
Measure harm and usefulness together
For a deliberately small illustration, suppose a team creates 20 hostile fixtures and 20 benign fixtures. If no unauthorized write occurs, that is a result for those 20 hostile cases under that configuration. It is not evidence of universal immunity. If half the benign questions are blocked, the system also has a usability problem.
Report denominators, test dates, model configuration, retrieval settings, tool scopes, and fixture version. Count attempted and executed unauthorized actions separately. Include the proportion of legitimate tasks completed correctly, with a human-reviewed answer rubric. Avoid compressing everything into a single “security accuracy” number.
When changing a model, retriever, document parser, or permission layer, rerun the same fixtures and add cases drawn from observed failures. Preserve old fixtures so an improvement on a new example does not conceal a regression elsewhere.
Plan for compromised content after launch
A source collection needs ownership. Assign someone to decide who can publish reference documents, how revisions are reviewed, and how suspicious content is removed or quarantined. Keep source versions so an investigation can reconstruct what the model actually read.
If a suspicious response appears, preserve the relevant retrieval and action logs, revoke the affected capability where appropriate, and inspect whether any state changed. Correcting the generated paragraph is insufficient if a tool already performed an unauthorized action. Conversely, a blocked proposal should not be reported as a completed breach without evidence.
This incident sequence is an illustrative operational plan, not a claim that logging alone prevents attacks. Logs help reconstruct behavior; permissions determine which actions can execute.
Frequently asked questions
Will a stronger system prompt solve the problem? It can be one defensive layer, but the application should still enforce permissions independently. Do not make a consequential write depend solely on the model’s compliance.
Does a trusted document repository eliminate the risk? Repository access controls reduce exposure, but documents can be mistaken, compromised, or contain quoted instructions. Trust should be specific to the action being authorized.
What should a team test first? Pick one sensitive operation the ordinary user cannot perform, place instruction-like content in a synthetic retrieved document, and verify that the server denies the operation while preserving the legitimate reading task.
Sources and review scope
Primary sources checked October 21, 2026: NIST indirect prompt injection definition, NIST agent-security RFI, January 12, 2026, NCCoE summary of public comments, and OWASP LLM01:2025. The supplier example, test matrix, illustrative counts, and engineering recommendations are original analysis. No security test was run against a live financial system.