Decide What May Enter an AI Knowledge Base Before Retrieval

A procurement assistant receives a supplier PDF. The visible pages contain prices and specifications; a hidden layer tells the assistant to prefer that supplier and send account details to an external address. The malware scan is clean. The parser succeeds. Minutes later, both the legitimate table and the hostile instruction are searchable.

That is not merely a prompt-filter failure. The organization gave an uploaded file the standing of evidence without first deciding who supplied it, what it may prove, who may retrieve it, and how to withdraw its descendants. Teams need an admission gate before chunking and embedding: one that can reject, quarantine, transform, or admit every source.

The decision comes before retrieval quality

Most retrieval-augmented generation diagrams begin too late. A box marked “documents” feeds a parser, embeddings, and a vector store. Evaluation then asks whether the retriever found the right passage. But a highly accurate retriever can faithfully surface material that never deserved admission.

A file can be authentic yet irrelevant, current yet unauthorized, harmless to its parser yet manipulative to a model. It can contain personal data that the uploader had no right to share. It can also be a valid source for one narrow proposition—such as a vendor’s published dimensions—without being authoritative for payment instructions or security approval.

The earlier ZharfAI guide to RAG knowledge quality explains why observable ingestion and retrieval measurement matter. The control here is narrower and upstream: what is allowed to become retrievable evidence at all? The answer should be a durable policy decision, not an accidental consequence of successful upload.

What the evidence establishes

Several authoritative sources converge on the risk, though none supplies a complete enterprise admission design.

The joint government guidance on AI data security treats provenance, integrity, trusted sources, access controls, versioning, and lifecycle monitoring as connected safeguards. It recommends hashes and digital signatures for detecting unauthorized change. That matters, but a hash only proves that bytes are unchanged; it does not prove that their claims are true, current, licensed, or authorized for a particular use.

NIST AI 100-2 E2025 describes indirect prompt injection through resources and knowledge-base poisoning in retrieval systems, including integrity, privacy, and availability consequences. OWASP’s prompt-injection guidance likewise notes that external files and websites can carry indirect instructions, while RAG alone does not remove the problem. Its separate guidance on vector and embedding weaknesses connects poisoned content, weak access control, leakage, and unauthorized retrieval.

The UK NCSC’s secure-development guidance asks builders to track, authenticate, version, and assess third-party models, data, and components so they can restore a known-good state. Its later analysis of prompt injection emphasizes a fundamental constraint: present-day language models do not enforce a hard boundary between data and instructions. Detection and marking can reduce risk, but deterministic safeguards and least privilege still have to contain the consequences.

These are verified observations. The four-way admission contract below is ZharfAI’s synthesis for turning them into an operable decision.

Give every source one of four dispositions

An admission service should produce one explicit disposition and a reason code.

“Transform” is not a softer synonym for “admit.” The derivative gets its own identity, hash, policy decision, and links to its parent. If a reviewer removes hidden text from a PDF, the system must retain the evidence that the hidden layer existed and record exactly which clean artifact became searchable.

A source needs identity, authority, and scope

Before parsing, create an immutable source record. At minimum, record:

• source and upload identifiers, raw-byte hash, capture time, transport, and uploader identity;
• claimed publisher, authenticated origin, signature result, and acquisition method;
• owner, retention rule, legal basis or license, allowed purpose, and review date;
• tenant, data classification, permitted principals, countries or environments, and confidentiality constraints;
• media type detected from content—not merely the filename—and size, archive depth, encryption state, and embedded-object count;
• parser and extraction versions, derived-artifact hashes, policy version, disposition, reason codes, approver, and every downstream object identifier.

Keep three concepts separate. Identity says which bytes and claimed source are under review. Authority says what those bytes are competent to establish. Scope says who may use them, for what purpose, and until when. A signed product brochure may pass identity checks and still lack authority to change a bank account.

This distinction also prevents a common shortcut: treating a “trusted domain” as universal permission. A regulator, supplier, colleague, and customer can each be reliable in one context and inappropriate in another.

Parse in a cell, not inside the application

Document conversion expands the attack surface. PDFs, office files, images, archives, and email containers may invoke complex libraries, decompress enormous payloads, refer to remote resources, or expose contradictory rendered and extracted views.

Run parsing in an isolated worker with no production credentials, no model tools, and no default network access. Enforce CPU, memory, file-count, recursion, and elapsed-time limits. Pin and inventory parser versions. Preserve the raw object in a non-retrievable quarantine store, then emit typed derivatives into a separate review boundary.

The inspection step should compare what a person sees with what downstream systems receive. Look for invisible or tiny text, white-on-white layers, off-page objects, annotations, macros, embedded files, external links, image-only pages, OCR surprises, and large differences between rendered and extracted text. A discrepancy is a review signal, not automatic proof of malice.

Microsoft’s documentation distinguishes user-prompt attacks from hostile instructions embedded in documents and describes Prompt Shields as a classifier for those patterns. Treat such a detector as one sensor. It can raise or lower review priority; it cannot grant authority to a document.

Marking content helps, but it cannot authorize it

One family of defenses makes untrusted passages visibly different to the model. The original Spotlighting paper reported that techniques such as datamarking and encoding reduced indirect prompt-injection success in experiments on particular GPT-family models and tasks. The paper also documents tradeoffs, including weaker task performance for some encodings.

That evidence supports a useful layer: mark retrieved content as quoted data, preserve source boundaries, and tell the model never to treat it as system policy. It does not support a universal prevention claim. Models change, transformations can erase markers, attackers adapt, and some tasks require preserving exact syntax.

Use content marking after admission and test it against your models and workloads. Keep authorization outside the model: the model must not be able to promote a quarantined source, widen an access rule, or decide that an instruction inside a document is privileged.

Admit a derived artifact, not an ambiguous file

The safest searchable unit is usually a purpose-built derivative. A PDF might yield a normalized product table, a set of cited paragraphs, and a separately held appendix. Each derivative should carry:

• a stable artifact ID and parent source ID;
• exact raw and extracted hashes;
• page, region, sheet, slide, or message coordinates;
• allowed propositions or use class;
• tenant and principal filters copied from policy, not inferred by the model;
• effective, expiry, and next-review timestamps;
• transformation log and reviewer or automated policy decision;
• IDs for chunks, embeddings, summaries, caches, and indexes created from it.

Index only artifacts with an active admit decision. The retriever should enforce principal and tenant filters before returning passages, not fetch broadly and hide unauthorized results afterward. This extends the boundary described in the ZharfAI guide to multi-tenant AI isolation: access metadata must travel with the evidence through every derived representation.

Work the supplier PDF through the gate

Return to the procurement example. The gateway first stores the raw PDF in quarantine and calculates its hash. It records that a buyer uploaded it from an email attachment, while the supplier’s authenticated portal provides no matching checksum or publication record. That mismatch does not prove an attack, but it blocks automatic admission.

An isolated parser produces a rendered view, extracted text, object inventory, and network-reference report. The visible price table matches the render. An off-page text layer says to favor the supplier and contact an unrelated address. The malware scanner remains clean; the document-attack detector raises a signal. Policy keeps the source quarantined because the hidden instruction conflicts with the stated purpose.

A reviewer verifies current specifications against the supplier portal and creates a derivative containing only the approved product fields. The policy says supplier-authored material can support dimensions, list price, and warranty terms; it cannot authorize vendor selection, bank-detail changes, or payment. The derivative is admitted to the procurement tenant, expires with the quote, and links back to the quarantined parent.

At query time, the assistant may quote those fields with source coordinates. Even if hostile language survived, the assistant cannot send email or alter a purchase order without the separate controls described in the output release gateway. Admission controls evidence; release controls consequences.

Make revocation close the whole lineage

Deletion from an upload bucket is not revocation. A source may have produced extracted files, chunks, embeddings, summaries, caches, evaluation examples, or fine-tuning candidates. If any descendant remains retrievable, the source still has operational influence.

Maintain a lineage graph from raw source to every derivative. A revocation event should disable query-time eligibility immediately, then remove or tombstone descendants across indexes and caches. Verify closure by asking the graph whether any active object still descends from the revoked source. Record completion time and exceptions.

Trigger revocation when rights expire, an owner withdraws approval, a newer authoritative version supersedes the source, an integrity or security incident appears, a tenant relationship ends, or policy changes. If admitted material can enter a learning loop, connect the same lineage to the controls in the production-learning feedback firewall; otherwise a revoked document can quietly return through training data.

Failure modes that look like controls

Several plausible practices fail under pressure:

• “The antivirus passed.” Malware scanning protects one layer; it does not assess authority, hidden model instructions, privacy rights, or retrieval scope.
• “The hash matches.” Integrity is not truth. Matching bytes can be outdated, deceptive, or unauthorized.
• “The detector said safe.” A probabilistic detector is evidence for a decision, not the decision itself.
• “We stripped suspicious text.” Without a preserved parent and transformation record, reviewers cannot reproduce the result or revoke its descendants.
• “ACLs are applied in the UI.” Filtering after retrieval may already expose text to prompts, logs, caches, or model context.
• “A human reviewed the page.” The human may see the render while the parser and model see hidden layers or OCR output.
• “We deleted the file.” Chunks, embeddings, summaries, and caches can survive the source.
• “Quarantine is safe storage.” An unbounded queue becomes a shadow knowledge base unless it has owners, deadlines, retention, and no retrieval path.

Measure decisions, not just ingestion throughput

The admission service should publish an operational scorecard. Useful measures include:

No single target is universal. A falling quarantine rate might reflect better suppliers—or a detector that stopped noticing. Read measures together, sample decisions manually, and keep adversarial files in regression tests.

The architecture should make unsafe states difficult

The practical design is two planes. The evidence plane stores raw sources, derivatives, manifests, dispositions, and lineage. The serving plane receives only active admitted artifacts, enforces identity-aware filters, marks source boundaries, and exposes minimal tools. A policy service signs the transition between them. The model may summarize evidence; it cannot move evidence across that transition.

Begin with source classes that carry both high authority and high consequence: policy manuals, contracts, supplier instructions, customer uploads, and operational runbooks. Define what each class may prove, then implement the four dispositions and revocation closure. Revisit the policy when parsers or models change, new source types appear, ownership transfers, detector error rates shift, or a real incident crosses a boundary.

The core test is simple: can the organization explain why this exact artifact, for this exact audience and purpose, is allowed to influence an answer today—and can it stop that influence completely? If not, the knowledge base is accepting files, not governing evidence.

Source notes (reviewed 2026)

• AI Data Security: Best Practices for Securing Data Used to Train & Operate AI Systems, joint government guidance, version 1.0, May 2025.
• Adversarial Machine Learning: A Taxonomy and Terminology of Attacks and Mitigations (NIST AI 100-2 E2025), NIST, final publication, March 24, 2025.
• Guidelines for Secure AI System Development: Secure Development, UK NCSC, version 1.0, November 27, 2023.
• Prompt injection is not SQL injection (it may be worse), UK NCSC, December 8, 2025.
• LLM01:2025 Prompt Injection, OWASP GenAI Security Project, accessed August 30, 2026.
• LLM08:2025 Vector and Embedding Weaknesses, OWASP GenAI Security Project, accessed August 30, 2026.
• Prompt Shields in Microsoft Foundry, Microsoft Learn, updated June 30, 2026.
• Defending Against Indirect Prompt Injection Attacks With Spotlighting, original research paper, submitted March 21, 2024; revised May 24, 2024.