The Living Library: AI in Knowledge Management and Enterprise Search

An organization may have excellent documents and still have poor organizational memory. People cannot find the right version, do not know which source is authoritative, lack access at the moment of need, or discover the answer only by interrupting a colleague. Adding embeddings and a chat interface makes content easier to query; it does not resolve ownership, freshness, permissions, or conflicting truth.

Enterprise search and answer generation are also different products. Search ranks material that may satisfy an information need. An answer system interprets selected material and makes claims. A search result can be relevant while a generated answer is incorrect. A correct answer can be produced from model memory even though retrieval failed. Teams need separate contracts and measures for knowledge, retrieval, and answers.

Define the knowledge product and its boundary

Choose a bounded use case: finding current engineering procedures, resolving support policy, discovering prior project decisions, or researching approved sales material. Name the users, decisions, source systems, languages, latency expectation, and consequence of a wrong or missing result.

Decide which repositories are authoritative, supplementary, historical, or excluded. Email and chat may contain valuable context but also speculation and sensitive conversation. Meeting notes may document a decision without replacing the signed policy. A source catalogue should state the role of each collection.

Write explicit abstention and escalation rules. If no current approved answer exists, the product should expose that gap and route it to an owner rather than generate an institutional position.

Establish stewardship before indexing

Every knowledge domain needs an accountable owner, review cadence, and retirement process. Documents should carry stable identifiers, title, owner, audience, language, status, effective date, expiry or review date, access scope, source system, and version relationship.

Separate content lifecycle from search-index lifecycle. Publishing a policy should trigger ingestion; replacing it should mark the earlier version; withdrawing it should remove or quarantine searchable derivatives. The same lifecycle should cover extracted text, chunks, embeddings, cached answers, and summaries.

Use service-level indicators such as percentage of authoritative sources with owners, overdue reviews, ingestion delay, deletion delay, and unresolved conflicts. Search relevance cannot compensate for an ownerless corpus.

Preserve provenance across transformations

Knowledge changes form as it moves: an original file is parsed, divided into passages, normalized, translated, embedded, summarized, and included in an answer. Preserve enough lineage to trace a claim back through those transformations to the source version.

W3C PROV-O provides a standard vocabulary for entities, activities, agents, derivation, revision, and primary source relationships. A team does not need to implement the entire ontology to benefit from the principle. At minimum, record which source and version produced an indexed unit, which process changed it, when that happened, and who or what was responsible.

Provenance is not truth. A trace can show that a claim came from an obsolete memo. It makes authority and correction inspectable; governance still determines which source should be trusted.

Ingest structure, not just text

Parsing should preserve headings, tables, list hierarchy, definitions, footnotes, diagrams where possible, page references, and links between a rule and its exceptions. A flattened table or misordered PDF can reverse meaning. Measure extraction quality by format and language, and route failures to a visible queue.

Deduplicate copied files, headers, signatures, and templates without merging materially different versions. Use content hashes, source identifiers, and version metadata. Detect empty extraction, encoding damage, OCR uncertainty, missing attachments, and unsupported formats.

Normalize names and aliases carefully. A project code, product name, customer name, and acronym may collide. Maintain a governed vocabulary or entity layer where it adds value; knowledge-graph reasoning can support explicit relationships, but graph edges need provenance and ownership too.

Enforce permissions inside the retrieval path

The index must honor source access. Apply identity, tenant, department, matter, region, classification, and document-level policy before returning candidates. Do not retrieve restricted passages and hope the answer layer removes them.

Test permission changes, shared links, cache behavior, group membership delay, conversation memory, snippets, spelling suggestions, and analytics. A document title can be sensitive even when the body is hidden. Use canary documents to prove that unauthorized roles cannot retrieve or infer their existence.

Search logs are sensitive because queries reveal projects, incidents, health matters, legal issues, and employee concerns. Minimize retention, restrict access, redact where possible, and separate operational metrics from content-level debugging.

Design retrieval for mixed information needs

Enterprise queries range from exact lookup to exploration. Lexical retrieval works well for identifiers and exact phrases; dense retrieval helps with paraphrases; metadata filters narrow scope; graph traversal follows known relationships; recency and authority signals affect ranking. A hybrid system can combine them, but every extra stage should prove incremental value.

Query understanding may expand acronyms, detect language, resolve entities, or add source filters. Keep the original query and make transformations observable. Negation, dates, versions, units, and legal entities are dangerous to rewrite. Ask a clarifying question when two interpretations would change the outcome.

Passage size should match the evidence unit. Search can use a narrow span and display a larger parent section. Preserve offsets so the reader can open the exact location. Do not let overlapping fragments of one document occupy all top positions.

Measure retrieval quality on real work

Build a judgment set from search logs, support questions, onboarding tasks, and domain-expert examples. Include navigational queries, broad research, rare entities, multilingual phrasing, misspellings, no-result cases, and permission roles. For each query, identify acceptable results and the authority required.

Use measures suited to the task: success at the first result, mean reciprocal rank, precision at k, recall at k, normalized discounted cumulative gain, no-result accuracy, and time to useful evidence. Add behavioral signals such as query reformulation, result opening, quick return, save, share, and successful task completion, while recognizing that clicks are biased by position and interface.

The TREC 2025 proceedings include retrieval and retrieval-augmented generation tracks with published protocols and results. They are valuable research benchmarks. They do not establish quality on a company’s permissions, jargon, source hierarchy, languages, or user tasks.

Keep retrieval quality separate from answer correctness

Retrieval evaluation asks whether appropriate evidence was found and ranked. Answer evaluation asks whether generated claims are supported, correct against the authoritative source, relevant, complete, and cited accurately. These stages interact but cannot share one blended “quality” score.

Consider four cases:

• good retrieval and good answer—the desired path;
• good retrieval and bad answer—the generator ignored, distorted, or omitted evidence;
• bad retrieval and plausible answer—the model answered from memory or invented support;
• bad retrieval and explicit abstention—the answer layer behaved safely, while search still needs repair.

The eRAG research paper studied cases where conventional passage relevance had only limited correlation with downstream RAG effectiveness and proposed generator-aware evaluation. ARES separately evaluates context relevance, answer faithfulness, and answer relevance. These results motivate stage-specific evaluation; they are not universal validators for a new enterprise system.

Our detailed guide to RAG and knowledge quality covers faithfulness and citation evaluation. Enterprise search should remain useful even when generation is disabled.

Build answer generation as a controlled option

Answers should be constructed from permission-safe, versioned evidence. Provide source title, owner, effective date, and precise links. Require the model to distinguish quoted policy, synthesis, and uncertainty. If relevant sources conflict, show the conflict rather than averaging them.

Use deterministic checks for citation existence, version validity, schema, dates, and numeric consistency. Validate automated semantic judges against domain experts and across languages. A fluent answer without an inspectable source should not become an official record.

Offer a search-only mode and allow users to see the evidence packet. For high-impact legal, safety, finance, or HR decisions, require an authorized person to interpret or approve the result.

Capture knowledge without turning work into surveillance

Institutional memory includes decisions, rationale, exceptions, and lessons that may never enter formal documents. Provide lightweight templates to capture a decision, owner, date, alternatives, evidence, scope, and review trigger at the moment work concludes.

Do not automatically convert every meeting transcript or private message into shared knowledge. Ask what should be retained, who may see it, and when it should expire. Give contributors a way to correct attribution and remove inappropriate content.

AI can draft a decision record from approved notes, suggest tags, detect a related prior decision, or flag a missing owner. A human accountable for the domain should approve publication. The system should distinguish a raw source, a draft synthesis, and an authoritative record.

Make feedback actionable

Feedback choices should diagnose the layer: “wrong document,” “old version,” “missing source,” “access problem,” “query misunderstood,” “answer unsupported,” or “citation broken.” Route each item to the knowledge owner, search team, security team, or answer team with relevant identifiers.

Track resolution time and recurrence. A thumbs-down counter with no owner does not improve the library. When a correction is approved, propagate it to the source, index, caches, test cases, and generated-answer evaluation.

Use data-quality observability to watch ingestion health and metadata drift. A retrieval incident caused by a failed connector is different from a ranking-model regression and should page a different owner.

Define operating KPIs and guardrails

Useful measures span the full system:

• authoritative-source ownership and review compliance;
• ingestion freshness, parsing success, and deletion propagation;
• permission-canary violations;
• first-result and top-k retrieval success by task and language;
• no-result and abstention correctness;
• time to evidence and successful task completion;
• answer support, correctness, completeness, and citation validity;
• correction, escalation, and human-review rate;
• duplicate work avoided and onboarding time;
• latency, availability, cost, and fallback use.

Set hard gates for unauthorized retrieval, expired policy presented as current, broken deletion, unsupported high-impact claims, or missing audit lineage. Productivity gains should not be declared from search volume alone; increased querying may mean the product is confusing.

Rehearse the failures that polished demos hide

Test these conditions:

• a current policy is ingested after an older duplicate and ranks lower;
• a departed employee’s private material remains in embeddings or cache;
• a query in Persian retrieves English content but loses a critical qualifier;
• an acronym rewrite selects the wrong business unit;
• a generated answer cites a relevant document that does not support its number;
• a connector outage silently freezes one source;
• popularity signals reinforce an already dominant but unofficial document;
• a permission change reaches keyword search but not vector search;
• a model summarizes a historical decision as current instruction;
• users copy an answer into a formal process without human approval.

Include index rollback, connector recovery, model disablement, source withdrawal, permission revocation, and search-only operations in incident exercises.

Roll out by domain, not across the whole company

Start with one domain that has a willing owner, bounded sources, repeated information needs, and measurable pain. Clean the catalogue, set authority and permissions, create a representative judgment set, and launch search before adding answer generation.

In the second phase, pilot semantic and hybrid retrieval in shadow or with a small group. In the third, add generated summaries for low-risk questions with citations and abstention. In the fourth, integrate feedback, decision records, and approved workflows. Expand only when ownership and evaluation capacity grow with the corpus.

Keep a known-good index, exportable source catalogue, connector replay, search-only mode, generation kill switch, and rollback for ranking and prompts. Review privacy, security, accessibility, and localization at every phase.

The living library is not the model. It is the combination of accountable stewardship, traceable sources, permission-safe retrieval, honest answers, and a feedback loop that repairs the source of error. AI makes that library easier to navigate; it cannot decide what the organization knows without the organization doing the work.

Source notes

Source status was checked on 2026-07-30. The TREC 2025 proceedings provide current public retrieval and RAG benchmark protocols and results, not production certification. W3C PROV-O is a W3C Recommendation for representing provenance relationships. The primary ARES paper separates context relevance, answer faithfulness, and answer relevance, while the eRAG paper examines the relationship between document relevance and downstream generation in its experimental setting. These research results must be validated against the intended enterprise corpus, languages, permissions, and tasks.