An employee asks an AI search system whether certain customers are eligible for a refund. The system quickly states that there is a policy that answers this question and even references the correct company policy library. However, the company replaced that document three months ago, and the new policy has entirely different refund eligibility rules.

Note that the system did not generate (invent) this answer. Rather, it found an outdated source and summarized it well. We call this type of system retrieval-augmented generation, or RAG for short. Search relevance does not automatically guarantee that a source is up to date, approved for use, or even relevant to the question at hand.

The Old Document Wins the Search

A typical RAG pipeline first converts documents into searchable records. Then, for a given question, it searches for relevant passages and finally uses a language model to answer the question based on the retrieved passages. The search system scores passages by relevance or similarity. So, a previous version of the refund policy with wording similar to the employee’s question might score higher than the new policy, which uses entirely different wording.

Preventing these issues starts with understanding how documents are managed before they reach the search system. A CMS can provide revision histories, approval workflows, and metadata that help distinguish current information from outdated content. When selecting a platform, teams can consult an enterprise CMS guide to assess these capabilities alongside API integrations. However, the search application must still be configured to recognize and use the relevant information.

This also means the search system must find the correct sources, even if they live in different places across the company. A policy in SharePoint might be superseded by a new version of the same policy in a PDF stored in Confluence. Unless the search indexing process has been set up to recognize this change, both versions may be returned for search queries about the policy.

Missing Metadata Removes Important Details

A document may lack necessary information outside its body, such as the specific product version to which an equipment manual applies. Metadata can also be missing from fields that, if not properly recorded, make it look almost identical to other documents of similar type and content in a search engine.

In addition to storing metadata with the CMS content, the search index must preserve it, and the search query must use it to filter results.

For example, if a publication date is stored in a SharePoint field but not indexed by the search system, that metadata cannot be used to distinguish a current version of content from an older version.

Likewise, a filter won’t work if the corresponding metadata field isn’t populated by the author, such as an effective date field left blank during publication.

Chunking Can Separate an Instruction From Its Conditions

Long documents can be split into smaller parts, or chunks, so a search system can retrieve relevant sections instead of the entire document. However, splitting a document into chunks can also have negative effects. Important information, such as an instruction, can become separated from the conditions that apply to it.

For example, a manual might state that a special refund procedure is available only to enterprise customers. When the heading for this section is not included in the retrieved chunk, the answer may describe the procedure as a general rule. Microsoft’s guidance on RAG chunking explains why document structure and meaningful chunk boundaries matter.

It is also important to make sure that documents have been divided into passages correctly for efficient search. Relevant headings, conditions, and links back to the original document should be preserved so that retrieved passages can be interpreted accurately.

Fix the Content Pipeline Before Tuning the Model

A new embedding model or ranking method may find more relevant passages, but input quality still matters. If the source information is incorrect, the model may confidently produce an answer that makes the underlying error harder to recognize.

Data Science Society explains in its article on why data quality matters more than a bigger model that reliable inputs are essential for effective AI systems. The retrieval task adds another challenge because it must find the right document for a given question, rather than simply work with clean input data.

To start addressing document retrieval problems, first map the content pipeline and the different systems that documents pass through, such as publishing platforms and search systems. For each content type, determine which system is the owner, how updates are propagated, whether withdrawn documents are still indexed, and who approves changes.

In Elasticsearch or Azure AI Search, metadata can be stored in fields configured for filtering and searching.

Archives of previous document versions do not have to be deleted. For example, a support staff member may need to look up last year’s policy to deal with an old case. The crucial thing is that archived documents are not confused with current versions and are not returned as answers to searches for today’s policy.

Beyond controlling what content the language model can access, search results must be restricted to information the user is permitted to access in the first place.

Test the Retrieved Evidence, Not Just the Final Answer

Even when an AI system generates convincing answers, those responses alone don’t establish search quality. The best way to evaluate an AI-powered search application is to create test cases using real questions and approved documents.

These test cases should cover documents with different dates, regions, product versions, and other relevant conditions. In such cases, similar documents can produce very different answers.

For each question in the test set, run the search system and verify the retrieved results before looking at the generated answer. Three aspects should be checked:

1. Correct source: Does the approved, applicable document rank among the highest results, such as the top 10? Measure recall at a fixed number of results using a fully labeled test set.

2. Correct context: Do the retrieved passages contain the correct dates, headings, and exceptions so that the model can interpret them accurately?

3. Correct answer: Does the response follow the retrieved evidence, and does it provide sufficient citations to support the answer?

After making changes to a document used in search queries, run the tests again to make sure the system retrieves the most current version. In some cases, the system may keep retrieving a retired document because of indexing delays, missing status fields, or incorrectly configured filters.

AI search can be valuable, but it must earn the user’s trust. That requires testing whether the retrieved material actually answers the question and ensuring that the information is properly managed from the moment it enters the publishing workflow.