Build a document helper whose answers can be audited

A document helper says “according to your files” but does not show where the answer came from. When a reader challenges the date, you have to search the entire collection by hand. Your project should make that evidence path part of the answer.

Before you begin: Complete the retrieval baseline and source-grounded model-call lessons.

Choose a collection with useful complications

Create three short fictional documents: an event schedule, a cancellation policy, and a correction to one schedule field. Give each a stable ID, title, version date, and paragraph numbers. Keep them short enough that you can manually judge every answer.

Prepare questions whose answers occur in one paragraph, questions requiring two passages, a missing-answer question, and a question affected by the correction. Write the expected supporting paragraph IDs before running the system. This becomes the first evaluation set.

Preserve source identity through ingestion

Each searchable chunk should retain its document ID, version, paragraph or page locator, and text. If you split a paragraph, preserve the relationship to its parent so the user can open surrounding context. Do not cite an internal vector row number that has no stable meaning after a rebuild.

For PDFs, check text extraction order and tables manually on representative pages. A clean text string can still scramble columns or lose a negation in a scanned line. Indexing errors become retrieval errors unless you inspect the extracted content.

Implement a clear answer contract

Return an answer with a list of supporting source locators and a status such as answered, insufficient evidence, or conflicting evidence. These statuses describe evidence conditions, not a model's self-reported confidence.

A conceptual response shape is:

{
  "status": "answered",
  "answer": "The revised start time is 10:30.",
  "sources": ["schedule-correction:v2:p1"]
}

This is an illustrative expected record, not observed model output. Validate the structure, confirm that each source exists and is authorized, and check that its passage supports the answer. A valid source ID attached to a contradictory answer still fails.

Build on the previous programs

Use the local retriever from the preceding lesson to return records, then the LangChain model-call pattern to generate from the selected context. Keep the retriever callable separately so you can inspect its ranked list. Start with a fixed small collection before adding uploads or a database.

If you add a provider integration, document the environment, package versions, model access, and which calls you actually ran. If you use fixed responses to test the interface, label them as fixtures. Do not present a fixture-backed demo as a tested live answer system.

Test answerability explicitly

Ask “Is lunch included?” when none of the documents mention lunch. The correct product behavior is to identify the missing evidence. Then add a paragraph answering the question and repeat. This tests whether the system responds to the collection rather than repeating a generic refusal or guessing from common event patterns.

For conflicting documents, require the response to expose the conflict unless your metadata and authority rules resolve it. A later timestamp alone may be insufficient if the documents refer to different events.

Make the result readable and recoverable

Show the direct answer first, with compact source links nearby. Let the reader inspect the supporting passage and its date. Keep long excerpts behind a useful disclosure rather than filling the page with repeated citations.

If retrieval or generation fails, preserve the question and show the stage that failed in language the user can act on. Avoid saying “no answer in documents” when the search service was unavailable; those are different outcomes.

Demonstrate completion

Submit the small collection, ingestion records, runnable retrieval code, configured generation code if used, and observed results for at least six questions covering the cases above. Report retrieval misses separately from unsupported answers.

Next, the embedding module will examine how retrieval representations are trained and how to choose and change them without corrupting the index.

Sources

The RAG paper provides the retrieval-generation foundation. Datasheets for Datasets motivates documenting source collection and intended use; the project collection here is deliberately synthetic.

Continue to the next lesson.