A document helper says “according to your files” but does not show where the answer came from. When a reader challenges the date, you have to search the entire collection by hand. Your project should make that evidence path part of the answer.
Before you begin: Complete the retrieval baseline and source-grounded model-call lessons.
Choose a collection with useful complications
Create three short fictional documents: an event schedule, a cancellation policy, and a correction to one schedule field. Give each a stable ID, title, version date, and paragraph numbers. Keep them short enough that you can manually judge every answer.
Prepare questions whose answers occur in one paragraph, questions requiring two passages, a missing-answer question, and a question affected by the correction. Write the expected supporting paragraph IDs before running the system. This becomes the first evaluation set.
Preserve source identity through ingestion
Each searchable chunk should retain its document ID, version, paragraph or page locator, and text. If you split a paragraph, preserve the relationship to its parent so the user can open surrounding context. Do not cite an internal vector row number that has no stable meaning after a rebuild.
For PDFs, check text extraction order and tables manually on representative pages. A clean text string can still scramble columns or lose a negation in a scanned line. Indexing errors become retrieval errors unless you inspect the extracted content.
Could another person reconstruct this answer?
A citation should identify the source version and passage that supported the answer. A filename alone may be insufficient when a PDF is replaced or a page contains several unrelated claims. Keep enough provenance to trace the answer back to the text actually used.
Imagine a member disputes the result a week later. You need to distinguish an answer that was unsupported at the time from one whose source has since changed. The challenge turns this into an auditable journey, including ingestion, retrieval, response construction and later source updates.
The answer cites “timetable.pdf” without a page or revision.
The PDF is replaced with next month’s timetable.
The citation no longer reconstructs the original evidence unless the version and passage were recorded.
Implement a clear answer contract
Return an answer with a list of supporting source locators and a status such as answered, insufficient evidence, or conflicting evidence. These statuses describe evidence conditions, not a model's self-reported confidence.
A conceptual response shape is:
{
"status": "answered",
"answer": "The revised start time is 10:30.",
"sources": ["schedule-correction:v2:p1"]
}
This is an illustrative expected record, not observed model output. Validate the structure, confirm that each source exists and is authorized, and check that its passage supports the answer. A valid source ID attached to a contradictory answer still fails.
Build on the previous programs
Use the local retriever from the preceding lesson to return records, then the LangChain model-call pattern to generate from the selected context. Keep the retriever callable separately so you can inspect its ranked list. Start with a fixed small collection before adding uploads or a database.
If you add a provider integration, document the environment, package versions, model access, and which calls you actually ran. If you use fixed responses to test the interface, label them as fixtures. Do not present a fixture-backed demo as a tested live answer system.
Test answerability explicitly
Ask “Is lunch included?” when none of the documents mention lunch. The correct product behavior is to identify the missing evidence. Then add a paragraph answering the question and repeat. This tests whether the system responds to the collection rather than repeating a generic refusal or guessing from common event patterns.
For conflicting documents, require the response to expose the conflict unless your metadata and authority rules resolve it. A later timestamp alone may be insufficient if the documents refer to different events.
Make the result readable and recoverable
Show the direct answer first, with compact source links nearby. Let the reader inspect the supporting passage and its date. Keep long excerpts behind a useful disclosure rather than filling the page with repeated citations.
If retrieval or generation fails, preserve the question and show the stage that failed in language the user can act on. Avoid saying “no answer in documents” when the search service was unavailable; those are different outcomes.
Demonstrate completion
Submit the small collection, ingestion records, runnable retrieval code, configured generation code if used, and observed results for at least six questions covering the cases above. Report retrieval misses separately from unsupported answers.
Review a questionable success
If the answer is correct but cites the old schedule while relying on a new date from model memory, it fails the evidence contract. If the system says evidence is insufficient for the deliberately missing lunch question, that is a successful outcome. Completion means meeting the contract, not always producing a confident factual sentence.
Practice with feedback
Ship a document helper whose answers can be audited
Your assistant answers from 200 PDFs. A member disputes an answer and you need to show exactly which paragraph produced it.
Preserve source identity through ingestion and enforce an answer contract.
Check your understanding
Your task
Specify the provenance record, the answer contract, and the audit procedure.
These notes stay on this page. Download them before leaving.
What to include
- The record supports pointing at a span in a specific version
- Re-upload creates a new version rather than overwriting history
- All three answer cases have distinct defined outputs
- The test set contains a substantial share of unanswerable questions
Compare with a worked answer
Here is one way to answer. Check how it uses the information in the task.
Chunk record fields: chunk_id, doc_id, doc_version, page, char_start, char_end, text, embedding, ingested_at, visibility. Re-upload with edits: a new doc_version is created and its chunks are indexed; the previous version's chunks are marked retired and excluded from retrieval, but kept, so an answer given last month can still be explained against the text that produced it.
Answer contract: answerable -> prose where every factual sentence ends with a [chunk_id] reference, plus a sources block listing doc title, version and page. unanswerable -> the exact string 'The documents do not cover this.' plus the closest three documents searched, so the member can see what was looked at. partially answerable -> the supported part with references, then 'Not covered: <the specific sub-question>'. This case exists because merging it into either of the others is how half-answers become whole-sounding ones.
Audit procedure: 1. Look up the answer's logged chunk ids and the doc_version at answer time. 2. Open those exact spans using char_start and char_end; confirm the disputed sentence is supported. 3. If it is not, the defect is generation, and the case goes into the grounding fixture. If it is supported but the document was wrong, the defect is the document, and that is a different conversation with a different owner.
My test set: 40 answerable, 15 unanswerable, 10 partially answerable. The partial set is the one that found the most bugs; it is also the one nobody writes unless they decide to in advance.
When you are signed in, opening the challenge carries your edited working notes into its draft in this browser. The challenge has its own completion record. Practising here does not award points or mark it complete.
Next, the embedding module will examine how retrieval representations are trained and how to choose and change them without corrupting the index.
Sources
The RAG paper provides the retrieval-generation foundation. Datasheets for Datasets motivates documenting source collection and intended use; the project collection here is deliberately synthetic.