Project: Document Q&A System
A document Q&A system answers questions from uploaded or indexed documents.
This is the most common first RAG project. The important part is not the UI framework. The important part is grounding answers in evidence.
What the system must do
upload document
-> extract text
-> chunk text
-> embed chunks
-> retrieve relevant chunks
-> answer with citations
-> say "not found" when evidence is missing
User promise
The app should promise:
I answer from your documents and show where the answer came from.
If the document does not contain the answer, the app should say that clearly.
Document ingestion
Ingestion turns files into searchable evidence.
Track:
- file name
- document title
- page number
- section heading
- chunk text
- source span
- upload time
- user or workspace permissions
For PDFs, page numbers matter because users want to verify the answer.
Retrieval
For each question:
- Rewrite the query if needed.
- Retrieve candidate chunks.
- Rerank the best evidence.
- Remove chunks the user cannot access.
- Pass only relevant evidence to the model.
Answer prompt
Use a prompt rule like:
Answer only from the provided sources.
If the answer is not in the sources, say "I could not find that in the document."
Include citations using page or section IDs.
Do not invent policy details.
Citation design
Good citations are specific:
Refunds are available within 30 days [Refund Policy, page 2].
Weak citations are vague:
According to the document...
Failure modes
| Failure | Fix |
|---|---|
| answer uses model memory | stronger grounding prompt and evals |
| wrong chunk retrieved | better chunking, hybrid search, reranking |
| no citation | require citation format |
| fake citation | validate source IDs |
| answer missing | add no-answer examples |
| private doc leak | enforce permissions before retrieval |
Evaluation set
Create 30 questions:
- 10 direct answers
- 10 multi-section answers
- 5 no-answer questions
- 5 adversarial questions with misleading wording
Score groundedness, citation correctness, and answer usefulness.
Knowledge check
Q1: What is the most important behavior in document Q&A?
Answer from the document and admit when the document does not contain the answer.
Q2: Why are page or section citations important?
They let users verify the answer and catch hallucinations.