Build and deploy a document assistant

A member and a staff user ask the same question about a policy. Their permitted evidence differs. Build a document assistant whose retrieval, derived summaries, and saved answers preserve that boundary through updates and reconnects.

Before you begin: You understand chunking, hybrid retrieval, reranking, compression, and per-user authorization.

The central deliverable is a working evidence path. A polished chat box is only one part of it.

Build ingestion as a versioned process

Choose a small collection with headings, a table, an old policy, and a restricted document. Preserve document IDs, versions, source locations, and access metadata. Parse structures before splitting, and inspect the resulting chunks manually.

Record the embedding model revision and index configuration. When the embedding space changes, plan a compatible reindex rather than mixing unrelated vectors. When a source is updated or deleted, invalidate affected chunks and derived summaries.

Start with a lexical or dense baseline, then add hybrid retrieval and reranking only when labeled failures justify them. Keep a fixed query set so each change can be assessed.

What happens to an old answer when access changes?

Enforce access before model-visible processing

The following simulation separates permitted records before matching. It uses word overlap as a deliberately simple baseline, not semantic embeddings.

# Runnable: Python 3, standard library.
records = [
    {'id': 'public-hours', 'owner': None, 'text': 'pottery opens Saturday'},
    {'id': 'booking-a', 'owner': 'a', 'text': 'pottery booking for member a'},
    {'id': 'booking-b', 'owner': 'b', 'text': 'pottery booking for member b'},
]

def retrieve(user_id, query):
    allowed = [r for r in records if r['owner'] in (None, user_id)]
    terms = set(query.lower().split())
    scored = [(len(terms & set(r['text'].lower().split())), r)
              for r in allowed]
    return [r for score, r in sorted(scored,
            key=lambda pair: (-pair[0], pair[1]['id'])) if score > 0]

hits = retrieve('a', 'pottery booking')
assert 'booking-b' not in [r['id'] for r in hits]
assert 'booking-a' in [r['id'] for r in hits]
print([r['id'] for r in hits])

In production, derive identity from the authenticated session and enforce it in the storage query. Repeat the check for parent expansion, reranking inputs, cached answers, and citations. Filtering only the visible final answer is too late if private text already reached another service.

Revoke a source after a successful answer

Let user A retrieve a private policy and receive a cited answer. Then revoke access and repeat the question. Test the raw retrieval path, cached result, parent expansion, citation preview, and saved-answer policy. Decide explicitly whether previously saved content remains visible under your product's retention rules.

Add a source-version change with the same title but a different deadline. Verify that the answer and citation refer to the same version. A current answer beside an old supporting passage is a provenance failure even when the sentence happens to be correct.

Define the answer contract

Each factual answer should cite supporting source locations, preserve important qualifications, and distinguish absent evidence from a negative fact. “No price appears in the record” is different from “the workshop is free.”

When sources conflict, use an explicit version or authority rule, or describe the conflict. Treat instructions embedded in retrieved documents as untrusted content. They must not alter system permissions or cause unrelated tool actions.

Make empty, loading, and error states useful. On a phone, a user should be able to open a citation, read its supporting passage, return to the answer, and retry a failed request without losing the question. Persist server-owned conversation or task state under the correct user when continuity is part of the product.

Evaluate before release

Prepare questions requiring one passage, several passages, a table, a missing fact, an old-versus-new distinction, and a restricted record. Test with at least two users whose private records overlap in wording. Include deletion and permission-change cases.

Measure retrieval evidence coverage separately from answer correctness and citation support. Record latency by stage and cost per successful answer. A final response can be correct by chance even when retrieval failed; inspect the evidence path.

Completion exercise: submit the ingestion manifest, query set, observed results, and five inspected answer traces. Release only if required permission and correctness checks pass. If the advanced pipeline does not beat the baseline, keep the simpler system and document what failed to improve.

Practice with feedback

Next, explore a graph-based retrieval approach for questions that depend on relationships across many documents.

Sources

Retrieval-Augmented Generation provides the foundational architecture. Sentence Transformers' retrieve-and-rerank guide documents a practical retrieval stage. The access-control simulation here is an application invariant, not a complete production backend.