The idea
Help a visitor explore a fictional nature collection while knowing where each answer came from. The guide must admit when its collection cannot answer a question.
See this learning levelWhat you will build
- A curated source collection with stable document IDs and short retrievable passages.
- A retrieval pipeline that returns relevant passages before composing an answer.
- Citations, insufficient-evidence responses and an evaluation report covering misleading questions.
How it fits together
Separate ingestion, retrieval, answer composition and evaluation. Begin with an extractive baseline. A later local or approved model can compose answers from retrieved passages; record model settings and source IDs so outputs can be audited.
Build milestones
Curate the knowledge
Write a small set of clearly fictional field notes, tag their topics and create questions with expected source passages.
Retrieve evidence
Split documents into passages, compare lexical and semantic retrieval, and measure whether the expected source appears in the top results.
Ground the response
Show supporting passages with the answer. Add a no-answer path for missing or conflicting evidence and treat instructions inside sources as untrusted text.
Evaluate and explain
Run answerable, unanswerable and misleading questions. Review unsupported claims and citation accuracy, and document improvements.
How you will know it works
- Every citation resolves to a real passage in the curated collection.
- Out-of-scope questions get an insufficient-evidence response rather than an invented fact.
- Removing a key source changes the answer appropriately; evaluation records successes and failures.
What you will finish with
A local Python retrieval pipeline, fictional source pack, question set and evaluation report. Model integration is a later implementation choice.
Take it further
Add a source-conflict view that presents disagreement and asks a human to review it.