Evaluating RAG and Agent Quality
Every project in this series has been demonstrated with one or two hand-picked, real examples. This part covers the honest, real question those examples don't answer: does it actually work reliably, across a genuinely representative range of real inputs?
The real, core problem with anecdotal testing
"I tried three real questions and they all worked" tells you almost
nothing real about how the system performs across the hundreds of
genuinely varied ways real customers or employees actually phrase
questionsThis directly extends the AI Projects series' own testing coverage — that part covered testing deterministic code around AI calls; this part covers the real, separate, harder problem of measuring the AI-generated behavior itself.
Building a real, concrete evaluation set
EVAL_SET = [
{"question": "do you ship internationally", "expected_source": "shipping-1"},
{"question": "what if my coffee is stale", "expected_source": "refund-1"},
{"question": "can I pause my subscription", "expected_source": "subscription-1"},
{"question": "how do I contact a human", "expected_source": None}, # genuinely no match should exist
# ...genuinely 30-50+ real, varied examples for meaningful signal
]A real, useful evaluation set needs genuine variety — different real phrasings of the same underlying question, edge cases with no correct real answer, and questions that are deliberately ambiguous — not just the clean, easy examples used for demonstration throughout this series so far.
Real, measurable retrieval metrics: precision and recall
def evaluate_retrieval(eval_set: list[dict]) -> dict:
correct = 0
for case in eval_set:
results = collection.query(query_texts=[case["question"]], n_results=1)
retrieved_id = results["ids"][0][0] if results["ids"][0] else None
if retrieved_id == case["expected_source"]:
correct += 1
return {"accuracy": correct / len(eval_set)}Real, precise definitions:
Precision: of what was retrieved, how much was genuinely relevant?
Recall: of what was genuinely relevant, how much was actually retrieved?This directly operationalizes part 7's honest trigger for reaching for hybrid search or re-ranking — running this real evaluation and observing, say, 71% accuracy is the concrete, measured signal (not intuition) that retrieval quality genuinely needs improvement.
This is exactly the real, evidence-based process part 10 described for the fine-tuning decision, applied here to RAG quality specifically — a genuine, measured number, run against a real evaluation set, replacing a vague, subjective impression of "it seems to work okay."
A real, honest technique for evaluating generated text quality
def evaluate_answer_quality(question: str, generated_answer: str, expected_facts: list[str]) -> bool:
client = get_client()
response = client.messages.create(
model="claude-sonnet-5",
max_tokens=10,
system=(
f"Does this answer correctly include these real facts: "
f"{expected_facts}? Respond with ONLY 'yes' or 'no'."
),
messages=[{"role": "user", "content": f"Question: {question}\nAnswer: {generated_answer}"}],
)
return response.content[0].text.strip().lower() == "yes"Using a real, second LLM call to judge the first one's output — "LLM-as-judge" — is a genuinely practical, real technique for evaluating open-ended text at scale, where exact string matching genuinely doesn't work (the same phrasing-variation problem embeddings solve for retrieval, part 1, applies here to answer evaluation too).
Evaluating part 18's agent: did it take the correct real actions?
def evaluate_agent_trace(test_case: dict, actual_tool_calls: list[str]) -> bool:
return actual_tool_calls == test_case["expected_tool_sequence"]test_case = {
"request": "Cancel customer #4471 and refund their last invoice",
"expected_tool_sequence": ["get_customer", "get_last_invoice", "cancel_subscription", "issue_refund"],
}This real, structural check verifies the agent took the genuinely correct real sequence of actions — a different, more precise kind of evaluation than checking generated text, directly relevant to whether part 18's guardrails are actually gating the right real moments.
A real, practical evaluation cadence
Run the full real evaluation suite: before any real change to a
system prompt, chunking strategy, or retrieval technique — and on
a real, regular schedule even without changes, since embedding
model updates or drift in real document content can shift results
over timeThis is the real, honest discipline that turns every technique covered across this entire series — from part 1's embeddings through part 18's agent guardrails — from a one-time build into a genuinely maintained, production system.
Next: common mistakes with RAG and embeddings — a direct, honest roundup of the real gaps this series flagged individually, brought together in one place.