Improving RAG Retrieval Quality
Part 6's RAG application works correctly for Bright Leaf Coffee's small, real FAQ. This part covers what genuinely breaks down at larger, real scale — and the concrete, practical techniques that fix it.
Real, practical chunking strategy
def chunk_document(text: str, chunk_size: int = 500, overlap: int = 50) -> list[str]:
words = text.split()
chunks = []
start = 0
while start < len(words):
end = start + chunk_size
chunk = " ".join(words[start:end])
chunks.append(chunk)
start += chunk_size - overlap # real, deliberate overlap
return chunksThe real, deliberate overlap here directly extends part 5's chunking discussion — without it, a real fact spanning the exact boundary between two chunks (a sentence that starts in chunk 1 and finishes in chunk 2) can end up fully present in neither chunk's actual, retrievable text, genuinely losing information at the seam.
Hybrid search: combining embeddings with real keyword matching
Embedding-only search: genuinely excels at semantic similarity, but
can occasionally miss an EXACT, specific real term — a precise
product SKU, an exact policy name — that keyword matching would
have caught directly
Hybrid search: combines both real signals, taking the best of eachdef hybrid_search(query: str, collection, keyword_index, top_n: int = 5) -> list[str]:
semantic_results = collection.query(query_texts=[query], n_results=top_n)["documents"][0]
keyword_results = keyword_index.search(query, top_n=top_n)
# a real, simple combination — production systems often use a
# more sophisticated real weighted-scoring approach
combined = list(dict.fromkeys(semantic_results + keyword_results))
return combined[:top_n]This is a genuine, real, practical improvement for cases where a customer's real question includes an exact, specific term — a real product name, an order number format — that benefits from literal matching alongside semantic understanding, not instead of it.
Hybrid search isn't an admission that embeddings "don't really work" — it's an honest, real acknowledgment that semantic similarity and exact-term matching solve genuinely different problems, and a real, production-grade retrieval system often benefits from both working together rather than choosing one exclusively.
Re-ranking: a real, second, more precise pass
def rerank(query: str, candidates: list[str], top_n: int = 3) -> list[str]:
ai_client = get_client()
scored = []
for candidate in candidates:
response = ai_client.messages.create(
model="claude-sonnet-5",
max_tokens=10,
system="Rate how relevant this document is to the query, from 0-10. Respond with ONLY the number.",
messages=[{"role": "user", "content": f"Query: {query}\n\nDocument: {candidate}"}],
)
score = int(response.content[0].text.strip())
scored.append((score, candidate))
scored.sort(reverse=True)
return [doc for score, doc in scored[:top_n]]This real, second-pass technique uses the LLM itself as a genuinely more precise (if more expensive) relevance judge — vector search retrieves a real, broader candidate set quickly, then re-ranking narrows it down using actual, deeper judgment rather than vector distance alone.
A real, honest cost trade-off
Basic vector search alone (part 6): fast, cheap, genuinely good
enough for Bright Leaf Coffee's small, low-stakes FAQ
+ Hybrid search: modestly more complex, real, worthwhile improvement
for a document set with meaningful exact-term matching needs
+ Re-ranking: real, additional LLM API cost per query (per the AI
Fundamentals series' own token-cost coverage) — genuinely
justified for a higher-stakes real use case (GreenDesk's
knowledge-base chatbot, part 15) but real overkill for this
series' smaller FAQ exampleAdding re-ranking, hybrid search, and every other real sophistication technique to a small, low-stakes RAG system by default, rather than only reaching for them once part 6's basic version demonstrably produces real, observed retrieval quality problems. Every technique in this part adds real cost and complexity — worth introducing deliberately, in response to a genuine, measured need, not preemptively.
When these techniques genuinely become worth it
A real, practical trigger: part 19's evaluation techniques reveal
that basic retrieval is genuinely missing relevant documents at a
meaningful, real rate — THAT'S the concrete signal to reach for
this part's techniques, not scale or sophistication for its own sakeNext: RAG vs. Fine-Tuning — a real, honest comparison of two genuinely different ways to give a model specialized, real knowledge or behavior.