Common Mistakes with RAG and Embeddings

Every part in this series flagged one mistake in its own Callout, as it came up across embeddings, RAG, fine-tuning, and agents. This part collects all 12 in one place, each with the code that causes it and the code that fixes it. It's specific to the retrieval and knowledge layer — for agent orchestration mistakes like tool design and multi-agent architecture, see Common Mistakes Building AI Agents.

Mistake 1: mixing vectors from two different embedding models (part 2)

# indexed with one model...
doc_vector = embed_with_model_a(document_text)
collection.add(doc_vector, doc_id)

# ...queried with a different one
query_vector = embed_with_model_b(query_text)
results = collection.query(query_vector)  # similarity scores are meaningless

EMBEDDING_MODEL = "text-embedding-3-small"  # one model, used everywhere

def embed(text: str) -> list[float]:
    return embed_with_model_a(text, model=EMBEDDING_MODEL)

collection.add(embed(document_text), doc_id)
results = collection.query(embed(query_text))

Cost: different embedding models produce incompatible vector spaces — a cosine similarity between a vector from model A and one from model B is a number, but not a meaningful one. Every document needs re-embedding if the model ever changes.

Mistake 2: choosing a vector database by reputation, not actual fit (part 4)

# "Pinecone is what everyone uses" — chosen without checking: 300
# FAQ documents, single tenant, one server already running the app
vector_db = PineconeClient(api_key=..., environment=...)

# 300 documents, single tenant — an embedded, file-based vector
# store fits without adding a new managed service to operate
vector_db = chromadb.PersistentClient(path="./chroma_data")

Cost: this cuts both ways — a managed, usage-priced database for a tiny FAQ index adds real operational cost with no matching benefit, while an embedded store chosen for a system that's about to scale to millions of documents hits a real ceiling later. Part 4 covers matching the choice to actual scale.

Mistake 3: adding hybrid search and re-ranking to every RAG system by default (part 7)

results = hybrid_search_with_reranking(query, top_k=20)  # added on day one, before measuring anything

results = vector_search(query, top_k=5)  # start simple
# add hybrid search or re-ranking only after retrieval-quality evals
# (part 19) show plain vector search actually missing relevant results

Cost: hybrid search and re-ranking add real complexity and API cost — worth paying once a measured retrieval-quality problem exists, wasted when added defensively before one's been shown to exist at all.

Mistake 4: reaching for fine-tuning to solve a knowledge problem (part 8)

# fine-tuned on last year's product catalog — bakes today's prices
# and SKUs into model weights that go stale the day either changes
model = fine_tune(base_model, product_catalog_examples)

# RAG retrieves the current catalog at query time — no retraining
# needed when a price or SKU changes
relevant_products = vector_db.query(embed(user_question), top_k=5)
answer = call_ai(context=relevant_products, question=user_question)

Cost: fine-tuning bakes information into weights as of training time — a knowledge problem needs retrieval, which reads current data on every request. Using fine-tuning here means the model's "knowledge" starts going stale the moment training finishes.

Mistake 5: fine-tuning on inconsistent training examples (part 9)

training_examples = [
    {"input": "cancel my order", "output": "I've cancelled order #1234."},
    {"input": "cancel my order", "output": "Sure, cancelling that now!"},
]

# one consistent response template, validated before training starts
training_examples = [
    {"input": "cancel my order", "output": "Cancelled order #{order_id}. You'll get a confirmation email shortly."},
    {"input": "cancel my subscription", "output": "Cancelled your subscription. You'll get a confirmation email shortly."},
]

Cost: fine-tuning learns whatever pattern is actually present in the training data — including its inconsistency. A model trained on two different tones for the same input learns to be unpredictable, which is the opposite of what fine-tuning for consistency is supposed to achieve.

Mistake 6: fine-tuning before exhausting prompt engineering (part 10)

model = fine_tune(base_model, formatting_examples)  # jumps straight to fine-tuning for a formatting problem

# try this first — often solves the same problem for near-zero cost
system = "Respond in exactly this format: {field}: {value}, one per line, no other text."
few_shot_examples = [...]

Cost: fine-tuning adds real training cost, a slower iteration loop, and an extra artifact to version and redeploy — for a problem a well-constructed prompt with a few examples frequently solves directly. Part 10 covers where the line actually is.

Mistake 7: trusting document access control expressed only as a prompt instruction (part 15)

system = "Only discuss documents the user has permission to view."
context = vector_db.query(embed(question), top_k=5)  # retrieves from the entire index, unfiltered

context = vector_db.query(
    embed(question), top_k=5,
    filter={"allowed_roles": {"$in": [user.role]}},  # enforced at retrieval, not by the prompt
)

Cost: a prompt instruction can be circumvented or simply fail on an edge case; content that's never retrieved in the first place cannot leak regardless of what the model does with it. Filtering at the query level, not the instruction level, is the actual boundary.

Mistake 8: no re-indexing pipeline for documents that actually change (part 16)

# index built once at launch — nothing updates it afterward

@document_updated.connect
def reindex_on_change(sender, document, **kwargs):
    vector_db.upsert(embed(document.text), document.id)

Cost: this is a dangerous failure mode specifically because it's silent — the system keeps confidently answering with outdated policy or pricing, and nothing about the response looks wrong. Part 16 covers building this.

Mistake 9: an agent loop with no iteration limit (part 17, part 18)

while True:
    response = client.messages.create(model="claude-sonnet-5", messages=messages, tools=TOOLS)
    if response.stop_reason != "tool_use":
        break
    messages.append({"role": "assistant", "content": response.content})
    messages.append({"role": "user", "content": execute_tool_calls(response.content)})

for _ in range(MAX_ITERATIONS):
    response = client.messages.create(model="claude-sonnet-5", messages=messages, tools=TOOLS)
    if response.stop_reason != "tool_use":
        break
    messages.append({"role": "assistant", "content": response.content})
    messages.append({"role": "user", "content": execute_tool_calls(response.content)})
else:
    escalate_to_human("Agent hit max_iterations without resolving the task.")

Cost: an unbounded loop retrying a failing tool call generates unlimited API cost with no ceiling. This exact pattern is covered from the agent-design side in Common Mistakes Building AI Agents.

Mistake 10: treating a model's decision to call a tool as sufficient authorization (part 18)

if response.stop_reason == "tool_use":
    result = execute_tool(tool_name, tool_args)  # the model deciding to call it is treated as enough

if response.stop_reason == "tool_use":
    if tool_name in CONSEQUENTIAL_TOOLS and not user_has_permission(current_user, tool_name):
        result = "Not authorized for this action."
    else:
        result = execute_tool(tool_name, tool_args)

Cost: a consequential action executing purely because the model chose to call it has no permission check independent of the model's own (non-deterministic) judgment — that judgment is exactly what a permission system exists to not have to trust blindly.

Mistake 11: fully automating a destructive agent action with no human confirmation (part 18)

if tool_name == "issue_refund":
    execute_refund(tool_args["order_id"], tool_args["amount"])  # executes immediately

if tool_name == "issue_refund":
    pending = PendingAction.objects.create(action="issue_refund", args=tool_args)
    notify_human_for_approval(pending)

Cost: an incorrect cancellation or refund that executes immediately leaves no chance for a human to catch the mistake before it's real. Part 18 covers gating consequential actions properly.

Mistake 12: judging system quality from a handful of anecdotal, hand-picked examples (part 19)

# "I tried 5 questions and they all looked right" — ships on that basis

eval_set = load_eval_set("rag_eval_150_questions.jsonl")  # a real, representative set with expected answers
results = [score_answer(q, expected, run_query(q)) for q, expected in eval_set]
print(f"Accuracy: {sum(results) / len(results):.1%}")

Cost: five hand-picked examples that look right produce a false sense of confidence with no real evidence about how the system performs across the actually varied input it will see in production. Part 19 covers building a real eval set.

The thread connecting all twelve

Nearly every mistake above comes from treating an advanced technique — embeddings, RAG, fine-tuning, agents — as a self-contained solution rather than one specific tool matched to one specific problem, requiring the same engineering discipline (access control, evaluation, safety limits) as any other production system.

FAQ

Which mistake here has the widest blast radius? Mistake 7 (prompt-only document access control) — a permissions gap at the retrieval layer means sensitive content can surface to the wrong user regardless of how carefully the rest of the prompt is written.

Is RAG always the right choice over fine-tuning? No — RAG fits knowledge that changes and needs to stay current; fine-tuning fits teaching a consistent format, tone, or behavior that doesn't depend on lookup. The capstone has the full decision framework.

How is this different from the AI agents mistakes page? This page covers the retrieval, embeddings, and fine-tuning layer — what to index, how to keep it current, and when each technique actually applies. Common Mistakes Building AI Agents covers a different layer: tool design, iteration limits, and multi-agent architecture. Mistakes 9-11 above touch agent territory specifically because RAG systems increasingly include an agent loop on top of retrieval.

Do I need a 150-question eval set before shipping anything? The number isn't the point — what matters is that the set is representative of real queries and has a defined correct answer to check against, so quality is measured rather than assumed. Start smaller and grow it as real failure cases turn up.

Next, and last: the capstone — a complete decision framework for choosing between prompting, RAG, fine-tuning, and agents for any new AI feature.