Keeping a Knowledge-Base Chatbot Up to Date
Part 5 flagged indexing and querying as genuinely separate phases. This part covers the real, practical problem that separation creates: what happens when GreenDesk's actual documents change after they've already been indexed.
The real, concrete staleness problem
Day 1: a real HR policy document is indexed — "vacation requests
need 2 weeks' notice"
Day 30: the real, actual policy changes to 1 week's notice, but the
document was never re-indexed
Day 31: an employee asks the chatbot, gets the STALE, now-incorrect
real answer, confidently and grounded-soundingThis is a genuinely serious, real risk specific to RAG systems — the grounding technique that prevents hallucination (per the AI Fundamentals series) provides zero real protection against confidently, accurately repeating outdated information, since from the model's perspective, the retrieved content simply IS the real, provided truth.
This is arguably a MORE dangerous real failure mode than a hallucination — a hallucinated answer might sound uncertain or get flagged by human review; a stale-but-grounded answer sounds exactly as confident and well-supported as a genuinely current one, since the retrieval and generation mechanism is functioning completely correctly. The problem is entirely upstream, in whether the index itself is current.
Real, practical change detection
import hashlib
import os
def get_file_hash(filepath: str) -> str:
with open(filepath, "rb") as f:
return hashlib.sha256(f.read()).hexdigest()
def find_changed_files(directory: str, known_hashes: dict) -> list[str]:
changed = []
for filename in os.listdir(directory):
filepath = os.path.join(directory, filename)
current_hash = get_file_hash(filepath)
if known_hashes.get(filename) != current_hash:
changed.append(filepath)
return changedThis real, practical technique — comparing a real, current file hash against the last-known hash — is what lets a re-indexing job process only genuinely changed documents, rather than wastefully re-embedding GreenDesk's entire, unchanged document set on every single run.
The real, complete re-indexing pipeline
def reindex_changed_documents(directory: str, known_hashes: dict) -> dict:
changed_files = find_changed_files(directory, known_hashes)
for filepath in changed_files:
filename = os.path.basename(filepath)
# remove the OLD, real, now-stale chunks for this document
collection.delete(where={"source_file": filename})
# index the new, current real content
text = extract_text_smart(filepath)
chunks = chunk_document(text)
collection.add(
documents=chunks,
ids=[f"{filename}-{i}" for i in range(len(chunks))],
metadatas=[{"source_file": filename} for _ in chunks],
)
known_hashes[filename] = get_file_hash(filepath)
return known_hashesThe explicit collection.delete() step before re-adding is genuinely important — without it, updating a document would leave the OLD, real, stale chunks in the index alongside the new ones, and retrieval could still surface the outdated real content even after "updating" it.
A real, practical scheduling decision
Running this pipeline on a real cron schedule (hourly, daily): simple,
genuinely reliable, but real changes take up to that interval to
actually reflect in the chatbot's answers
Running it on a real, direct trigger (a webhook when a document is
edited in GreenDesk's actual document system): near-immediate real
freshness, but genuinely more complex to wire up correctlyFor most real knowledge-base use cases, a scheduled real job (hourly, for GreenDesk's actual update frequency) is a genuinely reasonable, practical trade-off — instant freshness is rarely worth the real, added complexity of event-driven triggers, unless a specific real document type (an active incident runbook, say) has a genuine need for near-immediate propagation.
A real, honest safeguard: surfacing document recency to the user
def answer_with_recency_note(question: str) -> dict:
results = collection.query(query_texts=[question], n_results=3, include=["metadatas"])
# ...real, grounded generation as before
oldest_source_age = max(
(datetime.now() - m["last_indexed"]).days
for m in results["metadatas"][0]
)
if oldest_source_age > 90:
answer += "\n\n(Note: this may reference a policy older than 90 days — worth confirming with HR directly.)"
return {"answer": answer, "sources": results["ids"][0]}This real, honest, explicit caveat — surfaced directly to the employee when a retrieved document is genuinely old — is a practical, additional safeguard beyond the re-indexing pipeline alone, directly extending the same calibrated honesty the AI Fundamentals series applied to hallucination risk generally, now applied specifically to staleness risk.
Next: how AI agents actually work — moving beyond retrieval and generation into systems that can take real, independent action.