Caching AI Responses to Cut Real Costs

Every project in this series makes a real API call, at real, per-token cost, on every single request — even when the exact same real question or input was already processed minutes earlier. This part adds genuine, practical caching to avoid that repeated, unnecessary cost.

The real, concrete waste this solves

Part 12's Q&A app: ten different real customers all ask "do you
  ship internationally" over the course of a day — that's TEN
  separate, real, billable API calls for the exact same real question
Part 10's translator: the same real product description gets
  translated to French repeatedly, once per real page view, even
  though the translation genuinely never changes

Neither of these repeated real calls produces any new value — the correct, real answer or translation is identical every time, making every repeat call genuinely wasted spend.

Real, practical in-memory caching

from functools import lru_cache

@lru_cache(maxsize=256)
def translate_cached(text: str, target_language: str) -> str:
    return translate(text, target_language)

Python's built-in lru_cache is a real, genuinely simple starting point — the first real call for a given (text, target_language) pair actually calls the API; every subsequent identical call returns the cached real result instantly, with zero additional API cost.

The real, honest limit of lru_cache: it's per-process, in-memory only

Real, genuine limitation: lru_cache's cache lives only in the
  running Python process's memory — restarting the app clears it
  entirely, and it's not shared across multiple real server processes
  or instances

For a genuinely small, single-process real application, this limitation may not matter. For a real, production deployment (covered in part 20) running multiple server processes, a shared, external cache is the honest, correct upgrade.

Real, practical caching with Redis for production

import redis
import json
import hashlib

cache = redis.Redis(host="localhost", port=6379, db=0)

def cache_key(prefix: str, *args) -> str:
    raw = prefix + "|".join(str(a) for a in args)
    return hashlib.sha256(raw.encode()).hexdigest()

def translate_with_redis_cache(text: str, target_language: str) -> str:
    key = cache_key("translate", text, target_language)
    cached = cache.get(key)
    if cached:
        return cached.decode("utf-8")

    result = translate(text, target_language)
    cache.set(key, result, ex=60 * 60 * 24 * 30)  # 30-day real expiry
    return result

Redis, a real, widely used in-memory data store, solves both of lru_cache's real gaps at once — the cache genuinely persists across restarts and is shared across every real server process, so a translation cached by one process is immediately available to every other one.

A real, deliberate expiry decision

ex=60 * 60 * 24 * 30  → cached entries expire after 30 real days

This is a real, practical, deliberate trade-off — genuinely permanent caching risks serving a real, stale translation if the source content changes without the cache being explicitly invalidated; a real, sensible expiry (or, better, actively clearing the cache when the source content is actually edited) balances real cost savings against real staleness risk.

What genuinely should NOT be cached

Bright Leaf Coffee's chatbot (part 2): each real conversation is
  contextually unique — caching a full conversation response makes
  no real sense, since the exact same input rarely recurs
GreenDesk's lead qualification (part 6): every real lead is distinct
  — there's no real, repeated identical input to cache against
Translation (part 10) and Q&A (part 12): GENUINELY good caching
  candidates — the same real input recurs often, and the correct
  output doesn't change between requests

This is a real, important distinction to make deliberately rather than caching everything indiscriminately — caching is valuable specifically where the same real input recurs and produces a stable, correct output; applying it to a genuinely unique, one-off request like a chatbot conversation turn provides no real benefit and adds unnecessary complexity.

Next: testing AI-powered features without burning real API credits — mocking LLM calls correctly in a real test suite.