Context Windows: What They Are and Why They Run Out

Part 8 covered tokens as the real unit a model operates on. This part covers the real, hard limit on how many of them fit in a single request — a constraint that directly shapes how every real application in this series, including the one built starting in part 16, has to be designed.

A real, precise definition

A context window is the maximum real number of tokens — input plus
  output combined — a model can process in a single real request

Real, illustrative example: a model with a 200,000-token context
  window can handle a real system prompt, conversation history, and
  generated response that together total up to 200,000 tokens — not
  per message, but across the ENTIRE real request

This is a real, hard, technical limit tied directly to the transformer architecture from part 5 — not an arbitrary business restriction, but a genuine, real computational constraint on how much text the model's attention mechanism can process at once.

What genuinely happens when a real conversation exceeds it

A real, long-running conversation with Bright Leaf Coffee's support
  assistant (this series' project, starting part 16): every new
  real message adds to the token count of the FULL conversation
  history sent with each new request

Message 1: 200 real tokens
Message 5: conversation history now totals 3,000 real tokens
Message 50: conversation history now totals 40,000 real tokens
...eventually exceeding the real context window entirely

Once a real request's total token count exceeds the model's context window, the API call genuinely fails outright with an error — the request is rejected, not silently truncated or gracefully degraded. This is a real, practical failure mode any application handling long conversations has to explicitly plan for, not an edge case to ignore.

Real, practical strategies for managing a growing context

1. Truncation: drop the OLDEST real messages once a token budget is
   approached, keeping only the most recent real exchanges
2. Summarization: periodically replace older real conversation turns
   with a shorter, real AI-generated summary of what was discussed
3. Sliding window: keep only the last N real messages, discarding
   anything older, regardless of total token count

Each of these real strategies trades off differently — truncation is simple but genuinely loses real, potentially relevant early context; summarization preserves more real meaning at a lower token cost but adds real complexity and its own real API call; a sliding window is the simplest to implement but the least precise about what real context actually gets kept.

Context window size varies genuinely by model

Real, illustrative range across different models and providers:
  Smaller/older models: as low as a few thousand real tokens
  Modern, large-context models: several hundred thousand real
    tokens, sometimes over a million

This real variation directly affects application design choices covered later in this series — a model with a genuinely large context window can afford to include more of a real product catalog directly in the prompt (a simpler, real approach), while a smaller context window makes part 19's RAG technique (retrieving only the specific, relevant information needed) a genuine practical necessity rather than just an optimization.

A real, important distinction: context window vs. real "memory"

A model has NO real memory between separate API requests by default
  — the "memory" a chat application appears to have (part 18) is
  entirely the application re-sending the full real conversation
  history with every new request, not the model genuinely
  remembering anything on its own

This is a genuinely common, real point of confusion worth being precise about: a language model itself is stateless between requests — any real sense of ongoing memory in a product like ChatGPT or Bright Leaf Coffee's support assistant is an application-layer illusion, built by resending real context, not a genuine capability of the underlying model itself.

Next: what prompt engineering actually is — the real, practical skill of shaping what goes into that limited context window to get genuinely better, more reliable results.