What Are Tokens in AI?

Every part so far has mentioned tokens without fully defining them. This part covers them precisely — genuinely necessary before parts 9, 14, and 16 through 18, since tokens directly determine real cost, real context limits, and real API request structure.

A real, precise definition

A token is the actual, smallest unit of text a language model reads
  and generates — NOT the same thing as a word

"Bright Leaf Coffee roasts weekly"
  → real tokens (approximate, actual tokenization varies by model):
    ["Bright", " Leaf", " Coffee", " roast", "s", " weekly"]

Tokens are often whole words, but genuinely not always — "roasts" split into "roast" and "s" above is a real, common pattern, since tokenizers are built from statistical patterns across massive real training text, not a real dictionary of whole words. A rough, practical real rule of thumb: one token is approximately 4 characters of English text, or roughly ¾ of a word on average — a genuinely useful estimate, not an exact rule.

Why tokens exist instead of the model reading raw characters or words

Character-level: genuinely too granular — a real sentence becomes
  an extremely long sequence, making the model's job harder and
  slower for no real benefit
Word-level: genuinely too rigid — a fixed real vocabulary can't
  handle new words, typos, or the many real languages a model needs
  to support without an impractically enormous vocabulary
Token-level (subword): a real, practical middle ground — common
  whole words stay as single tokens, while rarer or unfamiliar words
  break into smaller, real, reusable pieces

This subword approach is genuinely why a model can handle a word it's never seen as one whole unit before — a brand name, an unusual real technical term — by breaking it into smaller, familiar real token pieces it has encountered before, rather than failing outright on unknown vocabulary.

Tokens directly determine real, actual API cost

Real, typical API pricing structure:
  Input tokens:  a real cost per token for what you SEND to the model
  Output tokens: a real, typically higher cost per token for what
                   the model GENERATES back

This is a real, direct, practical consequence for any application built later in this series — a longer real system prompt (part 11), a longer real conversation history, or a longer generated answer all directly, measurably increase real cost, since billing is based on genuine token counts, not a flat per-request fee. Part 15's coverage of reading real API documentation covers exactly where these real per-token prices are published.

Tokenization is real, but not identical across every model

The same real sentence can tokenize into a genuinely different
  number of tokens depending on which model's specific tokenizer
  processes it — there's no single, universal tokenization standard
  across every LLM provider

This is a real, practical detail worth knowing rather than assuming: a token-count estimate from one provider's tokenizer isn't necessarily accurate for another provider's model, which matters directly when estimating real cost or checking against a real context window limit (covered fully in the next part) for a specific model.

Checking real, actual token counts directly

# a real, practical way to check token counts before sending a
# request — most providers publish an official tokenizer library
# for exactly this purpose, rather than relying on the rough
# 4-characters-per-token estimate above

Relying on the rough character-based estimate is fine for a quick, real mental check, but a genuinely precise token count — needed to avoid a real, hard context-window error, covered next — requires actually running text through the specific model provider's real tokenizer, not estimating.

Next: context windows — what they actually are, why they're measured in the exact tokens covered in this part, and what genuinely happens when a real conversation runs out of room.