How Neural Networks and Training Actually Work

Part 5 mentioned neural networks and parameters without fully explaining either. This part fills that in — a real, concrete, non-mathematical explanation of the mechanism underneath the transformer architecture and every LLM this series discusses.

A real, minimal neural network

Input layer:   real numbers representing the input (for text, this
                 comes from tokenization, covered fully in part 8)
Hidden layer:  a real set of interconnected nodes, each combining
                 inputs from the previous layer using adjustable
                 REAL weights, then applying a simple mathematical
                 function
Output layer:  the real, final prediction — for a language model,
                 a probability for every possible next token

A neural network is genuinely just layers of simple, real mathematical operations, chained together — each node takes inputs from the previous layer, multiplies each by a real, adjustable "weight," sums them, and passes the result through a simple function. Nothing about any single node is complicated; the real capability comes from stacking enormous numbers of these simple operations together.

What a "parameter" actually is, concretely

Each connection between two nodes has one real, adjustable number —
  its weight
A model with "70 billion parameters" has roughly 70 billion of these
  real, individual adjustable numbers, spread across its many layers

This is the real, literal meaning behind the parameter counts mentioned when comparing models — not a marketing number, but a genuine count of the individual, adjustable weights the network's real behavior is built from. Training (covered next) is specifically the real process of finding good values for every one of these numbers.

Training: real, repeated, gradual adjustment

1. Show the model a real piece of training text with the next real
   token hidden
2. The model, with its CURRENT weights, predicts a next token
3. Compare that real prediction against the actual, real next token
   that was hidden
4. Adjust every real weight slightly, in the direction that would
   have made the correct prediction slightly more likely
5. Repeat this real process billions of times, across a massive
   real training dataset

This real, iterative process — called gradient descent, applied through a mechanism called backpropagation — is genuinely how a model's parameters go from random, meaningless starting values to values that produce real, coherent, useful predictions. No single one of these billions of small, real adjustments does much on its own; the real capability emerges from the sheer, cumulative scale of the process.

Pretraining vs. fine-tuning: two real, distinct training phases

Pretraining: the massive real phase from the process above, using a
  huge, general real text dataset — this is what makes a model
  broadly capable at language in general
Fine-tuning: a real, smaller, more targeted follow-up training phase,
  often using human feedback (RLHF — Reinforcement Learning from
  Human Feedback) to shape the model toward being genuinely helpful,
  honest, and safe in conversation, rather than just fluent

A raw, pretrained model (before fine-tuning) tends to produce real but often unhelpfully-shaped text — continuing a prompt in plausible ways without necessarily following an instruction or staying on-topic the way a real, fine-tuned assistant does. This real, second phase is specifically what turns a raw next-token predictor into something that behaves like a genuinely helpful assistant — directly relevant to part 11's coverage of system, user, and assistant roles.

Why this matters even though a developer never trains a model directly

This series' real coding project (parts 16-18) never involves
  training — but understanding this mechanism explains WHY the
  model behaves the way it does: fluent but occasionally wrong (part
  13), frozen in time (part 5), and shaped by its fine-tuning to
  follow instructions in a specific real format (part 11)

Every practical behavior covered later in this series — hallucination, the training cutoff, following a system prompt — traces back directly to this real training mechanism. Understanding it, even without ever performing it yourself, is what turns those later behaviors from confusing quirks into genuinely predictable, explainable consequences of how the model actually works.

Next: how ChatGPT specifically generates answers — tracing this general mechanism through one real, specific, familiar product.