Letters become tokens.
Start with “The ”. Select each character and compare its learned vector of numbers.
A language model, taken apart
Inside the next token.
Build a small GPT. Follow a prediction through the numbers.
Every prediction below comes from real model weights.
Loading the model…Small enough to inspect.
01 / EXPERIMENT
Runs in your browserA little context. A new possibility.
7 characters
More predictableMore surprising
Each step samples with the selected seed; a full continuation advances the random generator.
CONTINUATIONREADY WHEN YOU ARE
One character at a time. See where it goes.
THE NEXT CHARACTERT = 0.80
What might come next?
A prediction is a distribution of possibilities. Sampling picks one.
Loading real model weights…
MOST LIKELY—
UNCERTAINTY—
Probabilities include your temperature and top-k settings. Higher uncertainty means more possible next characters.
TEXT → TOKENS → NUMBERS
Select a token to inspect its embedding.There’s a vector behind every character.
Your input tokens will appear here.
02 / UNDER THE SURFACE
The future is off limits. ↙Which characters get attention?
Predicting from ↓
Attending to →
Each row distributes one token’s attention across itself and earlier tokens.
03 / LEARNING, MEASURED
Read the model card Practice leaves a trace.
CROSS-ENTROPY LOSS ↓
TrainValidation
Loading recorded training measurements…
Lower loss means better next-character predictions. Validation uses held-out text the model did not train on.
LEARNABLE PARAMETERS—A decoder-only transformer
TRAINING STEPS—Measured from the actual run
YOUR TEXT / YOUR WEIGHTS
Learning happens here. ↙Teach it something of your own.
Choose plain text, train a small GPT from scratch, then explore its predictions and attention in the lab. Your text stays on this device. No account, API key, or server training.
400–200,000 characters, 2–256 distinct characters. The first 90% trains the model; the last 10% is held out for validation.
Inspect the character vocabulary
02 / WATCH IT LEARNFrom random guesses
From random guesses
to familiar patterns.
One transformer layer, four attention heads, 32 embedding dimensions, and a 32-character context. This is smaller than the published Shakespeare model.
Choose your text to begin. Keep this tab open while training.
TRAIN LOSS—
VALIDATION LOSS—
| Step | Train | Validation |
|---|
04 / FIELD NOTES
The full six-lesson course Learn by changing something.
Where letters live.
Inspect both e’s in “letter”. The same character shares a learned vector, even in different positions.
Context changes everything.
Try “The ”, then “The king ”. Watch how the same space leads to a different next character.
Take one head away.
Disable an attention head, then re-enable it. See whether the likely next characters change.
Follow the learning curve.
Compare untrained, early, and trained checkpoints. A falling loss is evidence of learning, not a promise of wisdom.
Turn up the temperature.
Keep the same prompt and seed. Compare low and high temperatures as the distribution changes.