Colophon

How the numbers are computed

Every value in the figures comes from code running in your browser. There are no pre-rendered animations and no hand-picked numbers standing in for real ones.

Chapters 1 to 3 start from fixed inputs and weights and compute everything else from them. The three extra rows of W in chapter 3 come from a seeded random generator (mulberry32, seed 3).

Chapter 4 trains an MLP with two inputs, two hidden layers of 8 tanh units and a sigmoid output (105 parameters) on 200 two-moons points with noise 0.2. The data seed is 7 and the initialization seed is 1007. The forward pass, backprop, SGD and Adam are about 200 lines of TypeScript with no ML library, and the gradients are tested against finite differences.

The Llama 3.1 8B sizes (hidden size 4,096, MLP size 14,336, 32 layers) come from its published config.json. The total of 8.03 billion parameters comes from the model's safetensors metadata.

Who this learned from

• Bartosz Ciechanowski, for figures you can grab.
• Brendan Bycroft's LLM visualization, for showing a real model's actual numbers.
• Transformer Explainer from the Polo Club at Georgia Tech, for running a real model on your own text.
• Sam Rose, for small figures that each teach one idea.
• 3Blue1Brown, for transforming one object instead of cutting to a new one.
• TensorFlow Playground, for training a network live in the page.

Type and tools

Set in Archivo and Recursive. Built with Next.js, Motion and Lenis. Every figure is plain TypeScript.