Start with one weight.

This is a weight, w = 0.42. All it does is multiply. The input x = 0.8 comes in and x · w = 0.336 goes out.

Drag the cell up or down, or focus it and use the arrow keys.

Scroll and w sweeps from −2 to 2. The edges get thicker as |w| grows, and the color shows the sign: blue below zero, orange above, paper at exactly zero.

Every weight on this site uses that scale, so a matrix you haven't seen before still reads at a glance.

Add a bias b = 0.10 and you have a line, y = wx + b. The weight sets the slope and the bias moves it up or down. Right now y = 0.436.

A neuron is this with more inputs and a bend at the end.

Three inputs, three weights. Multiply each pair, add the products, then add the bias. The result is the pre-activation z = 1.838.

In PyTorch that's x @ w + b. Matmuls like this account for nearly all the FLOPs in a transformer, which is why the rest of the page keeps coming back to them.

Stack layers of plain weighted sums and the stack collapses into one weighted sum. A nonlinearity in between stops that. With z = 1.838:

ReLU zeroes anything negative and gives 1.838. GELU gives 1.777 and SiLU gives 1.586. Both are smooth and let a little negative signal through. Drag the weights until z goes negative: ReLU drops to zero, while GELU and SiLU dip just below it.

Drop to two inputs and the neuron fits in a picture. Each point on the plane is an input (x₁, x₂), and its color is the neuron's output. The ink line is z = 0, where w1x1 + w2x2 + b crosses zero.

The weights turn the line and the bias slides it. One neuron draws one line. Chapter 4 wires 17 of them together.

4 neurons reading the same 3 inputs: 12 weights, 4 biases, 4 outputs. Written like this it's that many separate dot products.

The same 12 weights, rearranged. Each neuron's weights become one row of a matrix W, the inputs become a vector x, and the separate dot products become one matrix-vector product, y = Wx + b.

Scroll back and forth through this. Nothing is recomputed. They're the same numbers in a different arrangement.

Row by row: each output is one row of W dotted with x, plus that row's bias. The running sum is underneath. The first row is the neuron from chapter 2, so y₁ = 1.838, the same z as before.

A GPU computes every row at once, and does it for every token in the batch in the same pass. That's why most of the work in a transformer gets written as matmuls.

Our W is 4 × 3. One MLP up-projection in Llama 3.1 8B is 14,336 × 4,096, which is 58,720,256 weights.

Each of its 32 layers has three matrices that size (gate, up and down), so the MLPs hold 5.6 billion of the model's 8.03 billion parameters. Ours is the dot in the corner.

200 points in two interleaved classes, solid and hollow. No straight line separates them, so a single neuron can't.

This network can. It has 105 parameters: two inputs, two hidden layers of 8 tanh units, and one sigmoid output.

At its random initialization the network still has an opinion about every point on the plane, and it's no better than a coin flip. The color is its predicted probability that a point is hollow.

The loss is binary cross-entropy averaged over all the points: 0.725.

Training runs on its own, and the pause button under the figure stops it. Each step computes the gradient of the loss over every point and takes one Adam step at learning rate 0.01.

Step 0, loss 0.725. Each hidden unit contributes a line like the one in chapter 2, and the layers above bend and blend them into the boundary you're watching form. Reset starts again from a different random initialization.

One step, slowed down. The forward pass runs left to right and ends in a loss. Then the gradient runs right to left, and every weight learns which direction would lower the loss and by how much. During that phase the edges show gradients, not weights. Last, every weight moves a small step against its recent average gradient.

This one uses plain SGD, so the learning rate decides everything. It's at 1 now. Drag it.

Around 1 it trains. At 0.01 it crawls. At 10, within a few steps it overshoots so far that every unit saturates, the gradient goes to zero, and it never recovers.

Try it on your own dots in the playground.