How does a model learn?
A neural network is a huge collection of numbers called weights. At the start they are random and the network's answers are nonsense. "Learning" means finding values for those weights that make the answers good.
There are billions of weights, so trying combinations is hopeless. The method that works is surprisingly simple: take many small steps downhill.
Measuring how wrong: the loss
First we need one number that says how badly the model is doing. Show it examples with known answers, compare its answers with the right ones, and combine the errors into a single score. That score is the loss. High means bad. Zero would be perfect.
Now picture just one weight. For every value it could have, there is a loss. Plot that and you get a curve: a valley, with the best value at the bottom.
Learning is the problem of getting to the bottom of the valley.
Feeling the slope
The model cannot see the whole curve. Calculating it would take for ever. But at the point where it currently stands, it can work out the slope: is the ground rising to the right or to the left, and how steeply?
That is enough to know which way is down.
Work out the slope at the current position
This is the gradient.
Take a step in the opposite direction
If the ground rises to the right, step left.
Make the step proportional to the slope
Steep ground: far from the bottom, take a big step. Nearly flat: close, take a small one.
Repeat
Each step lowers the loss a little.
In one line:
weight = weight - learning_rate * slopeThat is gradient descent. "Gradient" is the slope. "Descent" is going down.
The learning rate decides everything
The learning_rate is how big a step to take for a given slope. It is the most important setting in training, and it has to be chosen by the person running it.
Three learners start at the same spot on the same curve. The only difference is the learning rate.
Start. The curve is the loss: how wrong the model is for each setting of one weight. Lower is better, and the bottom is at 3. All three learners start at the same place and cannot see the curve. Each only knows the slope where it stands.
- Too small (0.02)
- loss 8.29
- About right (0.3)
- loss 8.29
- Too large (1.05)
- loss 8.29
- Too small (0.02): every step is in the right direction and almost nothing happens. It would get there eventually, after far more steps than anyone wants to pay for.
- About right (0.3): big strides at first, smaller ones as the ground flattens, and it settles at the bottom within a dozen steps.
- Too large (1.05): the first step jumps clean across the valley and lands higher up the other side. From there the slope is steeper, so the next step is bigger still. It bounces out and the loss explodes. This is called diverging.
Example
One step, worked through
The loss here is (weight − 3)² + 1, so the slope is 2 × (weight − 3).
At weight 0.3, the slope is 2 × (0.3 − 3) = −5.4. Negative means the ground falls to the right.
With learning rate 0.3: new weight = 0.3 − 0.3 × (−5.4) = 1.92. Closer to 3.
With learning rate 1.05: new weight = 0.3 − 1.05 × (−5.4) = 5.97. That is past 3 and further from it than where it started.
From one weight to billions
The demo has one weight, so the loss is a curve. With two weights it is a landscape of hills and valleys. With billions it cannot be pictured, but nothing about the method changes:
- work out the slope for every weight at once,
- nudge every weight a little, each against its own slope,
- repeat.
Working out all those slopes efficiently is the job of an algorithm called backpropagation. It starts from the error at the output and works backwards through the layers, calculating how much each weight contributed.
In practice
- Small batches. Computing the loss over all the training data for every step is too slow. Instead each step uses a small random handful of examples. The direction is a bit noisy, but the steps are thousands of times cheaper. This is stochastic gradient descent.
- A changing learning rate. Training usually starts with larger steps and shrinks them over time: stride at first, then tread carefully near the bottom.
- Not one valley. A real loss landscape has many dips. Gradient descent finds a good one, with no promise that it is the best one. In practice that turns out to be enough.
Key takeaways
- The loss is one number saying how wrong the model is. Training tries to make it small.
- Gradient descent works out the slope of the loss and steps the other way.
- The update is: weight = weight − learning rate × slope.
- Too small a learning rate crawls. Too large overshoots and diverges.
- Real training does the same for billions of weights at once, using small batches of examples.