Pointer Walk

Gradient Descent

A model learns by rolling downhill. Three learning rates: one crawls, one bounces out of the valley, one settles.

AI and Machine LearningIntermediate9 min read

How does a model learn?

A neural network is a huge collection of numbers called weights. At the start they are random and the network's answers are nonsense. "Learning" means finding values for those weights that make the answers good.

There are billions of weights, so trying combinations is hopeless. The method that works is surprisingly simple: take many small steps downhill.

Measuring how wrong: the loss

First we need one number that says how badly the model is doing. Show it examples with known answers, compare its answers with the right ones, and combine the errors into a single score. That score is the loss. High means bad. Zero would be perfect.

Now picture just one weight. For every value it could have, there is a loss. Plot that and you get a curve: a valley, with the best value at the bottom.

Learning is the problem of getting to the bottom of the valley.

Feeling the slope

The model cannot see the whole curve. Calculating it would take for ever. But at the point where it currently stands, it can work out the slope: is the ground rising to the right or to the left, and how steeply?

That is enough to know which way is down.

  1. Work out the slope at the current position

    This is the gradient.

  2. Take a step in the opposite direction

    If the ground rises to the right, step left.

  3. Make the step proportional to the slope

    Steep ground: far from the bottom, take a big step. Nearly flat: close, take a small one.

  4. Repeat

    Each step lowers the loss a little.

In one line:

python
weight = weight - learning_rate * slope

That is gradient descent. "Gradient" is the slope. "Descent" is going down.

The learning rate decides everything

The learning_rate is how big a step to take for a given slope. It is the most important setting in training, and it has to be chosen by the person running it.

Three learners start at the same spot on the same curve. The only difference is the learning rate.

One curve, three learning ratesInteractive
0510036the weightlossToo smallAbout rightToo large

Start. The curve is the loss: how wrong the model is for each setting of one weight. Lower is better, and the bottom is at 3. All three learners start at the same place and cannot see the curve. Each only knows the slope where it stands.

Too small (0.02)
loss 8.29
About right (0.3)
loss 8.29
Too large (1.05)
loss 8.29
  • Too small (0.02): every step is in the right direction and almost nothing happens. It would get there eventually, after far more steps than anyone wants to pay for.
  • About right (0.3): big strides at first, smaller ones as the ground flattens, and it settles at the bottom within a dozen steps.
  • Too large (1.05): the first step jumps clean across the valley and lands higher up the other side. From there the slope is steeper, so the next step is bigger still. It bounces out and the loss explodes. This is called diverging.

Example

One step, worked through

The loss here is (weight − 3)² + 1, so the slope is 2 × (weight − 3).

At weight 0.3, the slope is 2 × (0.3 − 3) = −5.4. Negative means the ground falls to the right.

With learning rate 0.3: new weight = 0.3 − 0.3 × (−5.4) = 1.92. Closer to 3.

With learning rate 1.05: new weight = 0.3 − 1.05 × (−5.4) = 5.97. That is past 3 and further from it than where it started.

From one weight to billions

The demo has one weight, so the loss is a curve. With two weights it is a landscape of hills and valleys. With billions it cannot be pictured, but nothing about the method changes:

  • work out the slope for every weight at once,
  • nudge every weight a little, each against its own slope,
  • repeat.

Working out all those slopes efficiently is the job of an algorithm called backpropagation. It starts from the error at the output and works backwards through the layers, calculating how much each weight contributed.

In practice

  • Small batches. Computing the loss over all the training data for every step is too slow. Instead each step uses a small random handful of examples. The direction is a bit noisy, but the steps are thousands of times cheaper. This is stochastic gradient descent.
  • A changing learning rate. Training usually starts with larger steps and shrinks them over time: stride at first, then tread carefully near the bottom.
  • Not one valley. A real loss landscape has many dips. Gradient descent finds a good one, with no promise that it is the best one. In practice that turns out to be enough.

Key takeaways

  • The loss is one number saying how wrong the model is. Training tries to make it small.
  • Gradient descent works out the slope of the loss and steps the other way.
  • The update is: weight = weight − learning rate × slope.
  • Too small a learning rate crawls. Too large overshoots and diverges.
  • Real training does the same for billions of weights at once, using small batches of examples.

Your progress

Check your progress

0%

  1. Read the page50% of this page

    When you have gone through the page and its animations, tick it off.

  2. Take the quiz50% of this page

    7 questions, from easy to hard. Take it as often as you like: the questions change, and only your best score counts.

Saved on this device only. See all your progress