Pointer Walk

Inside One Neuron

Multiply, add, squash. Change one weight and the same inputs give the opposite answer.

AI and Machine LearningBeginner7 min read

The smallest part of a neural network

A neural network with billions of settings sounds impossibly complicated. It is built from one very simple part, repeated an enormous number of times. That part is the artificial neuron, and it does three things:

multiply, add, squash.

A neuron makes one small decision

Take an everyday decision: should I go to the beach? Three things feed into it.

  1. Multiply: each input is scaled by a weight

    An input is a number: here 1 for yes, 0 for no. Its weight says how much it matters and in which direction. Sunshine is good (+2). Rain is very bad (−3).

  2. Add: sum everything, plus a bias

    The bias is a fixed number that shifts the result. A negative bias means "by default, stay home unless persuaded".

  3. Squash: turn the sum into a number between 0 and 1

    Big positive sums become nearly 1, big negative sums nearly 0. Read it as "how strongly yes".

Try it. Switch inputs on and off, then drag the weights.

One neuron: go to the beach?Interactive

1Multiply each input by its weight

1 × 2 = 2
1 × 1 = 1
0 × -3 = 0

2Add them up, plus a fixed number called the bias

2 + 1 + 0 + (-1) = 2.0

3Squash the sum into a number between 0 and 1

0 = no0.88 → Go to the beach1 = yes

Sum 2.0, squashed to 0.88: go. Keep the inputs as they are and drag one weight until the answer flips. Nothing about the world changed: only how much the neuron cares about it.

Example

The starting numbers

Sunny (1 × 2) + weekend (1 × 1) + not raining (0 × −3) = 3. Add the bias of −1 to get 2. Squashed, that is 0.88: go.

Now turn on rain. The sum becomes 2 + 1 − 3 − 1 = −1, which squashes to 0.27: stay home.

The weights are the knowledge

Here is the experiment that matters. Leave the inputs alone: sunny, weekend, no rain. Now drag the weight for "sunny" down until it is negative.

The answer flips to "stay home", on a sunny weekend.

The facts about the world did not change. What changed is how the neuron weighs them. Perhaps this neuron belongs to someone who burns easily.

That is the central idea of the whole field: a neuron's behaviour is entirely decided by its weights and bias. They are just numbers. Change the numbers and you have a different decision-maker.

python
import math

def neuron(inputs, weights, bias):
    total = sum(x * w for x, w in zip(inputs, weights)) + bias    # multiply, add
    return 1 / (1 + math.exp(-total))                             # squash

neuron([1, 1, 0], [2, 1, -3], -1)    # 0.88

Why the squash matters

Without the last step, a neuron only multiplies and adds. Stack a thousand of those and the whole stack can still be rewritten as one multiply-and-add. It could never learn anything more complicated than a straight line.

The squash, properly called the activation function, bends the output. With it, layers of neurons can build up curves, corners and, eventually, very intricate patterns. The S-shaped one used here is the sigmoid. Modern networks mostly use simpler ones, but the purpose is the same.

From one neuron to a network

One neuron draws a single line between yes and no. It cannot learn "go if it is sunny or the weekend, but not both".

  1. Put many neurons side by side

    That is a layer. Each looks at the same inputs with its own weights and notices something different.

  2. Feed their outputs into another layer

    The second layer's inputs are the first layer's answers. It combines simple findings into more complex ones.

  3. Repeat

    Early layers find simple things. Later layers combine them. In an image model: edges, then shapes, then faces.

A large language model has billions of weights. Every one of them plays the same role as the three sliders above.

Where do the weights come from?

Nobody sets them by hand. They start as random numbers, which makes the network useless. Then it is shown examples with known answers, and after each one the weights are nudged slightly in whichever direction would have made the answer less wrong. Repeat that billions of times.

That nudging process is gradient descent.

Key takeaways

  • A neuron multiplies each input by a weight, adds them up with a bias, and squashes the result.
  • Weights say how much each input matters, and in which direction.
  • The same inputs give a different answer when the weights change: the weights are what the neuron knows.
  • The squashing step lets layers of neurons learn patterns more complex than a straight line.
  • A network is many neurons in layers, and learning means adjusting all their weights.

Your progress

Check your progress

0%

  1. Read the page50% of this page

    When you have gone through the page and its animations, tick it off.

  2. Take the quiz50% of this page

    6 questions, from easy to hard. Take it as often as you like: the questions change, and only your best score counts.

Saved on this device only. See all your progress