- Foundation
- About 16 min
- 5 practice questions
Neural networks and deep learning: neurons, layers and weights
Understand how an artificial neuron calculates, how layers form a network, how training adjusts weights, and what CNNs and transformers are used for.
If something looks wrong, please tell us.
Neural networks are behind face unlock, voice assistants, translation apps and AI chatbots. The name loosely compares them with brain cells, but an artificial neural network is really many simple calculations joined together. Deep learning means using neural networks with many layers.
The artificial neuron
A single artificial neuron does three things. It takes some input numbers, multiplies each by a weight (how important that input is), and adds them up together with a bias (a starting offset). It then passes the total through an activation function, which decides the neuron's output. A common activation, called ReLU, outputs the total if it is positive and 0 otherwise.
Inside one artificial neuron
- Inputs Some input numbers.
- Multiply by weights Each weight says how important that input is.
- Add the bias The weighted inputs are added up with a starting offset.
- Activation function Decides the output; ReLU outputs the total if positive, else 0.
Worked example 1. A neuron has inputs 2 and 3, weights 0.5 and 1, and bias −1. Weighted sum = (2 × 0.5) + (3 × 1) + (−1) = 1 + 3 − 1 = 3. ReLU of 3 is 3, so the output is 3. If the sum had been −2, ReLU would output 0. The activation function matters: without it, stacking many layers would be no more powerful than a single layer, because combining straight-line calculations only gives another straight-line calculation.
Layers
- Input layer: receives the features, for example the brightness of each pixel in a photo.
- Hidden layers: layers of neurons between input and output; each takes the previous layer's outputs as its inputs and builds more useful patterns.
- Output layer: gives the final answer, such as a score for each possible class.
- A network with many hidden layers is called deep; that is where the name deep learning comes from.
The weights and biases are the network's parameters, the numbers it learns. In a layer where every input connects to every neuron, the count is easy: 3 inputs feeding 4 neurons need 3 × 4 = 12 weights plus 4 biases, giving 16 parameters. Large modern networks have millions or billions of such numbers.
How training adjusts the weights
How training adjusts the weights
- Forward pass Feed in a training example and compute the output.
- Measure the error A loss function says how far the output is from the correct label.
- Backpropagation Work backwards to find how much each weight contributed to the error.
- Gradient descent Nudge every weight a little to reduce the error, by the learning rate.
Then back to step 1: Forward pass
- Start with small random weights.
- Feed in a training example and compute the output (the forward pass).
- Measure the error: how far the output is from the correct label, using a loss function.
- Work backwards through the network to find how much each weight contributed to the error; this is called backpropagation.
- Nudge every weight a little in the direction that reduces the error; this is gradient descent, and the size of the nudge is set by the learning rate.
- Repeat over the whole training set many times; one full pass is called an epoch.
Worked example 2. A one-weight model predicts output = w × input. Start with w = 1, input 2, correct answer 6, learning rate 0.1. Prediction = 1 × 2 = 2, so the error (answer minus prediction) is 6 − 2 = 4. Update rule: new w = w + learning rate × error × input = 1 + 0.1 × 4 × 2 = 1.8. New prediction = 1.8 × 2 = 3.6, closer to 6. Next step: error = 2.4, new w = 1.8 + 0.1 × 2.4 × 2 = 2.28, prediction 4.56. Each step moves w towards 3, the value that gives exactly 6. Real networks do the same for every weight at once.
Neuron: multiply inputs by weights, add the bias, apply the activation. Training: forward pass, measure error, backpropagate, adjust weights, repeat.
Why big data and GPUs mattered
The basic ideas of neural networks are decades old, but deep networks became practical only when two things came together. First, large datasets of photos, speech and text became available, and deep networks with many parameters need many examples to learn well instead of memorising. Second, GPUs (graphics processing units), first designed to draw video-game graphics, turned out to be very good at doing thousands of simple multiplications in parallel, which is exactly what training a network needs. Better training methods added to the gains.
CNNs for images
A convolutional neural network (CNN) is designed for images. Instead of connecting every pixel to every neuron, it slides small filters across the image, each filter looking for one local pattern such as an edge. Early layers find edges and colours, middle layers combine them into shapes such as eyes or wheels, and later layers recognise whole objects. Because the same filter is reused across the picture, a CNN needs far fewer weights and can spot a pattern wherever it appears. CNNs are used for face recognition, reading number plates and finding disease in crop or medical images.
Transformers for language
A transformer is the network design behind most modern language tools, including chatbots and translation apps. Its key idea is attention: when processing a word, the model looks at all the other words in the passage and weighs how relevant each one is. In 'The bank of the river was muddy', attention to 'river' tells the model that 'bank' means a riverside, not a place for money. Transformers handle all the words of a passage in parallel, which suits GPUs. Large language models are transformers trained on huge amounts of text to predict the next piece of text.
Key takeaways
- A neuron = weighted sum + bias + activation; weights and biases are the learned parameters.
- Training uses backpropagation and gradient descent to reduce the error step by step.
- CNNs suit images (filters); transformers suit language (attention).
Check your understanding
-
1. A neuron has inputs 4 and 1, weights 0.5 and 2, and bias 1, with ReLU activation. What is its output?
- 3
- 4
- 7
- 5
Show answer
Answer: D. Weighted sum = (4 × 0.5) + (1 × 2) + 1 = 2 + 2 + 1 = 5. ReLU of a positive number is the number itself, so the output is 5.
-
2. What does backpropagation do during training?
- It collects more training data
- It works out how much each weight contributed to the error so the weights can be adjusted
- It splits data into training and test sets
- It draws the network as a picture
Show answer
Answer: B. Backpropagation passes the error backwards through the layers to find each weight's share of the blame. Gradient descent then nudges the weights to reduce the error.
-
3. Why did GPUs help deep learning grow?
- They can do many simple calculations in parallel, which training needs
- They store more labels than other chips
- They remove the need for training data
- They make networks need no weights
Show answer
Answer: A. Training a network is mostly a huge number of multiplications that can run side by side. GPUs, built for graphics, do this kind of parallel work very well. Data and weights are still needed.
-
4. Which network design is most associated with images and uses small filters that slide across the picture?
- Decision tree
- Transformer
- Convolutional neural network
- k-means
Show answer
Answer: C. A convolutional neural network (CNN) slides filters over an image to detect local patterns such as edges and shapes. Transformers are associated with language and attention; decision trees and k-means are not neural networks.
-
5. A fully connected layer has 5 inputs and 3 neurons, each neuron with one bias. How many parameters does the layer have?
- 8
- 15
- 16
- 18
Show answer
Answer: D. Weights = 5 × 3 = 15. Biases = 3. Parameters = 15 + 3 = 18.
Sign in to keep track of the lessons you finish.