Gradient Descent Explained Simply With Two Updates

Blog › Machine Learning Explained

· by Julia Berens · 4 min read

Here is gradient descent explained simply: measure which way the loss goes up, then take a step in the opposite direction. In machine learning, loss is a number that measures how far a model's predictions are from what we want, according to a chosen rule. Gradient descent changes the model's adjustable numbers to try to lower that loss. With just one adjustable number, you can do the arithmetic by hand and see why a step that points downhill can still end too far away.

Gradient descent explained simply on a loss curve

Imagine a model that always predicts the same number, w. For this example, it has only one training target: 3. Use squared error as the loss, so the difference between the prediction and the target is squared:

L(w) = (w − 3)2

If the model predicts 5, the loss is 4. If it predicts 1, the loss is also 4. Both predictions are two units away from the target, and squaring makes both errors count the same. Plot the prediction on the horizontal axis and the loss on the vertical axis. You get a U-shaped curve with its lowest point at w = 3, where the loss is 0.

In this example, you can spot the best prediction from the formula. But gradient descent does not need to know where the bottom is before it starts. It uses the slope at its current position. The slope is the derivative of the loss with respect to w. For our curve, it is L′(w) = 2(w − 3). If you have not studied derivatives yet, you can use this formula as a slope calculator.

At w = 5, the slope is +4, so a small move to the right raises the loss. At w = 1, the slope is −4, so a small move to the right lowers it. That sign tells you which way to go, but it does not tell you how far to go.

Two updates at a learning rate of 0.25

The learning rate is a positive number that scales the slope to set the step size. The update rule is:

New prediction = current prediction − learning rate × current slope

Subtracting a positive slope moves the prediction left. Subtracting a negative slope moves it right. The size of the move is the learning rate times the magnitude of the slope. Start at w = 5 and keep the learning rate at 0.25 for both updates.

Update 1 moves from 5 to 4

The starting slope is 2(5 − 3) = 4. Multiply it by 0.25 to get 1, then subtract: 5 − 1 = 4. The model now predicts 4 instead of 5. Its loss drops from (5 − 3)2 = 4 to (4 − 3)2 = 1.

Update 2 moves from 4 to 3.5

Calculate the slope again at the new prediction. It is now 2(4 − 3) = 2. Multiply by 0.25 to get 0.5, then subtract: 4 − 0.5 = 3.5. The new loss is (3.5 − 3)2 = 0.25.

The first step moved the prediction by 1. The second moved it by only 0.5, even though the learning rate stayed the same. The curve is flatter at 4 than at 5, so the slope is smaller there. If you reused the first slope instead of recalculating it, you would move another full unit. That happens to land at the minimum in this example, but it is not the second gradient descent update.

How the learning rate changes each step

Restart at 5 for every row in the table. The first slope is always 4, so changing the learning rate directly changes the length of that first step. For the second update, calculate the slope at the prediction where the first step ended.

Learning rateFirst moveSecond moveLoss after two updates
0.1Left 0.4 to 4.6Left 0.32 to 4.281.6384
0.25Left 1 to 4Left 0.5 to 3.50.25
0.75Left 3 to 2Right 1.5 to 3.50.25

With a learning rate of 0.1, the loss goes down on both updates, but the prediction is still farther from 3 than it is after two updates at 0.25. With a rate of 0.75, the first step crosses the minimum. Now the slope at 2 is −2, so the second update subtracts a negative number: 2 − 0.75 × (−2) = 3.5. Crossing the minimum is not automatically a problem. The prediction of 2 is closer to 3 than the starting prediction of 5 was, so the first step still lowers the loss.

When a downhill step raises the loss

Try a learning rate of 1.25 from the same starting point. The slope at 5 still points uphill to the right, so the update goes left. But the step is 1.25 × 4 = 5 units long. It takes the prediction all the way to 0, where the loss is 9, higher than the starting loss of 4.

The slope told you what happens near 5. It did not promise that the curve would keep falling for the full length of the step. A real model can have many adjustable numbers. Its gradient collects the slopes for those numbers, and the same idea applies: subtract the learning rate times the gradient. A rate that works well for our one-number model need not work well for another loss function. Google's Machine Learning Crash Course lesson on learning rate shows how this choice affects training.

Try an update from the other side

Start at w = 1 this time, with a learning rate of 0.25. Before doing the arithmetic, predict the direction of the first move. Then compute the slope and loss for each update. If you get 2 and then 2.5, compare their losses with those at 4 and 3.5. The predictions are on opposite sides of 3, but their distances from 3 match.

If you want more examples of how models learn, browse Machine Learning Explained. The International Machine Learning Competition (IMLC) also has training and practice problems for working through machine learning ideas on paper.

Frequently asked questions

Is the learning rate the same as the step size?
No. In this example, the step size is the learning rate multiplied by the magnitude of the current slope. Even with a fixed learning rate, the step size can change.

Can gradient descent cross the minimum and still lower the loss?
Yes. On this curve, moving from 5 to 2 crosses the minimum at 3 but lowers the loss from 4 to 1.

What happens when the slope is zero?
The basic update makes no move because the learning rate times zero is zero. On this curve that happens at the minimum, but a zero slope does not always mark a minimum.

About the International Machine Learning Competition: an online machine learning competition for high school students from around the world. Qualification Round 2026: submissions are open until Sunday, 13 December 2026. How the rounds work

Written by Julia Berens, Social Media Assistant of the IMLC team.

More articles

All articles