Gradient descent, by hand

An interactive, scrollable explainer: watch gradient descent pick its way down a loss surface as you change the learning rate, the momentum, and where it starts.

Gradient descent, by hand

Almost every model you have ever trained was fit the same way: start somewhere, look at which way is downhill, take a step, repeat. That is gradient descent, and it is small enough to watch happen. On this page you can change the step size, add some momentum, drag the starting point around the slope, and see where the little ball rolls.

We will descend one awkward surface throughout: a long, narrow valley. It shows why the step size is hard to get right and why momentum was invented, and it is simple enough that we can write its slope down by hand in a moment.

Play with it first

In the picture below, the rings are contours of the loss, low in the middle and high at the edges. The dot is where descent starts, and the line is the path it takes. Drag the dot anywhere on the slope; nudge the three sliders. Push the step size high enough and watch the path stop converging and start ricocheting across the valley.

step size (η)0.12

momentum (β)0

steps25

The slope, written down

The surface you were just playing on has a simple formula. Writing $\theta = (x, y)$ for a point on it, the loss is

$L(\theta) = \tfrac{1}{2}\!\left(x^2 + 6\,y^2\right).$

The factor of six sets the shape of the problem. Move the same small distance away from the floor along $x$ or along $y$ , and the $y$ direction costs six times as much loss. The rings show the resulting long, narrow trough, gentle along its length and steep across it.

The direction of steepest ascent is the gradient, the vector of partial derivatives, which for this loss is short enough to read off by hand:

$\nabla L(\theta) = \left(\frac{\partial L}{\partial x},\ \frac{\partial L}{\partial y}\right) = (x,\ 6y).$

Gradient descent walks the other way. It takes a fixed fraction $\eta$ of the gradient, the step size (or learning rate), and subtracts it:

$\theta \;\leftarrow\; \theta - \eta\,\nabla L(\theta).$

Figure 1 draws one such step. At a point $\theta$ up on the slope, $-\nabla L$ points inward, toward the floor, and a step of length $\eta\lVert\nabla L\rVert$ carries you part of the way there.

Figure 1: The loss surface as contour rings: shallow along the valley floor (the

x

axis), steep across it (the

y

axis). At a point on the slope, the descent step

-\nabla L

points inward toward the minimum.

The same $\eta$ multiplies both components of the gradient, but the two components live on different scales: the $y$ component is $6y$ , the $x$ component only $x$ . A step size gentle enough to crawl along the shallow $x$ direction barely moves; make it big enough to move usefully and it can overshoot the steep $y$ walls entirely. The step size slider lets you feel that tension.

One step at a time

The playground shows the whole path at once. To see why it bends the way it does, step through the five scenes below: the picture redraws for each one, from the bare landscape to a full run and, finally, to a run that falls apart.

scene

The landscape. The rings are the loss. Descent begins at the marked point, high on the steep left wall of the valley. Everything that follows is a rule for getting from there to the dark point in the middle.

Which way is downhill. At the start point, the negative gradient $-\nabla L$ points almost straight across the valley, not along it. The steepest direction is rarely the direction of the minimum.

One step. Multiply that direction by the step size and move. With a modest $\eta$ the first step lands well down the wall but overshoots the floor slightly, ending up on the far side. The path has already started to zig-zag.

Many steps. Repeat, and the zig-zags shrink as the walls get shallower near the floor. The path settles into the trough and crawls the long way toward the minimum. It works, but notice how much of the effort went sideways.

Too far. Now push the step size past the edge. Each step across the valley is longer than the last; instead of settling, the path climbs the opposite wall higher every time and ricochets out of the frame. The rule and the surface are the same as before; only the step size is too large.

Choosing the step size

Watching the path is one way to judge a step size; watching the loss fall is another, and often clearer. The chart below plots the loss at each iteration for whatever the three sliders are set to right now. A healthy run drops fast and flattens near zero; a too-large one bottoms out early and then climbs, the curve turning back upward as the path escapes the valley.

The step-size ceiling

For a quadratic like this one, plain gradient descent converges only while $\eta < 2/\lambda_{\max}$ , where $\lambda_{\max}$ is the largest curvature, here the $6$ of the steep direction. That puts the ceiling at $\eta < \tfrac{1}{3}$ . Below it, smaller is safer but slower; above it, no amount of patience helps. Most of the art of training is keeping $\eta$ under a ceiling you cannot see directly.

Momentum

The slow part of every run above was the crawl along the valley floor, where the gradient is tiny and each step barely moves. Momentum fixes this by giving the ball inertia: instead of stepping by the gradient alone, it accumulates a velocity that carries over from step to step,

$v \leftarrow \beta\,v - \eta\,\nabla L(\theta), \qquad \theta \leftarrow \theta + v.$

The friction term $\beta$ (the momentum slider) decides how much of the past velocity survives. Along the shallow floor the gradient keeps pointing the same way, so the velocity builds and the ball rolls faster; across the steep walls the gradient keeps flipping sign, so those contributions cancel and the wobble is damped. Figure 2 shows both runs from the same start for the same number of steps: plain descent is still short of the minimum while the momentum run has arrived.

Figure 2: Two descent runs from the same start, fourteen steps each. Plain gradient descent (dashed) is still crawling along the floor; the momentum run (solid) has reached the minimum.

Scroll back to the playground and raise the momentum slider on a run that was converging slowly: the same $\eta$ now reaches the floor in far fewer steps. Push it too high, though, and momentum overshoots on its own, a third way to make the path unstable.

What to carry away

Summary

Steepest is not shortest. The negative gradient points across the valley, not at the minimum, so descent zig-zags. Curvature, not distance, sets the direction.
The step size has a ceiling. One number, shared across every parameter, has to be small enough for the steepest direction and is therefore usually too small for the rest. Cross the ceiling and the run diverges no matter what.
Momentum shortens the crawl. Accumulating velocity accelerates the shallow directions and cancels the wobble in the steep ones, which is why nearly every modern optimizer keeps some form of it.

Every trick that came later (adaptive rates, per-parameter scaling, learning-rate schedules) is a more careful answer to the questions you worked through by hand here: which way to step, and how far.