Gradient Descent in Machine Learning: Intuition, Math & Code

Most explanations of gradient descent pick a lane. The intuition-only pages give you a nice mountain metaphor and send you off with no idea how to actually implement anything. The math-heavy pages throw a wall of partial derivatives at you and assume you'll figure out why any of it matters. Neither approach really works if you're trying to get this to click before an interview, or before your next model just refuses to train properly.

This piece does all three layers with one running example: fitting a straight line to five data points. Same numbers, all the way from the plain-English version through to working NumPy code. If a number shows up in a table below, it was actually computed, not estimated.

What Is Gradient Descent in Machine Learning?

Gradient descent is an iterative optimization algorithm that minimizes a loss function by repeatedly moving a model's parameters in the direction that reduces error the fastest. That's the whole idea. Everything else in this article is detail on top of that one sentence.

It's not one specific technique buried inside machine learning, it's closer to the engine room. Linear regression uses it. Every neural network you've heard of uses some flavour of it. Whenever people say a model "learned" something, what actually happened underneath is gradient descent nudging numbers in a slightly better direction, over and over, until the errors got small enough to stop caring about.

The Math: The Gradient Descent Formula Explained

Here's the update rule, in full:

theta_new = theta_old - eta * grad(J(theta))

Reading it left to right, in plain words:

• theta is whatever parameter you're updating (w or b in our example)

• eta (the learning rate) controls how big a step you take

• grad(J(theta)) is the gradient, the direction of steepest increase in loss

• Subtracting it moves you in the opposite direction, downhill, toward lower loss

For our MSE cost function, the partial derivatives with respect to w and b work out to:

dJ/dw = (2/n) * sum((w*x_i + b - y_i) * x_i) dJ/db = (2/n) * sum(w*x_i + b - y_i)

If deriving those from first principles feels shaky, that's just a calculus gap, not an ML gap, and it's worth closing separately. Scaler's free Maths for Machine Learning course covers exactly this kind of derivative work if you want the refresher before going further.

Hiring Partners:

GoogleAmazonMicrosoftFlipkartAdobe1200+ more

Gradient Descent Step by Step: A Worked Example

Here's the running example: fitting y = wx + b to five points.

• Data: (1, 2.9), (2, 5.2), (3, 6.8), (4, 9.3), (5, 10.9)

• Starting point: w = 0, b = 0 (about as wrong a guess as you can make)

• Learning rate: eta = 0.01

Three iterations, by hand, all numbers to four decimal places:

Loss falls from 57.3980 to 19.5360 in three steps. It doesn't reach zero, and it shouldn't yet, this is three iterations out of what ends up being hundreds before it settles. Run the same setup for 1,000 epochs and w and b converge to roughly 2.0138 and 0.9764, which lines up almost exactly with the closed-form least-squares solution (w = 2.0100, b = 0.9900) you'd get by solving the linear regression problem directly. Gradient descent isn't cheating its way to an answer, it's crawling toward the same one algebra would give you, just without needing to invert a matrix.

Learning Rate: The Most Important Hyperparameter

Get the learning rate wrong and nothing else in this article matters.

• Too small: training crawls. Technically correct, practically useless if it takes 200,000 epochs to get anywhere.

• Too large: each step overshoots the minimum, loss starts bouncing around, and it can genuinely diverge (loss going up instead of down, which is a special kind of frustrating to debug at 1am).

• Just right: steady, smooth decrease, converges in a reasonable number of steps.

In practice almost nobody picks one fixed learning rate and walks away. Learning rate schedules (decaying the rate as training progresses) and adaptive methods are both common enough that "just tune eta once" is more of a teaching simplification than something you'd actually do on a real project.

Types of Gradient Descent

Three variants, and they mostly differ on one question: how much data do you look at before taking a single step?

Worth saying plainly since a lot of pages dance around it: mini-batch is what deep learning actually runs on. Not batch, not pure SGD. Mini-batch gets you a reasonable amount of the stability batch gives you, most of the speed SGD gives you, and it happens to map extremely well onto how GPUs like to chew through data in parallel chunks.

Challenges: Local Minima, Saddle Points, and Plateaus

Vanilla gradient descent can get stuck, and not always for the reason people assume.

• Local minima: a dip that looks like the bottom but isn't the actual lowest point anywhere on the surface.

• Saddle points: flat in some directions, sloped in others, so the gradient shrinks toward zero even though you haven't actually found a minimum.

• Plateaus: stretches where the loss surface is nearly flat everywhere, so progress just crawls.

Here's the part that surprises people: in high-dimensional loss surfaces (which is most real neural networks), saddle points vastly outnumber true local minima. Dauphin et al. (2014) made this case directly, and it's part of why modern optimizers spend more design effort escaping saddle points than avoiding local minima, the local-minima problem was never really the main villain in high dimensions. The Deep Learning Book's optimization chapter is the standard reference if you want the fuller mathematical treatment here.

Beyond Vanilla: Momentum, RMSProp, and Adam

Momentum adds a bit of memory to the process, imagine a ball rolling downhill that keeps some of its previous velocity instead of stopping and recalculating direction from scratch at every single step. It smooths out the noisy parts of the path and helps push through small bumps and shallow local dips. Our momentum-based gradient descent page goes deeper into the mechanics if this is the variant you need for a project.

RMSProp and Adam take a different angle: adaptive learning rates, where each parameter effectively gets its own step size based on how its gradients have behaved recently. Adam (Kingma & Ba, 2014) in particular has become close to a default choice in a huge share of modern training setups, combining momentum with per-parameter adaptive rates. It's not always the objectively optimal choice for every problem, but "just use Adam" is a genuinely reasonable starting point rather than a cop-out.

Our optimizers in deep learning page covers the full lineup in more depth than fits here. If you want to actually train models with these instead of just reading about them, Scaler's free Deep Learning course is built around exactly that kind of hands-on practice.

Gradient Descent in Python (From Scratch)

Here's the running example automated, no libraries beyond NumPy:

import numpy as np

x = np.array([1, 2, 3, 4, 5], dtype=float)
y = np.array([2.9, 5.2, 6.8, 9.3, 10.9], dtype=float)
n = len(x)

w, b = 0.0, 0.0
eta = 0.01
epochs = 1000

for epoch in range(1, epochs + 1):
pred = w * x + b
error = pred - y
loss = np.mean(error ** 2)

dw \= (2 / n) \* np.sum(error \* x)  
db \= (2 / n) \* np.sum(error)  

w \-= eta \* dw  
b \-= eta \* db  

if epoch % 200 \== 0:  
 	print(f'Epoch {epoch}: loss={loss:.4f}, w={w:.4f}, b={b:.4f}')  

print(f'Final: w={w:.4f}, b={b:.4f}')

Running that prints roughly this (actual output, not made up for the article):

Epoch 200: loss=0.0451, w=2.0666, b=0.7856
Epoch 400: loss=0.0394, w=2.0388, b=0.8862
Epoch 600: loss=0.0379, w=2.0246, b=0.9373
Epoch 800: loss=0.0375, w=2.0174, b=0.9632
Epoch 1000: loss=0.0374, w=2.0138, b=0.9764
Final: w=2.0138, b=0.9764

That's the entire algorithm in about twenty lines. Everything sklearn's LinearRegression or PyTorch's optimizer objects do is a more general, more optimized version of this exact loop, with a lot more error handling and vectorization tucked away where you don't have to look at it.

Gradient Descent vs Backpropagation

This mix-up comes up constantly, understandably, since the two terms show up together so often they start to blur.

• Backpropagation computes the gradients, it's the process of working out, layer by layer, how much each weight in a neural network contributed to the final error.

• Gradient descent uses those gradients, it's the update rule that actually moves the weights once backprop has handed over the direction to move in.

One computes the "which way is downhill," the other does the "take a step." You genuinely can't have one without the other in a neural network, they're two different stages of the same training loop, not competing techniques. If backpropagation itself still feels fuzzy, our dedicated backpropagation guide is the natural next read, it's built to connect directly back into everything covered here.

Ready to train models end to end? Explore Scaler's AI & ML Program for the structured path from these fundamentals through to production-grade model training.