What Is Optimization in Deep Learning?

Optimization in deep learning is the process of adjusting a network's weights to reduce its loss. It is what turns a random set of numbers into a model that works. This article explains what the optimizer does, how gradient descent and Adam differ, why loss curves misbehave, and what to check when training stalls.

The short answer

Optimization in deep learning is the process of changing a network's weights, step by step, so that its loss goes down. The loss is a single number that measures how wrong the model's predictions are. An optimizer is the algorithm that decides how big each change should be and in which direction.

That is the whole job. A neural network with 100 million parameters starts life as 100 million random numbers, and it is useless. Optimization is the only thing that turns those random numbers into something that can recognise a tumour in a scan or complete your sentence. Architecture decides what the model could learn. Optimization decides whether it actually does.

Here is the mental picture that holds up. You are standing on a hilly surface in thick fog, and you want to reach the lowest valley. You cannot see the map. All you can feel is the slope under your feet. So you take a step downhill, feel the slope again, take another step. That slope is the gradient. The step size is the learning rate. Repeat a few hundred thousand times and you have trained a neural network.

How one training step actually works

Training runs in a loop, and every pass through it has four parts. Once you see them as separate things, most confusion about "where the learning happens" disappears.

Backpropagation and optimization get mixed up constantly, so be precise about it. Backprop computes the gradients. The optimizer uses them. Backprop is calculus, mechanical and settled. The optimizer is a design choice, and it is where you have actual decisions to make.

• Forward pass: feed a batch of data through the network, get predictions.
• Loss: compare predictions to the correct answers, get one number.
• Backward pass (backpropagation): compute how much each weight contributed to that loss. This is the gradient.
• Optimizer step: nudge every weight in the direction that reduces loss, scaled by the learning rate.

Why plain gradient descent is not enough

Classic gradient descent computes the gradient over the entire dataset before taking one step. On ImageNet, that means 1.2 million images per step. Nobody does this.

What everyone actually uses is mini-batch stochastic gradient descent. You take a batch of 32 or 256 samples, compute a noisy estimate of the gradient, and step. The estimate is rough, and that turns out to be a feature, not a bug. The noise helps the model rattle out of bad flat regions instead of settling there.

But raw SGD has real problems. In a narrow ravine it zigzags across the walls instead of moving along the floor. It uses the same step size for every parameter, even though the embedding layer and the final classifier need very different treatment. And once the gradient goes nearly flat, it crawls.

The fixes arrived in layers. Momentum (1986, popularised in deep learning around 2013) adds a running average of past gradients, so the model builds up speed in consistent directions and damps the zigzag. RMSProp scales each parameter's step by how large its recent gradients have been, so rarely-updated weights get bigger steps. Adam, published in 2014, combines both, and it is still the default for most people training transformers and vision models a decade later. AdamW, a 2017 correction to how Adam handles weight decay, is what almost every large language model is trained with now.

Optimizer	Core idea	Where it wins	Watch out for
SGD	Step against the gradient	Simple, generalises well on vision tasks	Slow, needs careful learning rate tuning
SGD + momentum	Adds velocity from past gradients	ResNets, CNNs, most image classification	Momentum too high causes overshoot
RMSProp	Per-parameter step scaling	RNNs, non-stationary problems	Can stall on some convex problems
Adam	Momentum plus per-parameter scaling	Fast convergence, works almost everywhere	Slightly worse generalisation than tuned SGD
AdamW	Adam with decoupled weight decay	Transformers, LLM fine-tuning	Needs sensible weight decay value

The learning rate is the setting that decides everything

If you change one thing about how you train models, change how seriously you treat the learning rate. It matters more than the choice of optimizer, more than the batch size, and more than almost any architectural tweak you are tempted to make.

Too high and the loss explodes or turns into NaN within a hundred steps. Too low and training technically works but takes forty times longer, and you will give up before it converges. The workable range is usually narrow, and it is not the same for a fine-tune as it is for training from scratch. For AdamW on a transformer fine-tune, people typically land somewhere between 1e-5 and 5e-5. Train a small CNN from scratch and 1e-3 is a reasonable starting guess.

Schedules matter as much as the initial value. Almost every serious training run now uses warmup (start tiny, ramp up over the first few hundred steps) followed by cosine decay down to near zero. The warmup exists because Adam's per-parameter scaling estimates are garbage at step one, and taking full-size steps on garbage estimates wrecks the model early. If you want the full treatment on this one dial, read AI Learning Rate: What It Is, How to Pick One, and Why Training Fails Without It.

What optimization is not trying to do

There is a misconception worth killing. Optimization is not searching for the global minimum of the loss function.

A deep network's loss surface has an astronomical number of local minima and saddle points. Finding the true global minimum is computationally hopeless, and here is the thing: you would not want it anyway. The global minimum on your training data usually means the model has memorised the training set. Perfect training loss, useless on anything new.

What you actually want is a solution that is low enough and sits in a broad, flat basin rather than a sharp one. Flat basins tend to generalise better, because small shifts in the input distribution do not push the model off a cliff. This is why techniques that look like they hurt optimization often help the final model. Dropout adds noise. Weight decay pulls weights towards zero, away from the training-loss optimum. Early stopping halts training before the loss bottoms out. All three make training loss worse and test performance better.

So optimization in deep learning is a compromise between two things that pull in opposite directions: driving the loss down fast, and not driving it down so precisely that the model learns your dataset's quirks instead of the pattern underneath.

What to check when training goes wrong

Most training failures are optimization failures, and they show up in a handful of recognisable patterns. Loss flat from step one, usually a learning rate too low or a bug where gradients never reach the weights. Loss spiking to NaN, learning rate too high or a numerical blowup in the loss itself, and gradient clipping at norm 1.0 fixes a surprising amount of it. Training loss falling while validation loss climbs, you are past the useful point and should stop. Loss dropping then plateauing far above where it should, often a schedule that decayed too early.

The honest lesson from working on this: you learn optimization by watching loss curves, not by reading about gradients. Run a small model, deliberately set the learning rate 100x too high, and watch what happens. Then set it 100x too low. Twenty minutes of that teaches more than a chapter.

That is also the gap most people hit after reading an explainer like this one. You understand the concept and still cannot debug your own run. If you want to close it with structured practice and someone reviewing your training runs, the AI Fellowship is built around building and breaking things over eight weeks rather than consuming theory. Either way, get your hands on a training loop this week. Optimization stops being abstract the first time you fix a loss curve yourself.

FAQs

1. Is backpropagation the same as optimization?

No. Backpropagation computes the gradients, meaning how much each weight contributed to the loss. The optimizer then decides how to use those gradients to update the weights. They run one after the other in every training step, but they are different jobs.

2. Which optimizer should I use by default?

AdamW, with a warmup and cosine decay schedule. It is the default for transformers and works acceptably on almost everything else. SGD with momentum can beat it on image classification if you are willing to tune the learning rate carefully.

3. Why does my loss become NaN during training?

Almost always a learning rate that is too high, causing weights to blow up past the range of floating point. Other common causes are dividing by zero inside a custom loss, or unstable mixed-precision training. Try cutting the learning rate by 10x and adding gradient clipping at norm 1.0.

4. What is the difference between a loss function and an optimizer?

The loss function defines what "wrong" means, for example cross-entropy for classification or mean squared error for regression. The optimizer is the algorithm that reduces that loss by changing weights. You pick the loss based on the task and the optimizer based on training behaviour.

5. Does a bigger batch size make optimization better?

It makes gradients less noisy and better for GPU throughput, but not automatically better for the final model. Very large batches often need a proportionally larger learning rate plus longer warmup, and can generalise slightly worse. Batch sizes between 32 and 512 are a safe range for most work.

6. How is optimization in deep learning different from optimization in machine learning generally?

The maths is the same family, but classical machine learning often has convex loss surfaces with a single guaranteed minimum, solvable in closed form or with a few iterations. Deep networks have non-convex surfaces with millions of parameters, so you rely on iterative gradient methods and accept a good-enough solution rather than a provably optimal one.

Related reading

• AI Learning Rate: What It Is, How to Pick One, and Why Training Fails Without It
• what is optimization in machine learning
• The AI Learning Process, Explained Step by Step
• What Is Model Quantization? The Plain Explanation, With Numbers
• How to Prevent AI Model Collapse: A Practical Guide for Anyone Training on Web Data