AI Learning
AI Learning Rate: What It Is, How to Pick One, and Why Training Fails Without It
The learning rate controls how big a step your model takes each time it corrects itself. Too big and training explodes, too small and it crawls. This guide explains the idea without heavy maths, gives you working default values for common optimisers, covers schedules like cosine decay and warmup, and shows how to diagnose a bad learning rate from the loss curve alone.
Admin ·
The learning rate controls how big a step your model takes each time it corrects itself. Too big and training explodes, too small and it crawls. This guide explains the idea without heavy maths, gives you working default values for common optimisers, covers schedules like cosine decay and warmup, and shows how to diagnose a bad learning rate from the loss curve alone.
The learning rate in one paragraph
The learning rate is a number, usually between 0.000001 and 0.1, that decides how much a model changes its internal weights after each mistake. Training works like this: the model makes a prediction, measures how wrong it was, calculates which direction each weight should move, then moves it. The learning rate is the multiplier on that move. Set it at 0.1 and the model takes huge jumps, often overshooting the good answer and diverging into nonsense. Set it at 0.0000001 and the model technically improves, but you would need weeks to reach a result you could get in an hour. Everything else in training is negotiable. This one is not, and it is the first thing experienced practitioners check when a run goes wrong.
Values that actually work, by situation
Do not start from theory. Start from what the field has converged on after a decade of experiments, then adjust. The table below covers the cases most people run into. If you are using a framework default and things are training fine, leave it alone. Note the pattern: the more pre-trained your model is, the smaller the learning rate needs to be, because you are nudging existing knowledge rather than building it from scratch. Fine-tuning a large language model at 1e-3 will destroy it in a few hundred steps. People do this and then blame the dataset.
| Situation | Typical learning rate | Notes |
|---|---|---|
| Adam or AdamW, training from scratch | 1e-3 | The classic default. Works for most small and medium networks. |
| SGD with momentum, from scratch | 0.1 (with decay) | Needs a schedule. Often beats Adam on image tasks with enough tuning. |
| Fine-tuning BERT-style models | 2e-5 to 5e-5 | 3e-5 is a safe first try. 1e-4 is usually too aggressive. |
| Full fine-tuning of an LLM | 1e-5 to 2e-5 | Lower than you expect. Damage is hard to undo. |
| LoRA / adapter fine-tuning | 1e-4 to 3e-4 | Higher is fine because you are only training a small added module. |
| Transfer learning, frozen backbone, new head | 1e-3 | Only the new layer is learning, so it can move fast. |
| Tabular data, gradient boosting (XGBoost, LightGBM) | 0.05 to 0.1 | Called shrinkage here. Lower it and raise the tree count. |
Reading the loss curve like a diagnosis
You do not need to guess whether your learning rate is wrong. The training loss tells you within the first few hundred steps. A loss that shoots to NaN or infinity in the first epoch means the rate is far too high, usually by a factor of ten or more. A loss that bounces up and down without a downward trend means it is somewhat too high, so halve it. A loss that falls in a clean, almost straight, very slow line means it is too low, so try five times bigger. A loss that drops fast then flattens at a mediocre value often means the rate was fine at first but needs to shrink later, which is what schedules are for.
There is a faster method. Run a learning rate finder: train for a few hundred steps while increasing the rate exponentially from 1e-7 to 1, and plot loss against rate. You get a curve that descends, hits a minimum, then spikes. Pick a value roughly one order of magnitude below the point of steepest descent. PyTorch Lightning and fastai both ship this as a one-line call. It takes two minutes and saves an afternoon.
- NaN loss in early steps: rate is 10x too high or more
- Loss oscillating with no trend: halve the rate
- Loss falling in a slow straight line: try 5x higher
- Good start then early plateau: add a decay schedule
- Loss decreasing but validation loss rising: this is overfitting, not a learning rate problem
Schedules: why one number for the whole run is rarely right
Early in training the weights are random, so big steps are useful. Later the model is close to a good solution and big steps just knock it away. So most serious training uses a schedule that changes the rate over time.
Step decay is the oldest approach: cut the rate by ten every so many epochs. Simple, still effective, common in older computer vision recipes. Cosine annealing smoothly reduces the rate from its starting value down to near zero following a cosine curve, and it has become the default for transformer training because it needs no tuning of decay points. Warmup does the opposite at the start: begin near zero and ramp up over the first few hundred to few thousand steps. Warmup exists because early gradients in large models are noisy and unstable, and a full-size step on step one can wreck the run. Nearly every large language model trained since 2019 uses linear warmup followed by cosine decay. That combination is a reasonable default and you can copy it without shame.
Adaptive optimisers like Adam adjust per-parameter step sizes internally, which makes them forgiving. They do not make the learning rate irrelevant. Adam still has a global rate and it still matters. If someone tells you Adam removes the need to tune it, they have not trained anything large.
If the mechanics of gradients and weight updates still feel fuzzy, it helps to step back and read the ai learning process end to end before tuning anything, because the learning rate only makes sense as one knob inside that loop.
Batch size, and the practical order of tuning
Learning rate and batch size are linked. Increase the batch size and the gradient estimate gets less noisy, so you can afford a bigger step. The common rule of thumb is linear scaling: double the batch size, double the learning rate. This held up in Facebook's 2017 ImageNet work where they trained ResNet-50 in an hour on 256 GPUs using linear scaling plus warmup. The rule breaks down at very large batches, so treat it as a starting adjustment rather than a law.
Here is the order that saves time. Fix your architecture and data first. Set batch size to the largest that fits in memory. Then find the learning rate. Then add a schedule. Only after that touch weight decay, dropout or anything else. People tune in the reverse order and wonder why nothing helps.
One last thing worth internalising: the learning rate is not a quality setting where higher means better or lower means safer. It is a step size, and the right value depends entirely on the shape of the loss surface you happen to be walking on. Which is why you measure it instead of arguing about it.
- Lock the model and dataset
- Set batch size by memory limit
- Run a learning rate finder or a short sweep of 3 values
- Add warmup plus cosine decay
- Tune regularisation last
FAQs
1. What is a good default learning rate if I have no time to tune?
Use 1e-3 with AdamW when training a small model from scratch, and 3e-5 when fine-tuning a pre-trained language model. Both are widely used starting points that rarely diverge, and you can refine from there if the loss curve looks wrong.
2. Does a lower learning rate always give better accuracy?
No. Very low rates can leave the model stuck in a poor region and often just waste compute, since training stops improving before it reaches a good solution. Moderately higher rates add useful noise that helps the model escape shallow minima.
3. What is the difference between learning rate and epochs?
The learning rate is how far the model moves on each update, while epochs count how many times it passes through the full dataset. A tiny learning rate with many epochs and a larger learning rate with few epochs can reach similar places, but the first costs far more time.
4. Why does my loss become NaN after a few steps?
Almost always the learning rate is too high, so weights blow up beyond the range the numbers can hold. Divide it by ten and retry, and if you are using mixed precision training also check that gradient scaling and warmup are enabled.
5. Do I need to tune the learning rate when using Adam?
Yes. Adam adapts the step size per parameter, but it still multiplies everything by a global learning rate that you set. Adam is more forgiving than plain SGD, not immune.
6. What is learning rate warmup and when should I use it?
Warmup starts training at a near-zero learning rate and ramps it up over the first few hundred or few thousand steps. Use it for transformers, large batch sizes, and any run that diverges in the first epoch despite a sensible target rate.