The learning rate is a number set before training that scales how far a model's parameters move after each gradient calculation. Set too low, training crawls and may stall; set too high, the loss bounces or climbs and the model never converges. No single value works everywhere, so the rate is tuned for each model and dataset, often with a schedule that changes it during training.
Key takeaways
- The learning rate multiplies the gradient to produce each parameter update, so it controls step size, not direction.
- Google's machine learning course states that too small a rate makes convergence slow and too large a rate keeps the loss from converging at all.
- Common fixes are warmup, decay schedules, and cyclical schedules, which PyTorch and other libraries ship as built-in tools.
- When the batch size grows, a published rule of thumb scales the rate in proportion, with a warmup period at the start.
- Good values are found by testing, since the best rate depends on the model, the data, and the optimizer.
What the learning rate does in one update
Training a neural network means repeatedly measuring how wrong the model is (the loss), computing the gradient that shows which direction reduces it, and nudging the parameters that way. The Google Machine Learning Crash Course defines the learning rate as a floating-point number that controls how quickly the model converges: the gradient is multiplied by it to get the next parameter values. Its example uses a gradient of 2.5 and a rate of 0.01, which changes the parameter by 0.025.
The course lists the two failure modes plainly. With a rate that is too small, convergence takes many iterations and each one improves the loss only slightly. With a rate that is too large, the loss keeps bouncing around or rises. It also says there is no ideal value that suits every model and dataset, which is why the rate is called a hyperparameter: it is chosen by the practitioner, not learned from the data.
A worked example with one parameter
The behavior is easy to see on the simplest possible loss, L(w) = w squared, whose gradient is 2w. Each update is w minus the rate times 2w, which equals w multiplied by (1 minus 2 times the rate). The numbers below are an illustrative example, starting at w = 1, and are not taken from any published experiment.
| Learning rate | Multiplier per step | w after four steps | What happens |
|---|---|---|---|
| 0.01 | 0.98 | 0.92 | Moves toward zero, very slowly |
| 0.1 | 0.8 | 0.41 | Steady progress toward zero |
| 0.5 | 0 | 0 | Lands on the minimum in one step |
| 1.0 | -1 | 1 | Flips between 1 and -1 forever |
| 1.1 | -1.2 | 2.07 | Overshoots by more each step and diverges |
Real losses have millions of parameters and uneven curvature, so no formula this clean exists. The pattern still carries over: below a threshold, the rate trades speed against safety; above it, the updates grow instead of shrinking.
Why a fixed rate is rarely the whole answer
Early in training a large step can be useful, because the parameters are far from any good solution. Late in training the same step can keep the model jumping around a minimum instead of settling into it. Practitioners therefore change the rate over time. The PyTorch documentation on optimizers lists several ready-made schedulers, including StepLR, CosineAnnealingLR, OneCycleLR, ReduceLROnPlateau, and LinearLR. It also warns that the scheduler should be called after the optimizer's update, because calling it first skips the first value of the schedule.
These options cover different needs. A step schedule cuts the rate at fixed points. A cosine schedule lowers it along a smooth curve. A plateau schedule waits until a monitored metric stops improving, then lowers the rate. A linear schedule can ramp the rate up at the start, which is the idea behind warmup.
Cyclical rates and the range test
Leslie N. Smith's paper Cyclical Learning Rates for Training Neural Networks (first submitted June 3, 2015, presented at WACV 2017) argues against only ever lowering the rate. It lets the rate rise and fall between two bounds. The abstract says the approach practically eliminates the need to search for a single best value and schedule, and it describes finding reasonable bounds by increasing the rate linearly over a few epochs. The author reports improved accuracy, often in fewer iterations, on CIFAR-10 and CIFAR-100 with ResNets, Stochastic Depth networks, and DenseNets, and on ImageNet with AlexNet and GoogLeNet. The abstract gives no single headline figure, and results from one paper do not guarantee the same gain on a different task.
Learning rate and batch size
Batch size changes how noisy each gradient is, so it interacts with the rate. In "Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour," submitted June 8, 2017, Priya Goyal and colleagues report no loss of accuracy with minibatches of up to 8,192 images. Their recipe scales the learning rate linearly with minibatch size and adds a warmup scheme for the early phase of training. They trained ResNet-50 on 256 GPUs in one hour.
As an illustrative calculation, a rate of 0.1 at a batch of 256 would become 0.4 at a batch of 1,024 under the linear rule. The rule is a starting point reported for one family of image models, not a law, and other architectures and optimizers may need different treatment.
A practical checklist for choosing a rate
- Start from the default or the value suggested by the optimizer's original paper or the library documentation.
- Run a short range test: raise the rate gradually over a few epochs and note where the loss starts to worsen.
- Watch the training loss curve: a flat line suggests the rate is too low, while spikes or growth suggest it is too high.
- Add warmup when training large models or using large batches.
- Change one thing at a time, and compare runs on a held-out validation set rather than on training loss alone.
- Record the rate, the schedule, and the random seed so a run can be repeated.
Adaptive optimizers such as Adam, introduced by Diederik Kingma and Jimmy Ba in a paper submitted December 22, 2014, adjust step sizes per parameter using estimates of gradient moments. The authors say its hyperparameters have intuitive interpretations and typically require little tuning. A base learning rate still has to be set, though, and schedules are still commonly applied on top of it.
FAQ
What is a good learning rate for a neural network?
No single value is correct for every model and dataset. Google's course says the ideal rate is problem-dependent, so the practical method is to start from a documented default and test nearby values on validation data.
How can a learning rate that is too high be recognized?
The loss bounces around or rises over iterations instead of falling. In the one-parameter example above, any rate over 1.0 makes each step larger than the last.
Is the learning rate the same as the batch size or the number of epochs?
No. The learning rate scales each update, the batch size is how many examples are processed before an update, and an epoch is one full pass through the training data. All three are hyperparameters set before training.
Do large language models use a learning rate?
Any model trained by gradient descent has one. Values for specific commercial models are often unpublished, so none are stated here.
Related reading
- How AI differs from machine learning
- AI hardware and what training benchmarks measured
- The GPT-4.5 announcement in context
- Linear regression as a predictive model
More in AI Basics.
Last verified: October 2026
Sources
- Linear regression: Hyperparameters, Machine Learning Crash Course, Google for Developers, accessed October 2026
- Smith LN, Cyclical Learning Rates for Training Neural Networks, arXiv 1506.01186, WACV 2017
- Goyal P et al., Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour, arXiv 1706.02677, 2017
- Kingma DP and Ba J, Adam: A Method for Stochastic Optimization, arXiv 1412.6980, ICLR 2015
- torch.optim documentation, PyTorch, version 2.14, 2026