Day 32: Optimizers: SGD, Adam, learning rate schedules
Smarter ways to step downhill
Day 24's plain gradient descent steps every parameter by lr × gradient. Real optimizers improve on this. SGD with momentum accumulates a velocity so it powers through small bumps and flat regions instead of crawling. Adam adapts a *per-parameter* learning rate from running estimates of each gradient's mean and variance — it's the reliable default that just works for most problems, including the Garment Classifier.
import torch.optim as optim
# Adam: the sensible default
optimizer = optim.Adam(model.parameters(), lr=1e-3, weight_decay=1e-4)
# SGD + momentum: often generalizes better for CNNs, needs more tuning
# optimizer = optim.SGD(model.parameters(), lr=0.1, momentum=0.9)
# a learning-rate schedule decays lr over time for a finer final fit
scheduler = optim.lr_scheduler.CosineAnnealingLR(optimizer, T_max=num_epochs)Learning rate schedules
A fixed learning rate is a compromise: high enough to make early progress, low enough not to overshoot late. Schedules resolve this by starting high and decaying over training — big steps early to cover ground, small steps late to settle precisely into a minimum. Step decay, cosine annealing, and warmup-then-decay are the common shapes; cosine is a strong default.
weight_decay = free regularization
The weight_decay argument adds a small penalty on large weights, nudging the model toward simpler solutions that generalize better — L2 regularization, built into the optimizer. It's one of the cheapest overfitting countermeasures, and you'll reach for it on Day 39.
Key terms
- Momentum
- Accumulating a velocity from past gradients so updates carry through small bumps and flat regions.
- Adam
- An optimizer that adapts a per-parameter learning rate from running mean/variance estimates of the gradients — the common default.
- Learning-rate schedule
- A rule that changes the learning rate over training, typically decaying it from high to low.
- Weight decay
- A penalty on large weights (L2 regularization) added to the optimizer to reduce overfitting.
Why do learning-rate schedules typically start high and decay toward the end of training?
