Day 24: Loss functions & gradient descent, from scratch
Measuring wrongness, then reducing it
A loss function turns 'how wrong is the model?' into a single number to minimize. Gradient descent is the loop that minimizes it: forward pass to compute the loss, backward pass to get every parameter's gradient, then step each parameter a small amount *against* its gradient. The step size is the learning rate — the most important hyperparameter you'll tune all stage.
Choosing the right loss
- Mean squared error (MSE) — for regression; penalizes large errors quadratically.
- Cross-entropy — for classification; measures the distance between predicted probabilities and the true label. This is what the Garment Classifier will use.
- The loss must be differentiable, because gradient descent needs its gradient with respect to every parameter.
for step in range(100):
# forward: predict and compute loss
preds = [model(x) for x in xs]
loss = sum((p - y)**2 for p, y in zip(preds, ys))
# backward: zero old grads, then fill new ones
for p in model.parameters():
p.grad = 0.0
loss.backward()
# update: step each parameter against its gradient
lr = 0.05
for p in model.parameters():
p.data -= lr * p.grad
print(step, loss.data)The learning rate is a cliff on both sides
Too small and training crawls (or gets stuck). Too large and the loss oscillates or explodes to NaN as steps overshoot the minimum. Watching the loss curve — is it descending smoothly? — is how you diagnose a bad learning rate, and it's the first thing to check when training misbehaves.
Loss over training steps: a good learning rate descends smoothly; too high a rate bounces and may diverge.
Key terms
- Loss function
- A differentiable measure of model error that training seeks to minimize.
- Cross-entropy loss
- The standard classification loss, measuring the gap between predicted class probabilities and the true label.
- Gradient descent
- Iteratively updating parameters by stepping them against the gradient of the loss.
- Learning rate
- The step-size multiplier controlling how far each gradient-descent update moves a parameter.
During training the loss jumps around erratically and sometimes shoots up to a huge value or NaN. What is the most likely cause?
