Day 27: Autograd: how PyTorch computes gradients automatically
Your micrograd, industrialized
PyTorch's autograd is micrograd at scale. Set requires_grad=True on a tensor and PyTorch records every operation involving it into a graph. Call .backward() on a scalar loss and it walks that graph in reverse — exactly your Day-23 topological backprop — depositing each tensor's gradient into its .grad attribute. Because you built the toy version, none of this is mysterious.
import torch
w = torch.tensor([2.0], requires_grad=True)
x = torch.tensor([3.0])
y = (w * x) ** 2 # forward pass builds the graph
y.backward() # reverse pass fills w.grad
w.grad # tensor([36.]) — dy/dw = 2*(w*x)*x = 2*6*3 = 36no_grad and detach: turning autograd off
During inference you don't need gradients, and tracking them wastes memory and time. with torch.no_grad(): disables graph recording for a block — you'll wrap every inference call in it (Stage 0 Day 2 previewed this pattern). .detach() pulls a tensor out of the graph when you need its value but not its history.
zero_grad, explained by what you built
Remember micrograd's += gradient accumulation? PyTorch does the same — gradients add up across .backward() calls. That's why every training loop calls optimizer.zero_grad() first: without it, step 2's gradients pile on top of step 1's, and training silently breaks. You understand this ritual because you implemented the accumulation.
Key terms
- requires_grad
- A tensor flag telling autograd to track operations on it so its gradient can be computed.
- .backward()
- Triggers reverse-mode differentiation from a scalar, filling every upstream tensor's .grad.
- torch.no_grad()
- A context manager that disables gradient tracking — used during inference to save memory and time.
- zero_grad()
- Resets accumulated gradients to zero before a new backward pass, since PyTorch accumulates by default.
A training loop that omits optimizer.zero_grad() before loss.backward() will misbehave. Why?
