Day 22: Building micrograd: the autograd engine from scratch
Why build an autograd engine by hand
PyTorch computes gradients for you automatically, which is a gift — and a trap, if you never see how. Karpathy's micrograd is a tiny (~100-line) engine that does exactly what PyTorch's autograd does, small enough to hold in your head. Building it is the single most demystifying exercise in this stage: after it, 'the model backpropagates the loss' is something you've *implemented*, not memorized.
The Value: a number that remembers where it came from
The core trick: wrap every number in a Value object that records not just its data, but the operation and the operands that produced it. This builds a computation graph as your math runs. Each Value also stores a grad (its derivative of the final output) and a local _backward function that knows how to propagate gradient to its inputs using the chain rule.
class Value:
def __init__(self, data, _children=(), _op=""):
self.data = data
self.grad = 0.0 # dOutput/dSelf, starts at 0
self._backward = lambda: None # how to push grad to children
self._prev = set(_children) # the operands that made this Value
self._op = _op
def __add__(self, other):
out = Value(self.data + other.data, (self, other), "+")
def _backward():
# d(a+b)/da = 1, d(a+b)/db = 1 — gradient flows through unchanged
self.grad += out.grad
other.grad += out.grad
out._backward = _backward
return out
def __mul__(self, other):
out = Value(self.data * other.data, (self, other), "*")
def _backward():
# d(a*b)/da = b, d(a*b)/db = a — the chain rule, locally
self.grad += other.data * out.grad
other.grad += self.data * out.grad
out._backward = _backward
return outThe += is not a typo
Gradients *accumulate* with += because a Value can feed into multiple downstream operations, and the chain rule says their contributions add. This is also why PyTorch makes you call optimizer.zero_grad() each step — otherwise gradients from the previous step keep accumulating. You'll understand that ritual because you built the reason for it.
Key terms
- Autograd (automatic differentiation)
- Automatically computing exact derivatives by recording operations and applying the chain rule mechanically.
- Computation graph
- A DAG of the operations that produced a value, built as the forward computation runs, and traversed backward to compute gradients.
- micrograd
- Karpathy's ~100-line educational autograd engine implementing scalar reverse-mode automatic differentiation.
In micrograd, why does each Value accumulate gradient with `self.grad += ...` rather than overwriting with `=`?
