Day 97: Building nanoGPT (part 2): training loop on a toy corpus
Training your GPT to generate text
Train nanoGPT on a toy corpus (Karpathy uses tiny Shakespeare; use a small fashion-description corpus to keep it on-theme). The training loop is *exactly* Stage 1 Day 29 — forward, cross-entropy loss on next-token prediction, backward, step. The objective is self-supervised: predict the next token from the previous ones. No labels needed; the text is its own supervision.
for step in range(max_steps):
xb, yb = get_batch(train_data) # xb: tokens, yb: same shifted by 1
logits = model(xb) # (batch, seq, vocab)
loss = F.cross_entropy(
logits.view(-1, vocab_size), yb.view(-1)) # next-token loss
optimizer.zero_grad()
loss.backward()
optimizer.step()
# generation: sample one token at a time, feeding each back in
def generate(model, idx, n):
for _ in range(n):
logits = model(idx[:, -block_size:])[:, -1, :] # last position
probs = F.softmax(logits, dim=-1)
idx = torch.cat([idx, torch.multinomial(probs, 1)], dim=1)
return idxSelf-supervision is why LLMs scale
The genius of next-token prediction is that it needs no human labels — any text is training data. That's why LLMs can train on trillions of tokens of internet text. Your Stage 1 classifier needed hand-labelled garments; a language model supervises itself from raw text. This is the single reason the field could scale to today's models.
Train nanoGPT, then whiteboard Q/K/V
Train nanoGPT on a toy corpus until it generates vaguely coherent text — proof the architecture and loop work. Then, per the Stage 3 exit criterion, whiteboard the Q/K/V attention computation unprompted. If you can build it and explain it, you own the transformer — the conceptual peak of this stage.
Key terms
- Next-token prediction
- The self-supervised objective of training a language model to predict each token from the preceding ones.
- Self-supervised learning
- Learning from data that is its own label (predicting part of the input from the rest) — no human annotation needed.
- Sampling generation
- Producing text by repeatedly predicting a next-token distribution and drawing from it, feeding each token back in.
Why can language models train on trillions of tokens without human labelling?
