Day 7: Linear algebra refresher: vectors, matrices, dot product
The only math course in this roadmap
This roadmap's design decision #1 is that math isn't a separate phase — you learn each concept the week a model forces you to. Today's the exception that proves the rule: a light pass over vectors, matrices, and the dot product, ideally alongside 3Blue1Brown's *Essence of Linear Algebra* episodes 1-4 as commute viewing. You'll re-meet every one of these ideas concretely in Stage 1 (gradients), Stage 2 (image transforms), and Stage 3 (embeddings) — today just gives you vocabulary and intuition to hang that on.
Vectors
A vector is just an ordered list of numbers — [3, 4]. Geometrically, it's an arrow from the origin to that point, with a length (magnitude) and a direction. You already used vectors on Day 4: a pixel's RGB value, [220, 50, 30], is a 3-dimensional vector. A 512-dimensional sentence embedding in Stage 3 is exactly the same idea, just with 512 numbers instead of 3.
The dot product: a similarity measure
The dot product of two vectors — multiply corresponding elements, sum the results — collapses two vectors into a single number that's large and positive when they point the *same* direction, near zero when they're roughly perpendicular (unrelated), and negative when they point opposite ways. This is not abstract: it's the literal mechanism behind Stage 3's RAG retrieval ('which stored embedding is most similar to this query?') and Stage 3's attention mechanism ('how much should this word attend to that one?').
import numpy as np
a = np.array([1, 2, 3])
b = np.array([4, 5, 6])
# by hand: (1*4) + (2*5) + (3*6)
by_hand = 1*4 + 2*5 + 3*6 # 32
# vectorized (this IS what a.dot(b) or a @ b computes)
np.dot(a, b) # 32
a @ b # 32 — the @ operator is matrix/dot multiplicationMatrices as transformations
A matrix can be read as a table of numbers — or, more usefully, as a *function* that transforms vectors (rotates, scales, skews them). Multiplying a vector by a matrix produces a new vector. A convolution (Stage 1, CNNs) and a fully-connected neural-network layer (Stage 1, PyTorch) are both, underneath, structured matrix multiplications — this is why 'the model is just matrix math' stops sounding like a dismissal once you've felt it directly.
Key terms
- Vector
- An ordered list of numbers, interpretable as a point or direction in space.
- Dot product
- Sum of the element-wise products of two equal-length vectors; a single number measuring directional similarity.
- Matrix
- A 2D table of numbers, interpretable as a linear transformation applied to vectors.
Two embedding vectors have a dot product close to zero. What does that suggest?
