Day 92: Embeddings & tokenization: how text becomes vectors
Turning language into numbers
A model can't process raw text — it processes vectors. Two steps bridge the gap. Tokenization splits text into tokens (subword pieces, not always whole words) and maps each to an integer id. Embedding looks up a learned vector for each id. So 'black trousers' becomes token ids, then a sequence of vectors — the same vector-as-meaning idea from Stage 0 Day 7, now for language.
Subword tokenization
Modern tokenizers (BPE, SentencePiece) use *subword* units: common words are single tokens, rare words split into pieces ('tokenization' → 'token' + 'ization'). This keeps the vocabulary manageable while handling any word, including ones never seen in training. It's why token counts differ from word counts — critical when you're paying per token or fitting a context window (tomorrow and Day 100).
from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained("meta-llama/Llama-3.2-1B")
ids = tok.encode("formal shirts under 2500")
print(ids) # e.g. [128000, 63316, 44380, 1234, ...]
print(tok.convert_ids_to_tokens(ids)) # see the subword pieces
print(len(ids), "tokens") # token count != word countEmbeddings encode meaning geometrically
Learned embeddings place similar words near each other in vector space — 'shirt' and 'blouse' end up closer than 'shirt' and 'shoe'. This is exactly the property RAG (Day 112) exploits: embed a query and a catalog of products, and nearest-neighbor search finds semantically relevant items. The embedding you meet here for the model's input is the same tool you'll use for retrieval.
Key terms
- Tokenization
- Splitting text into tokens (often subword units) and mapping each to an integer id.
- Subword tokenization
- Breaking rare words into smaller known pieces (BPE/SentencePiece), keeping vocabulary bounded while covering any word.
- Embedding
- A learned vector representing a token, positioning semantically similar tokens near each other.
Why do modern LLMs use subword tokenization rather than one token per whole word?
