Day 17: Precision, recall, F1, confusion matrix, ROC-AUC
Why accuracy is not enough
If only 2% of orders are returned, a model that predicts 'never returned' is 98% accurate and completely worthless. This is why you need metrics that look at *what kind* of right and wrong. They all derive from the confusion matrix: true positives (TP), false positives (FP), true negatives (TN), false negatives (FN).
Predicted YES Predicted NO
Actual YES TP (correct) FN (missed)
Actual NO FP (false alarm) TN (correct)Precision, recall, F1
- Precision = TP / (TP + FP) — of everything the model flagged, how much was correct? (Cost of false alarms.)
- Recall = TP / (TP + FN) — of all the real positives, how many did the model catch? (Cost of misses.)
- F1 = harmonic mean of precision and recall — one number balancing both, punishing a model that sacrifices one for the other.
The trade-off is a business decision
For flagging likely returns, do you care more about not annoying good customers (high precision) or catching every risky order (high recall)? There is no universally correct answer — the right operating point depends on the cost of a false alarm versus a miss. Stating this trade-off out loud is exactly what interviewers listen for.
ROC-AUC
A classifier outputs a *score*, and you choose a threshold to turn it into yes/no. The ROC curve plots the true-positive rate against the false-positive rate across *every* possible threshold; AUC (area under that curve) summarizes it as one number from 0.5 (random) to 1.0 (perfect). AUC is threshold-independent, which makes it the go-to for comparing models before you've committed to an operating point.
Key terms
- Confusion matrix
- The 2×2 table of TP/FP/TN/FN counts from which all classification metrics are derived.
- Precision
- TP / (TP + FP): the fraction of positive predictions that were correct.
- Recall
- TP / (TP + FN): the fraction of actual positives the model successfully caught.
- F1 score
- The harmonic mean of precision and recall — a single balanced metric.
- ROC-AUC
- Area under the true-positive-vs-false-positive curve; a threshold-independent measure of ranking quality (0.5 = random, 1.0 = perfect).
A fraud model catches 95% of all real fraud (high recall) but half of its fraud alerts are false alarms (low precision). Which metric best captures this imbalance in a single number?
