Day 20: Project: return-probability model — training & evaluation
Training and honestly evaluating the model
With features ready, you train a baseline and evaluate it with the metrics from Day 17 — not just accuracy. Returns are imbalanced (most orders aren't returned), so you'll lean on precision, recall, F1, and AUC. Start simple: logistic regression or a random forest. A strong baseline you understand beats a complex model you can't debug.
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import classification_report, roc_auc_score
model = RandomForestClassifier(n_estimators=200, class_weight="balanced", random_state=42)
model.fit(X_train, y_train)
proba = model.predict_proba(X_test)[:, 1] # probability of return
preds = (proba > 0.5).astype(int)
print(classification_report(y_test, preds)) # precision/recall/F1 per class
print("AUC:", roc_auc_score(y_test, proba)) # threshold-independentclass_weight="balanced" earns its keep
On imbalanced data, telling the model to weight the rare class more heavily (class_weight='balanced') often does more for recall than any hyperparameter tuning. It's the tabular-ML equivalent of not letting the model take the lazy 'always predict the majority' shortcut.
Feature importance: what did it learn?
A random forest can tell you which features drove its decisions. If size_deviation tops the list, your Day-19 domain insight paid off. Inspecting feature importance is both a sanity check (is the model using sensible signals or leaking something?) and interview gold — you can *explain* your model, not just report its score.
Key terms
- Baseline model
- A simple, well-understood model established first, against which more complex approaches must prove they improve.
- Class weighting
- Assigning higher loss weight to rare classes so an imbalanced dataset does not push the model toward always predicting the majority.
- Feature importance
- A measure of how much each input feature contributed to a model's predictions.
Ship the return-probability model
Train your model, print the full classification report and AUC, and write two sentences on the precision/recall trade-off you'd choose for a real returns-flagging feature and why. Note the top three features by importance. This is your first end-to-end supervised model — small, but complete and defensible.
Your return-prediction dataset is 95% "not returned". A model predicting "not returned" for everything scores 95% accuracy. Why is this model still bad?
