Day 5: Pandas basics: DataFrames, Series, cleaning tabular data
Pandas: NumPy with labels
A DataFrame is a 2D table — rows and named columns — built on top of NumPy arrays, one per column. A Series is a single labeled column. Where NumPy is the array mental model for images and tensors, Pandas is the mental model for the *tabular* data you'll use in Stage 1's return-probability model (order history, sizes, prices, whether an item was returned).
import pandas as pd
orders = pd.read_csv("orders.csv")
orders.head() # first 5 rows
orders.shape # (n_rows, n_columns)
orders.dtypes # dtype per column
orders["was_returned"].value_counts() # a Series methodCleaning: the part that takes 70% of the time
Real order data has missing values, wrong types (a price stored as text), and duplicate rows. Pandas' cleaning vocabulary — isna, fillna, dropna, astype, drop_duplicates — is what you'll reach for constantly, here and again when building the Stage 4 fashion-instruction dataset.
# find missing values
orders.isna().sum()
# fill missing size with the mode, drop rows missing a price
orders["size"] = orders["size"].fillna(orders["size"].mode()[0])
orders = orders.dropna(subset=["price"])
# fix a wrongly-typed column
orders["price"] = orders["price"].astype(float)
# remove exact duplicate rows
orders = orders.drop_duplicates()Filtering and grouping
Boolean-mask filtering works the same way it did on NumPy arrays yesterday — Pandas is built on that exact mechanism. groupby is the tool for 'return rate by category', 'average order value by size' — the aggregate questions Stage 1's tabular model will be built from.
# filter: orders over ₹2000
expensive = orders[orders["price"] > 2000]
# groupby: return rate per garment category
return_rate_by_category = orders.groupby("category")["was_returned"].mean()Key terms
- DataFrame
- A 2D labeled table of data, made of columns that are each a Series, built on NumPy arrays.
- Series
- A single labeled 1D column of data — the building block of a DataFrame.
- groupby
- Splitting a DataFrame into groups by a column's values, then applying an aggregate (mean, sum, count) to each group.
You want the average order price broken down by garment category. Which is the right tool?
