Day 11: OpenCV IV: GrabCut background removal
The centerpiece: real background removal
Everything this week has been building toward this: GrabCut, OpenCV's algorithm for separating foreground from background given only a rough starting rectangle. It's the classical-CV answer to 'remove the background from this garment photo' — and worth understanding well, because Stage 2 replaces it with a neural segmentation model (SAM/SAM2) doing conceptually the same job far more robustly. Feel where GrabCut struggles this week, and Stage 2's upgrade will make concrete sense instead of feeling like an arbitrary tool swap.
How it works
You give GrabCut a rectangle roughly containing the foreground object. It initializes two Gaussian Mixture Models (GMMs) — one modeling the color distribution of likely foreground pixels, one for background — from the pixels inside vs. outside that rectangle. It then treats the image as a graph (Day 7's matrices made concrete: each pixel is a node) and solves a min-cut — the cheapest way to sever the graph into a foreground group and a background group, where 'cheap' means pixels with similar colors and adjacent positions strongly prefer to stay on the same side. The result reclassifies pixels; the GMMs are re-estimated from the new classification; and it repeats for a fixed number of iterations, refining the boundary each round.
mask = np.zeros(img_bgr.shape[:2], np.uint8)
bgd_model = np.zeros((1, 65), np.float64) # GrabCut's internal GMM scratch space
fgd_model = np.zeros((1, 65), np.float64)
# rough rectangle: (x, y, width, height) roughly containing the garment
rect = (30, 30, img_bgr.shape[1] - 60, img_bgr.shape[0] - 60)
cv2.grabCut(img_bgr, mask, rect, bgd_model, fgd_model, iterCount=5, mode=cv2.GC_INIT_WITH_RECT)
# GrabCut labels pixels 0=bg, 1=fg, 2=probable-bg, 3=probable-fg
foreground_mask = np.where((mask == 1) | (mask == 3), 1, 0).astype("uint8")
cutout = img_bgr * foreground_mask[:, :, np.newaxis] # Day 4's broadcasting againWhere GrabCut breaks — and why that motivates Stage 2
GrabCut struggles when the garment's color closely matches the background, when the initial rectangle is a poor fit, or on cluttered/patterned backgrounds — because its foreground/background model is just a color-and-position statistic, with no actual understanding of 'garment-ness'. SAM (Stage 2) replaces this color-based iterative optimization with a neural network trained on millions of images to recognize objects directly — same job, fundamentally different (and far more robust) approach.
Key terms
- GrabCut
- An iterative foreground/background segmentation algorithm using Gaussian Mixture Models and a graph min-cut, seeded by a rough bounding rectangle.
- Gaussian Mixture Model (GMM)
- A statistical model representing a distribution (here, pixel colors) as a weighted mixture of several Gaussian distributions.
- Min-cut
- The cheapest way to sever a graph into two groups, used here to separate foreground and background pixels.
Why does GrabCut typically fail on a garment that closely matches its background in color?
