Day 64: U-Net architecture: encoder-decoder, skip connections
The architecture behind segmentation — and diffusion
SAM is a black box you *use*; U-Net is one you'll *build*, because it's the fundamental segmentation architecture and — crucially — the backbone of the Stage 4 diffusion model. Building a tiny U-Net now means diffusion won't introduce a new architecture later, just a new training objective. This is the roadmap's design at work: learn U-Net once, in the week a task forces it, and collect the payoff twice.
Encoder-decoder with skip connections
- Encoder (contracting path) — successive conv+pool blocks shrink spatial size while growing channels, building an abstract understanding of *what* is in the image (Day 34's feature hierarchy).
- Decoder (expanding path) — upsampling blocks grow the spatial size back to full resolution, turning understanding into a per-pixel mask.
- Skip connections — feature maps from each encoder level are concatenated into the matching decoder level, restoring the fine spatial detail pooling threw away. (Yes — the same skip-connection idea as ResNet, Day 36.)
Why the skips are the whole trick
The encoder knows *what* but loses *where* (pooling discards precise position); the skip connections hand the decoder the high-resolution 'where' from the encoder so the final mask is both semantically correct and spatially sharp. Without them you get blurry, imprecise masks. This is the single idea that makes U-Net work.
Key terms
- U-Net
- An encoder-decoder segmentation architecture with skip connections; also the backbone of diffusion models.
- Encoder / decoder
- The contracting path that builds abstract understanding, and the expanding path that reconstructs full-resolution output.
- Skip connection (U-Net)
- Concatenating encoder feature maps into matching decoder levels to restore fine spatial detail.
What do U-Net's skip connections contribute to the output mask?
