Day 42: Why distribution is hard
This is the theory phase that explains everything after it
Every tool from here forward — etcd, Postgres replication, RabbitMQ, Kafka, Redis Sentinel — is a specific engineering answer to the same handful of hard problems. Learn the problems here, once, and every one of those tools becomes an instance of a pattern you already understand instead of unrelated trivia.
Partial failure
On one machine, a function either returns or the whole process crashes — failure is total and obvious. Across a network, one machine can fail while others keep running fine, and worse: you often can't tell the difference between 'that server is dead' and 'that server is just slow to respond.'
Unreliable networks
Packets get dropped, delayed, duplicated, or reordered (Phase 2's TCP mitigates this for a single connection, but says nothing about the many hops and services in a distributed system). A response never arriving could mean the request never arrived, or arrived but the response got lost, or both processed fine and only the network in between failed — and from the caller's side, all three look identical.
No global clock
Every machine's clock drifts slightly, and no two clocks tick in perfect sync. You cannot reliably ask 'which of these two events on different machines happened first?' using wall-clock timestamps alone — Day 46's logical clocks exist precisely because of this.
The one sentence that summarizes this whole phase
Every distributed systems pattern you'll meet is really just: what do we do when we can't fully trust the network, the clock, or whether the other machine is even still alive?
Key terms
- Partial failure
- Some components of a distributed system fail while others keep working normally.
- Network partition
- A network failure that splits a system into groups that can't communicate with each other.
Why is a timeout with no response fundamentally ambiguous in a distributed system?
