In a single machine, time is simple — one clock, one truth. In a distributed system, every node has its own clock, and those clocks drift apart. NTP in a well-configured datacenter achieves ~1ms precision between servers. Over the public internet, NTP precision degrades to 50–100ms. Google's internal systems observed clock skew of up to 200ms between machines using standard NTP — which is why Google built TrueTime for Spanner, bounding uncertainty to ~7ms using GPS receivers and atomic clocks in each datacenter. When two events happen on different machines "at the same time," you cannot determine which happened first using wall clocks alone. This is not a bug — it is a fundamental property of physics in distributed systems.

Hybrid Logical Clocks (HLC) combine physical time with logical counters — giving you a timestamp that respects causality while staying close to wall-clock time. Used by CockroachDB and Spanner. Google Spanner uses TrueTime (GPS + atomic clocks) to bound clock uncertainty to ~7ms — expensive hardware solving a fundamental physics problem.

The impossibility of perfect time ordering in asynchronous distributed systems is provable. The FLP impossibility theorem (Fischer, Lynch, Paterson 1985) states that in an asynchronous system, no consensus protocol can guarantee termination if even one node might crash. Time synchronization faces the same class of impossibility. Every practical solution (NTP, Lamport, vector clocks, TrueTime) is an engineering compromise — not a solution to the fundamental problem.

Consensus is the problem of getting multiple nodes to agree on a single value — even when some nodes crash, messages are lost, and the network partitions. It sounds simple until you realize that in an asynchronous system, you cannot distinguish a crashed node from a slow one. That impossibility (FLP theorem) means every practical consensus algorithm makes trade-offs: timeouts, leader election, quorum sizes. Paxos solved it first. Raft made it understandable.

Raft nodes are in one of three states: Follower, Candidate, or Leader. All nodes start as Followers. The election process is sequential:

1. Election trigger: a Follower does not hear from a Leader within the election timeout (randomized 150–300ms). It assumes the Leader has failed and becomes a Candidate.
2. Voting: the Candidate increments its term number, votes for itself, and sends RequestVote RPCs to all other nodes. A node grants its vote if it has not voted in this term and the Candidate's log is at least as up-to-date as its own.
3. Majority wins: if the Candidate receives votes from a majority (N/2 + 1), it becomes the new Leader and immediately sends heartbeats (AppendEntries with empty payload) to prevent new elections.
4. Split vote: if no Candidate wins a majority (two Candidates start simultaneously), all wait for a new randomized timeout and try again. The randomization makes simultaneous elections unlikely.

Why randomized timeouts matter: if all nodes used the same timeout, they would all call elections simultaneously every time the Leader failed. The randomization ensures one node almost always starts its election before others — preventing indefinite election loops.

Once a Leader is elected, it handles all client writes. Each write is appended to the Leader's log as an uncommitted entry. The Leader sends AppendEntries RPC to all Followers in parallel. Once a majority of nodes (including the Leader) have written the entry to their logs, the Leader commits the entry and applies it to its state machine. The Leader then notifies Followers on the next heartbeat, and they commit in turn.

Consistency guarantee: if an entry is committed, it will not be lost even if the Leader fails. This is because the entry exists on a majority of nodes, and any new Leader elected must have all committed entries (Raft's log matching property ensures this).

Single-database transactions are well-understood: BEGIN, do work, COMMIT or ROLLBACK. But when a business operation spans multiple services — each with its own database — how do you ensure atomicity? Either all succeed or all roll back. Two-Phase Commit (2PC) was the traditional answer; the Saga pattern is the modern one. Both have significant trade-offs, and choosing wrong means either blocking failures or inconsistent states that require manual repair.

Replication exists for two reasons: fault tolerance (if one copy dies, others survive) and performance (serve reads from nearby replicas). But the choice of replication strategy determines your consistency guarantees, write scalability, and failure modes. Single-leader is simple but limits write throughput. Multi-leader scales writes but creates conflicts. Leaderless avoids coordination but requires quorum math. Every database you use makes one of these choices — understanding which helps you predict its behavior under failure.

Replication lag is the delay between a write committed on the Leader and visible on a Follower. In a healthy cluster: milliseconds. Under high write load or a slow replica: seconds or minutes.

The read-after-write problem: a user writes data (posts a comment), then immediately reads from a replica that has not yet received the write. The user sees their comment missing — as if the write was lost. The write was not lost; it is on the Leader and will propagate. But the user's experience is broken.

Solutions: route reads to the primary for a short window (typically 60 seconds) after any write from that user. Track the replication position on each write and route reads to replicas that have caught up to at least that position. For critical reads (a user's own profile after editing), always read from the primary.

In a cluster of 100 nodes, how does each node know which others are alive? A central coordinator creates a single point of failure. Polling all nodes is O(N²). Gossip protocols solve this elegantly: each node periodically tells a few random peers what it knows. Information spreads exponentially — like a rumor. Within O(log N) rounds, every node has the full picture. No coordinator, no single point of failure, and bandwidth cost per node is constant regardless of cluster size.

Gossip tells nodes what other nodes exist and their health. But when a node comes back online after a failure, its data is stale. Three mechanisms reconcile data differences:

When you distribute data across N servers using modulo hashing (key % N), adding or removing one server forces nearly all keys to remap — a catastrophic cache miss storm or massive data migration. Consistent hashing solves this elegantly: when a node joins or leaves, only 1/N of keys move. It is the foundation of every distributed cache (Memcached), every distributed database (Cassandra, DynamoDB), and every CDN (Akamai). Understanding it is non-negotiable for distributed systems work.

Virtual nodes solve the imbalance problem. With only a few physical nodes on the ring, data distribution is uneven — a server assigned a large arc receives proportionally more traffic and stores more data than its neighbors. This is not theoretical; with 3 random positions, arcs are almost certainly unequal. Virtual nodes place each physical server at 100–200 positions (vnodes). The key space is now divided into hundreds of small arcs — statistical averaging ensures each physical server ends up with roughly equal load. Additional benefit: when a physical server is removed, its virtual nodes are distributed among all remaining servers — load spreads evenly, instead of falling on a single clockwise neighbor.