Distributed Systems¶
A distributed system coordinates components that communicate over a network. Messages can be delayed, duplicated, reordered, or lost; components can fail independently; and no perfectly synchronized global clock is available.
Failure model first¶
An algorithm's guarantee depends on its assumed failures: crash-stop, crash- recovery, omission, timing, or Byzantine behavior. “Handles failure” is too vague without this model.
Delivery and processing¶
Labels such as at-most-once and at-least-once describe delivery attempts, not automatic end-to-end business effects. Exactly-once effects require coordinated state or idempotent/deduplicated processing under a defined boundary.
An idempotency key should identify one logical operation, have a lifecycle, and bind to the operation's parameters and result. Retrying with a new key defeats deduplication.
Consistency¶
Linearizability makes each operation appear atomic between invocation and response. Sequential, causal, eventual, and session guarantees are different models. Eventual consistency alone says convergence occurs after updates stop; it does not define conflict resolution or a useful time bound.
The CAP result concerns consistency and availability during a network partition. It is not a general instruction to choose only two properties during normal operation.
Time and ordering¶
Use monotonic time for local durations. Wall clocks can jump and differ between machines. Logical clocks capture causal or total ordering properties without pretending to provide perfectly synchronized physical time.
Reliable workflows¶
- assign stable operation and message identifiers;
- make consumers idempotent or deduplicate durably;
- use bounded retries with backoff, jitter, and deadlines;
- use a transactional outbox when a local commit and durable event publication must agree;
- define poison-message and dead-letter handling;
- monitor lag, saturation, retries, and reconciliation with bounded-cardinality logs, metrics, and traces;
- test partial failures, not only complete outages.
Exercises¶
- Explain why a client timeout does not prove the server failed to commit.
- Design an idempotent message consumer.
- Contrast read-your-writes with linearizability.