A service that calls another service synchronously is only as reliable as that dependency. If the payment service times out, the checkout service times out. If the email service is slow, the registration endpoint is slow. Synchronous coupling is invisible until a dependency fails — and then it is catastrophically visible. Asynchronous communication breaks that coupling at the cost of complexity. The question every system designer must answer is deceptively simple: does the caller need the result right now, or just eventually?

Every distributed system interaction fits one of three patterns. Understanding which pattern you are using — and which you should be using — is the first step to designing reliable inter-service communication. Most teams default to request-response for everything, then retrofit async when latency problems appear. Designing for the right pattern from the start is cheaper.

REST is so ubiquitous that most engineers use it without knowing what it actually is. It is not a protocol. It is not a standard. REST is an architectural style — a set of constraints defined by Roy Fielding in his 2000 dissertation. Most APIs called "REST" violate at least two of its six constraints. Understanding what REST actually requires — and where most implementations deviate — is the difference between an API that ages well and one that requires constant versioning pain.

Idempotency is the property that matters most for reliability. If an operation is idempotent, clients can safely retry it after a network failure without producing duplicates. GET, PUT, and DELETE are inherently idempotent. POST is not — and that distinction drives how you design retry logic, payment APIs, and everything else that must not create duplicates.

REST on HTTP/1.1 with JSON is convenient — every developer knows it, every tool supports it, every debugger can read it. But convenience has a cost: JSON is verbose, HTTP/1.1 is request-response only, and text parsing is slower than binary. When you are making millions of service-to-service calls per second inside a datacenter, those costs compound. gRPC was built by Google for exactly this problem — high-throughput, low-latency, strongly-typed communication between internal services. It is not a replacement for REST. It is the right tool for a specific context.

REST was designed with servers deciding what data to return. GraphQL flips that relationship — clients specify exactly what they need and the server returns exactly that, no more and no less. This solves real problems: mobile clients that cannot afford to fetch large payloads they only partially use, frontend teams blocked waiting for backend teams to add fields to an endpoint, multiple round-trips to assemble a single view from separate resources. GraphQL is not always the answer. Its flexibility comes with costs that most teams underestimate until they are already in production.

HTTP was designed for documents — request something, receive it, connection closes. That model works for 99% of web interactions. But live chat, collaborative editing, real-time dashboards, and multiplayer games do not fit the request-response model. You need data to flow from server to client unprompted, or between clients in real time. Long polling was the first hack. Server-Sent Events was the clean HTTP solution for the server-to-client case. WebSockets were the right answer when both sides need to talk simultaneously.

In a synchronous system, service A asks service B for something and waits. Service A knows about service B. When service B slows down, service A slows down. Event-driven architecture inverts this. Service A emits an event — "an order was placed" — and does not know or care who reacts. Service B, C, and D each react independently. They can fail, be slow, or not exist yet — service A is unaffected. This decoupling is genuinely powerful. It is also a source of operational complexity that teams consistently underestimate until they are debugging a broken saga at 2am.

Most systems use the same data model for reading and writing. This works until read and write patterns diverge enough that optimizing one hurts the other. CQRS separates them: a write model optimized for consistency and validation, and a read model optimized for query performance. The cost is eventual consistency — the read model is updated asynchronously from the write model.

Distributed transactions across multiple services cannot use traditional database 2PC (two-phase commit) — it is impractical at service boundaries. The Saga pattern breaks a distributed transaction into a sequence of local transactions, each publishing an event to trigger the next step. If any step fails, compensating transactions run in reverse to undo completed steps.

An API is a promise. Unlike internal code which you can refactor whenever you want, an API has consumers you may not control. Change it carelessly and you break systems you did not write, maintained by teams you may not even know exist. The practices in this chapter exist because APIs live longer than the engineers who designed them. The decisions you make at version one — around idempotency, pagination, error format, deprecation — will either protect you or haunt you for years.

Every API error must return the same structure. Inconsistent errors force clients to write defensive parsing logic for every endpoint. One bad error format early in an API's life creates years of backward compatibility debt. Define the contract once and enforce it across every endpoint from day one.

{
  "error": {
    "code": "PAYMENT_DECLINED",
    "message": "The card was declined by the issuer",
    "details": [{ "field": "card_number", "issue": "Invalid" }],
    "request_id": "req_abc123",
    "documentation_url": "https://api.example.com/docs/errors/PAYMENT_DECLINED"
  }
}

X-RateLimit-Limit: 1000        # requests allowed per window
X-RateLimit-Remaining: 743     # requests remaining in current window
X-RateLimit-Reset: 1703721600  # Unix timestamp when window resets
Retry-After: 30                # seconds to wait (on 429 response only)