Microservices Communication: REST, Events and the API Gateway

A patient taps "refill my prescription" in a pharmacy app. Behind that tap the request touches an account service, a prescription service, a stock service and a payment service, and at the end the patient expects one clear answer. If the stock service takes 30 seconds to reply, what does the patient see, what do the three services in between do with their open connections, and what happens when the app retries? How services talk to each other is mostly the answer to those questions.

Synchronous calls: always set a deadline

A synchronous call (HTTP or gRPC) is the right default when the caller genuinely cannot continue without the answer, such as checking whether a prescription is valid. Its dangers all come from waiting. A call with no timeout can hold a thread or connection for as long as the other side hangs, and a handful of such calls can exhaust a pool and make a healthy service look dead. So every outbound call needs a timeout shorter than your own caller's patience, and ideally a deadline that shrinks as the request travels deeper.

Retries are the second tool, and also a hazard. Retrying immediately, from many clients at once, converts a short hiccup into a flood against a service that is trying to recover. The standard remedy is capped exponential backoff with jitter: wait a random time up to a ceiling that doubles after each failure, never beyond a cap, and give up after a few attempts.

import random


class Unavailable(Exception):
    pass


def call_with_retry(call, attempts=4, base=0.1, cap=2.0, sleep=None, rng=random.uniform):
    """Retry a call that may fail, using capped exponential backoff with full jitter."""
    for attempt in range(1, attempts + 1):
        try:
            return call()
        except Unavailable:
            if attempt == attempts:
                raise
            wait = rng(0, min(cap, base * 2 ** (attempt - 1)))
            print(f"attempt {attempt} failed, waiting {wait:.3f}s")
            sleep(wait)


# A fake pharmacy-stock service that fails twice, then answers.
responses = iter([Unavailable(), Unavailable(), {"sku": "AMX-500", "in_stock": 14}])


def check_stock():
    result = next(responses)
    if isinstance(result, Exception):
        raise result
    return result


rng = random.Random(42).uniform  # seeded only so this demo prints the same every run
print(call_with_retry(check_stock, sleep=lambda s: None, rng=rng))

attempt 1 failed, waiting 0.064s
attempt 2 failed, waiting 0.005s
{'sku': 'AMX-500', 'in_stock': 14}

The waits are random on purpose; the seed only freezes them for the demo. A circuit breaker goes one step further: after enough consecutive failures it stops calling the troubled service for a while and fails fast, so callers do not queue behind a dependency that is clearly down.

Retries need idempotency

If the payment call timed out, did the charge happen? You cannot know, and retrying a non-idempotent request risks charging twice. The fix is an idempotency key: the client generates a unique id per intended action and sends it with every attempt; the server remembers the key and returns the original result for repeats instead of acting again.

POST /payments
Idempotency-Key: refill-83412-attempt-group-1
Content-Type: application/json

{"accountId": "A-209", "amountMinor": 1850, "currency": "USD"}

Server rule:
  first time we see this key -> perform the charge, store (key -> response)
  key seen again             -> return the stored response, do nothing else
  key seen, different body   -> reject with 422 (client bug)

Asynchronous messaging: when the answer can wait

Not every step needs the patient to wait. Sending a pickup-ready text message, updating a stock forecast or writing an audit trail can happen later. For these, a service publishes a message to a broker and moves on, and consumers handle it when they can. Two benefits follow: the producer is not slowed or broken by a slow consumer, and new consumers can be added without changing the producer. Two costs follow too: you must live with eventual consistency, and delivery is typically at least once, so a consumer may see the same message twice and must be idempotent. A tool such as Apache Kafka is one common broker; the principles apply to any.

{
  "eventId": "9d4f2a10-6c1b-4a75-8a52-1e0c3f7d2b11",
  "type": "RefillApproved",
  "occurredAt": "2031-02-14T08:30:00Z",
  "version": 1,
  "data": {
    "refillId": "RF-83412",
    "pharmacyId": "PH-12",
    "readyForPickupBy": "2031-02-17T18:00:00Z"
  }
}

Include a unique eventId so consumers can deduplicate, a version so the format can evolve, and only the data consumers need. An event is a fact about the past (RefillApproved); a command (ApproveRefill) is a request addressed to one service. Mixing them up produces confusing designs where "events" are really remote procedure calls in disguise.

The API gateway: a front door, not a brain

Clients (mobile apps, websites) should not know your internal topology. An API gateway is a single entry point that routes requests, terminates TLS, checks authentication tokens, applies rate limits, and sometimes aggregates several backend calls into one response for a given client (often called a backend-for-frontend when one gateway is tailored to one client type). The rule that keeps it healthy: no business rules. Once the gateway decides whether a refill is allowed, it is a hidden service that every team must change.

  Mobile app
      |  HTTPS
      v
  +-------------+   routing, auth check,
  | API gateway |   rate limit, response shaping
  +-------------+
     |        |
     v        v
 +---------+ +--------------+     events     +----------+
 | Account | | Prescription | -------------> | Notifier |
 +---------+ +--------------+   (async)      +----------+

Common mistakes and the symptoms they cause

• No timeouts. Symptom: when one dependency stalls, every upstream service grows its thread pool usage until it stops answering, and the whole system appears to be down.
• Immediate retries everywhere. Symptom: a service that recovers is knocked over again by the burst of retries, so the outage lasts longer than the original fault.
• Retrying non-idempotent calls. Symptom: customers report double charges or duplicate shipments after a network blip.
• Assuming exactly-once delivery. Symptom: duplicate emails or double stock deductions after a consumer restarts. Deduplicate by eventId.
• A smart gateway. Symptom: every feature needs a gateway change and the gateway team becomes a bottleneck.

How this shows up in real projects and interviews

"What happens if the payment service is slow?" is one of the most common follow-up questions in a design interview, and the strong answer lists timeout, bounded retries with backoff and jitter, idempotency keys, a circuit breaker and a user-facing fallback, in that order. On real teams you rarely hand-roll these: libraries and service meshes provide them, but you must still choose the numbers. Pairing this lesson with REST API design covers how the synchronous contracts themselves should look.