Post cover image

September 3, 2026

10 Microservices Lessons You Only Learn After a Bad Outage

We had an outage once that took four people three hours to trace back to one root cause: a retry loop in Service A hammering Service B…

By Prince kumar Maurya

6 min read

We had an outage once that took four people three hours to trace back to one root cause: a retry loop in Service A hammering Service B, which was already struggling, which made Service A retry more, which made Service B struggle more. Nobody had done anything wrong exactly. We'd just never actually implemented backoff, because "retries" felt like a solved problem we didn't need to think about twice.

That's the thing about microservices. Everyone can draw the boxes-and-arrows diagram. Service A calls Service B calls Service C, message queue in the middle, done. The diagram is the easy part. The stuff that actually keeps a distributed system alive under real load is almost never in the diagram, and it's almost never taught properly either — you mostly learn it by getting paged at 2 AM once.

Here are ten of those things. Some are patterns with names, some are just habits that separate teams who've been burned from teams who haven't yet.

1. Idempotency keys on every mutating endpoint

If a client calls POST /payments and the response times out, what does the client do? Retries, obviously. Now you might have charged someone twice.

@PostMapping("/payments")
public ResponseEntity<Payment> createPayment(
        @RequestHeader("Idempotency-Key") String idempotencyKey,
        @RequestBody PaymentRequest request) {
    Optional<Payment> existing = paymentRepository.findByIdempotencyKey(idempotencyKey);
    if (existing.isPresent()) {
        return ResponseEntity.ok(existing.get());
    }
    Payment payment = paymentService.process(request, idempotencyKey);
    return ResponseEntity.ok(payment);
}

Client generates a UUID once per logical operation, sends it every time it retries that same operation. Server checks if it's seen that key before returning the same result instead of doing the work again. This one's not optional if money or anything irreversible is involved — I'd genuinely put it above almost everything else on this list.

2. Exponential backoff with jitter, not just "retry 3 times"

Naive retry logic looks like this:

for (int i = 0; i < 3; i++) {
    try {
        return callDownstream();
    } catch (Exception e) {
        Thread.sleep(1000);
    }
}

Which is exactly what caused the outage I mentioned at the top. Every failing client retries at the same interval, in sync, so your downstream service gets hit with synchronized waves of traffic instead of a smoothed-out trickle. Add jitter:

long baseDelay = (long) (1000 * Math.pow(2, attempt));
long jitter = ThreadLocalRandom.current().nextLong(baseDelay / 2);
Thread.sleep(baseDelay + jitter);

Spreads retries out across time instead of all landing on the same beat. Resilience4j does this for you out of the box if you don't want to hand-roll it — I'd recommend that route unless you have a specific reason not to.

3. Correlation IDs threaded through every log line

Debug a bug that spans four services without a correlation ID and you'll spend an hour trying to line up timestamps across four different log streams, guessing which log line in Service C corresponds to which request in Service A. Do it with one, and it's a single search query.

@Component
public class CorrelationIdFilter extends OncePerRequestFilter {
    @Override
    protected void doFilterInternal(HttpServletRequest req, HttpServletResponse res, FilterChain chain)
            throws ServletException, IOException {
        String correlationId = req.getHeader("X-Correlation-Id");
        if (correlationId == null) correlationId = UUID.randomUUID().toString();
        MDC.put("correlationId", correlationId);
        res.setHeader("X-Correlation-Id", correlationId);
        try {
            chain.doFilter(req, res);
        } finally {
            MDC.clear();
        }
    }
}

The header has to be propagated forward on every outbound call too, not just logged locally — that part's easy to forget and it makes the whole thing useless if you skip it.

4. The outbox pattern for "update database and publish event" consistency

Classic bug: you update a row in Postgres, then publish a Kafka event about it. Except the database commit succeeds and the Kafka publish fails (network blip, broker down, whatever), and now your database says one thing and the rest of your system never heard about it.

The outbox pattern writes the event to a table in the same transaction as the actual data change:

BEGIN;
UPDATE orders SET status = 'SHIPPED' WHERE id = 123;
INSERT INTO outbox_events (aggregate_id, event_type, payload)
  VALUES (123, 'ORDER_SHIPPED', '{"orderId": 123}');
COMMIT;

A separate poller (or Debezium reading the database's write-ahead log, if you want to go fully async) picks up rows from outbox_events and actually publishes them to Kafka, marking them as sent. Now the database update and the event are atomic — either both happen or neither does. This one took me embarrassingly long to actually understand the point of, until I saw the alternative fail in production.

5. Circuit breakers that actually open

Everyone name-drops "circuit breaker pattern" in interviews. Fewer people have actually configured one to genuinely trip:

@CircuitBreaker(name = "inventoryService", fallbackMethod = "fallbackInventory")
public InventoryResponse checkInventory(String sku) {
    return inventoryClient.check(sku);
}
public InventoryResponse fallbackInventory(String sku, Exception e) {
    return InventoryResponse.unavailable(sku);
}
resilience4j.circuitbreaker:
  instances:
    inventoryService:
      failure-rate-threshold: 50
      wait-duration-in-open-state: 30s
      sliding-window-size: 10

The point isn't the annotation — it's deciding what the fallback actually does. Returning a cached last-known value? A generic "temporarily unavailable" response? Degrading gracefully instead of just failing the whole request chain? That decision matters more than the config, and it's the part most tutorials skip entirely.

6. Bulkhead isolation — one slow dependency shouldn't sink the whole service

If Service A calls three downstream services and one of them gets slow, a shared thread pool means that one slow dependency can eventually consume every available thread, blocking requests that had nothing to do with the slow service at all.

resilience4j.bulkhead:
  instances:
    slowDependency:
      max-concurrent-calls: 10
      max-wait-duration: 5ms

Named after the physical bulkheads on a ship — flood one compartment, the rest stay dry. Give each downstream dependency its own bounded pool of concurrent calls so a struggling dependency can only ever hurt itself, not everything calling anything else.

7. Distributed tracing that actually gets used, not just installed

Half the teams I've seen have Jaeger or Zipkin running. Maybe a fifth actually check it regularly. It usually gets set up once during an onboarding sprint and then nobody opens it again until there's a really bad incident.

management:
  tracing:
    sampling:
      probability: 1.0
otel:
  exporter:
    otlp:
      endpoint: http://otel-collector:4317

The habit that actually matters: when something's slow, open the trace first, before opening logs. A trace tells you which specific hop in a five-service chain ate the latency, in seconds, instead of you guessing and checking logs one service at a time.

8. Contract testing between services, not just each service's own unit tests

Service A's team changes a response field name. Service A's own tests all pass — of course they do, nothing inside Service A actually broke. Service B, which consumes that field, breaks in production three days later when someone finally deploys.

Consumer-driven contract testing (Pact is probably the most common tool) catches this before deploy:

@Pact(consumer = "order-service")
public RequestResponsePact createPact(PactDslWithProvider builder) {
    return builder
        .given("inventory exists for sku ABC123")
        .uponReceiving("a request for inventory")
        .path("/inventory/ABC123")
        .willRespondWith()
        .status(200)
        .body(newJsonBody(body -> body.stringType("sku", "ABC123").numberType("quantity", 10)).build())
        .toPact();
}

The consumer defines what it expects; the provider's CI pipeline runs that contract against its actual implementation before merge. It genuinely closes a gap that unit tests, by design, can never catch — because unit tests only know about the service they're inside.

9. Rate limiting at the gateway, scoped per client, not globally

A global rate limit on your API gateway protects against nothing in particular. One misbehaving client (a buggy retry loop, a scraper, whatever) can still eat the entire limit and starve every other legitimate client, because the limit doesn't know the difference between them.

spring:
  cloud:
    gateway:
      routes:
        - id: orders-route
          uri: lb://order-service
          filters:
            - name: RequestRateLimiter
              args:
                redis-rate-limiter.replenishRate: 10
                redis-rate-limiter.burstCapacity: 20
                key-resolver: "#{@apiKeyResolver}"

Scoping the limiter to a key resolver (API key, client ID, whatever identifies the caller) means one bad actor gets throttled without dragging everyone else down with them.

10. Saga pattern for distributed transactions that actually need to roll back

There's no such thing as a real two-phase commit across microservices in most practical setups — you can't wrap a payment service call and a shipping service call in one atomic database transaction the way you could with two tables in the same database. The saga pattern handles this with a sequence of local transactions plus explicit compensating actions if something downstream fails:

1. Order Service: create order (pending)
2. Payment Service: charge card
   → if fails: Order Service compensates → cancel order
3. Inventory Service: reserve stock
   → if fails: Payment Service compensates → refund charge
                Order Service compensates → cancel order
4. Order Service: mark order confirmed

Every forward step needs a matching compensating step, and you have to actually build and test the failure paths — not just the happy path, which is the part almost everyone skips until the first real failure in production forces the issue.

Why none of this shows up in tutorials

Tutorials build one service, maybe two, calling each other over localhost, no real network partitions, no actual load. Every one of these ten things only starts to matter once you have real traffic, real failures, and enough scale that "it works when I test it manually" stops being good enough. Nobody teaches this stuff up front because it genuinely sounds like overengineering right up until the day it isn't.

If you only fix one thing off this list: add idempotency keys to anything that moves money, inventory, or anything else you genuinely can't undo. It's the cheapest insurance on this entire list against the most expensive kind of bug.