Post cover image
Photo From Unsplash By Getty Images

October 12, 2026

8 Distributed Systems Mistakes That Turn Small Bugs Into Production Disasters

Most production outages don’t start with a big failure. They start with one slow dependency, one retry loop, or one message that won’t…

By The Latency Gambler

4 min read

Most production outages don't start with a big failure. They start with one slow dependency, one retry loop, or one message that won't process. Then the system's own reactions make it worse.

Here are eight mistakes I see over and over, what each one does, and the small Go fix for it. If you're newer to backend work, this is the checklist I wish someone had handed me early.

1. No timeouts

Go's default http.Client has no timeout. If a dependency hangs, your goroutine hangs with it. Multiply that by a few thousand requests and you're out of connections and memory, even though your own code is fine.

client := &http.Client{Timeout: 2 * time.Second}

// better: per-request control, so callers can cancel too
ctx, cancel := context.WithTimeout(r.Context(), 2*time.Second)
defer cancel()

req, _ := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
resp, err := client.Do(req)

A timeout turns an unbounded wait into a failure you can handle. Don't set it too aggressively, though. A 50ms timeout on a call that normally takes 80ms will create failures out of thin air.

2. Retrying blindly

Retries help with transient failures. They're dangerous when done carelessly. If 1,000 requests fail and each retries 3 times, the struggling service now gets 4,000 requests. You've DDoS'd your own dependency right when it was trying to recover.

Client ──► call fails ──► retry ──► retry ──► retry
                                │
               1,000 failures × 3 retries = 4,000 extra hits
               on a service that's already struggling

A safe retry has a limit, exponential backoff, and jitter so clients don't all retry in sync:

func retry(ctx context.Context, attempts int, fn func() error) error {
	var err error
	delay := 100 * time.Millisecond

        for i := 0; i < attempts; i++ {
          if err = fn(); err == nil {
           return nil
          }
          wait := delay + time.Duration(rand.Int63n(int64(delay))) // jitter
          select {
          case <-time.After(wait):
          case <-ctx.Done():
           return ctx.Err()
          }
          delay *= 2
         }
         return fmt.Errorf("gave up after %d attempts: %w", attempts, err)
}

Only retry operations that are safe to repeat. That brings us to mistake 6.

3. No circuit breaker

If a dependency keeps failing, keep calling it and you just add load while your own latency climbs. A circuit breaker stops the calls for a while and lets the dependency recover.

Closed ────────► Open ────────► Half-open
 (normal)   too many  (fail fast)  cooldown   (test a call)
    ▲       failures                  over          │
    └────────────── success ◄─────────────────────┘
var ErrOpen = errors.New("circuit open")

type Breaker struct {
 mu          sync.Mutex
 failures    int
 maxFailures int
 openedAt    time.Time
 cooldown    time.Duration
 open        bool
}

func (b *Breaker) Do(fn func() error) error {
 b.mu.Lock()
 if b.open && time.Since(b.openedAt) < b.cooldown {
  b.mu.Unlock()
  return ErrOpen // fail fast, don't touch the dependency
 }
 b.mu.Unlock() // cooldown over: let this call through as a test
 err := fn()
 b.mu.Lock()
 defer b.mu.Unlock()
 if err != nil {
  b.failures++
  if b.failures >= b.maxFailures || b.open {
   b.open = true
   b.openedAt = time.Now()
  }
  return err
 }
 b.failures, b.open = 0, false
 return nil
}

This is simplified. Production breakers limit how many test calls get through in half-open. Thresholds also need tuning and monitoring, or the breaker will trip when it shouldn't.

4. Sharing one resource pool for everything

If every dependency draws from the same pool of goroutines or connections, one slow service can eat all of it. Then unrelated features go down too. Bulkheads give each dependency its own limited pool.

┌─► pool A (10) ─► Service A  ← slow, only this pool fills
  requests ──────┼─► pool B (10) ─► Service B  ← still healthy
                 └─► pool C (10) ─► Service C  ← still healthy
type Bulkhead struct{ slots chan struct{} }

func NewBulkhead(n int) *Bulkhead {
 return &Bulkhead{slots: make(chan struct{}, n)}
}

func (b *Bulkhead) Do(fn func() error) error {
 select {
 case b.slots <- struct{}{}:
  defer func() { <-b.slots }()
  return fn()
 default:
  return errors.New("bulkhead full, rejecting")
 }
}

The tradeoff is extra resources and more things to configure.

5. Letting producers outrun consumers

When work arrives faster than you can process it, an unbounded queue just moves the crash to later, usually as an out-of-memory kill. Backpressure means slowing down or rejecting work on purpose.

Producer (fast) ──► [ ■ ■ ■ ■ ■ ] ──► Consumer (slow)
                       queue filling
                           │
              full? reject or slow the producer
var ErrOverloaded = errors.New("overloaded, try later")

jobs := make(chan Job, 100) // bounded on purpose

func submit(j Job) error {
 select {
 case jobs <- j:
  return nil
 default:
  return ErrOverloaded // tell the caller now, e.g. HTTP 429/503
 }
}

A fast "no" is better than a slow, silent collapse.

6. Assuming a request runs only once

In distributed systems, duplicates are normal. Networks drop acknowledgements, clients retry, queues redeliver. If processing the same request twice charges a customer twice, that's a design bug. Idempotency means the same request produces the same result no matter how many times it arrives.

func (s *Store) Charge(ctx context.Context, key string, amountCents int) error {
	res, err := s.db.ExecContext(ctx, `
		INSERT INTO charges (idempotency_key, amount_cents)
		VALUES ($1, $2)
		ON CONFLICT (idempotency_key) DO NOTHING`,
		key, amountCents)
	if err != nil {
		return err
	}
	if n, _ := res.RowsAffected(); n == 0 {
		return nil // already processed, safe to ignore
	}
	// first time seeing this key: do the real work here
	return nil
}

The client sends a unique key with each logical request, and the database's unique constraint does the deduplication. In a real system, do the side effect in the same transaction as the insert.

7. Retrying poison messages forever

Some messages will never succeed. Bad data, a bug, a missing record. If you keep retrying them, they block everything behind them. After a limit, move them to a dead letter queue and look at them later.

Messages ──► Processing ──► Success
                  │
                fails N times
                  ▼
           Dead Letter Queue ──► investigate later
const maxAttempts = 5

func process(msg Message, queue, dlq chan<- Message) {
 if err := handle(msg); err == nil {
  return
 }
 msg.Attempts++
 if msg.Attempts >= maxAttempts {
  dlq <- msg // park it, don't block the pipeline
  return
 }
 queue <- msg
}

A DLQ nobody watches is just a slower way to lose data. Alert on its size and have a process for handling what lands in it.

8. Flying blind

This one makes the other seven worse. You can't make failures predictable if you can't see them. The basics are four things: logs (what happened), metrics (how often), traces (where the request went), and alerts (when a human needs to care).

logger := slog.New(slog.NewJSONHandler(os.Stdout, nil))

start := time.Now()
err := callPayments(ctx, orderID)

logger.Info("payments call",
 "order_id", orderID,
 "request_id", requestID,
 "duration_ms", time.Since(start).Milliseconds(),
 "error", err,
)

Structured logs with a request ID let you follow one request across services. Without that, you're debugging in the dark.

The short version

Mistake Fix No timeouts Set one on every outbound call Blind retries Limit, backoff, jitter Hammering a failing service Circuit breaker One shared pool Bulkheads Producers outrun consumers Bounded queues, reject early Duplicates break things Idempotency keys Poison messages Dead letter queue Can't see failures Logs, metrics, traces, alerts

None of these are advanced. They're boring, and that's why they work. Most disasters I've read postmortems for weren't caused by one huge mistake. They came from two or three of these missing at the same time, so small bugs spread. Pick the one your system is missing today and fix that first.