The Art of Debugging Non-Deterministic Bugs

Have you ever encountered a bug that seems to appear at random? It works perfectly on one machine, fails in production, and mysteriously disappears as soon as debugging begins. These are often called non-deterministic bugs, but they're rarely random. They simply depend on variables that are not yet understood. This article presents a practical methodology to transform an unpredictable bug into a reproducible one — the first step toward fixing it.

A Bug Is Rarely Random

Software doesn't make random decisions. Given the same inputs, the same state, and the same execution order, it will produce the same result every time.

So why do some bugs seem unpredictable?

Because not every variable is under control. The scheduling of threads, network latency, CPU load, memory pressure, garbage collection, cache state, or even the order in which events arrive can all influence the execution.

A bug only appears non-deterministic because one or more of these variables remain hidden.

The goal of debugging is not to "get lucky" and reproduce the bug. It is to identify and progressively eliminate unknowns until the bug becomes deterministic.

Reduce the Search Space

At first, every hypothesis seems possible.

Was it the network? A race condition? A timeout? A corrupted cache? A specific input? A memory issue? A recent deployment?

The search space is simply too large.

The purpose of debugging is to progressively eliminate possibilities until only one explanation remains.

Thousands of possibilities
        ↓
Hundreds
        ↓
Dozens
        ↓
A handful
        ↓
One reproducible scenario

Thousands of possibilities
        ↓
Hundreds
        ↓
Dozens
        ↓
A handful
        ↓
One reproducible scenario

Every experiment should answer a simple question:

What hypothesis can be eliminated?

A failed experiment is not wasted effort. It reduces uncertainty and brings the root cause one step closer.

A Practical Methodology

Once the objective is clear — reduce the search space — the next step is to apply a systematic approach. Each experiment should either increase the reproduction rate or eliminate a hypothesis.

Step 1: Observe Before Changing Anything

The first instinct is often to modify the code. Resist that temptation.

Instead, collect as much information as possible:

• Add meaningful logs.
• Record timestamps.
• Capture thread IDs.
• Monitor resource usage.
• Preserve stack traces and error messages.

The goal isn't to fix the bug yet — it's to understand the conditions under which it occurs.

Step 2: Identify Reproduction Conditions

Ask simple questions:

• Does it only happen in production?
• Does it require high CPU load?
• Is it related to a specific operating system?
• Does it appear after several hours?
• Does it depend on the input data?

Every recurring pattern is a valuable clue.

Step 3: Remove Variables

Simplify the environment as much as possible.

Reduce the number of moving parts:

• Run with a single thread.
• Use a single request.
• Disable caches.
• Use the same dataset.
• Fix the random seed.
• Freeze configuration changes.

Every variable removed makes the bug easier to reason about.

Step 4: Amplify the Problem

Sometimes the bug is simply too rare.

Instead of waiting for it to happen, make it happen more often.

• Increase concurrency.
• Add artificial network latency.
• Generate more memory allocations.
• Run stress tests.
• Execute the same scenario thousands of times.

A bug that occurs once every ten minutes is difficult to study. A bug that occurs every five seconds is much easier to investigate.

Step 5: Isolate the Root Cause

Keep reducing the problem until only the essential code remains. A million-line application can often be reduced to a hundred-line reproduction. Once a minimal reproducible example exists, the root cause usually becomes obvious.

Build Your Debugging Toolbox

No single tool can solve every non-deterministic bug. The key is choosing the one that provides the missing piece of information.

Logging

Good logs explain what happened, when, and under which conditions.

A useful log entry often includes:

• A timestamp
• A thread or goroutine ID (when applicable)
• A request or correlation ID
• Relevant state information

The goal is not to generate more logs, but better ones.

Metrics

Some bugs don't leave stack traces.

Memory leaks, CPU spikes, queue growth, or increasing response times often appear first as abnormal metrics.

Monitoring trends can reveal problems long before they become failures.

Tracing

In distributed systems, a single request may cross dozens of services.

Distributed tracing reconstructs the complete execution path, making it much easier to identify where things started to go wrong.

Dumps and Profilers

When the application cannot explain what's happening, a memory dump, thread dump, or CPU profile often can.

These snapshots provide a view of the application's internal state at a specific point in time.

Assertions

Assertions help detect invalid states as early as possible.

Instead of allowing the application to fail much later, they make incorrect assumptions visible immediately, often much closer to the actual root cause.

Stress and Load Testing

Some bugs only appear under pressure.

Increasing concurrency, traffic, or memory usage can dramatically increase the reproduction rate, making investigation much easier.

The most valuable debugging tool isn't the one that produces the most information. It's the one that removes the most uncertainty.

Beware of the Traps

Non-deterministic bugs have a frustrating habit: they often disappear as soon as they are observed.

The Heisenbug Effect

Adding a log statement, setting a breakpoint, or attaching a debugger changes the execution.

A few extra milliseconds may be enough to alter the scheduling of threads, hide a race condition, or change the timing of a network request.

Sometimes, the very act of observing the bug makes it disappear.

Confirmation Bias

Once a hypothesis seems plausible, it's tempting to interpret every observation as supporting it.

Instead, actively look for evidence that disproves the current hypothesis.

Eliminating the wrong explanations is just as valuable as finding the right one.

Changing Too Many Variables

When several things are modified at once, it's impossible to know which change affected the outcome.

A good debugging experiment changes only one variable at a time.

Assuming Correlation Means Causation

Just because a bug started after a deployment doesn't necessarily mean the deployment caused it.

Likewise, a fix that appears to work once may simply have changed the timing enough to hide the issue.

Always verify a hypothesis through repeated, reproducible experiments.

Debugging is not about proving a hypothesis right. It's about trying to prove it wrong.

A Debugging Checklist

When facing a bug that seems impossible to reproduce, systematically question every assumption.

Environment

• Does it only happen in production?
• Does it depend on the operating system or hardware?
• Does it occur in containers but not locally?
• Does it depend on a specific version of a dependency?

Timing

• Does it happen only under high CPU load?
• Does adding a log or a breakpoint make it disappear?
• Could it be a race condition?
• Does it occur after running for a long time?

Data

• Does it depend on a particular input?
• Is there a minimum or maximum data size that triggers it?
• Can the same input reproduce the issue consistently?

Infrastructure

• Is the network involved?
• Could a timeout or retry mechanism be responsible?
• Is a cache hiding or exposing the problem?
• Does the order of incoming events matter?

Memory

• Is the application under memory pressure?
• Could the garbage collector influence the timing?
• Is there a memory leak or resource exhaustion?

Experiments

• Can the environment be simplified?
• Can the reproduction rate be increased?
• Can one more variable be eliminated?
• Is there a minimal reproducible example?

Key Takeaways

• A bug is rarely random. It usually depends on variables that are not yet understood or controlled.
• The primary goal of debugging is to make the bug reproducible. Once it can be reproduced consistently, finding the root cause becomes much easier.
• Treat debugging like a scientific experiment. Form hypotheses, change one variable at a time, and eliminate possibilities through observation.
• Use the right tools to reduce uncertainty. Logs, metrics, tracing, profilers, and stress tests each reveal different pieces of the puzzle.
• Challenge assumptions, not just code. The root cause often lies in an assumption about timing, concurrency, data, or the environment that turns out to be false.

Hoping you'll be able to tackle any non-determinism problem now. If you liked this article, feel free to clap it, comment it or share it. To stay updated with my content, you can follow me on Medium.