AI Helped Me Debug a Production Bug by Explaining It Before Fixing It

The Fastest Way to Fix a Bug Is Often the Wrong One

When something breaks in production, there's an instinct that kicks in immediately: find the fix, ship the fix, close the ticket. And now that AI tools exist, that instinct has a new shortcut. Paste the error, ask for a fix, copy the answer.

Sometimes it works beautifully. Sometimes it fixes the symptom, passes the check, and leaves half the bug alive in production.

So I tried something different. I asked AI to explain the bug first, and not to fix anything yet. No code, no patches, just: tell me what you think happened, and how we could prove it.

That one change turned a messy incident into a clean investigation. In this article, I'll show you exactly what happened, with a small example you can run yourself, the prompts I used, what AI got right, and where it needed a human.

A Quick Note on the Example

To keep this honest and reproducible, I built a small simulation of a very common production problem rather than using a real client system. The pattern (a nightly sync job that creates duplicate records) is something many teams, including Siebel and CRM integration teams, will recognise. The numbers you'll see below come from an actual run of the code, which is included at the end so you can try it yourself.

The Scenario: The Job That Said "Success"

Imagine a nightly job that copies new customers from a CRM into a billing system. It runs at 1 AM. Nobody watches it.

The next morning, finance reports that some customers appear twice in billing. Duplicates mean double invoices, angry customers and an awkward meeting.

Here's what the team could see:

The log looked perfectly healthy:

INFO  sync started: 20 records
WARN  timeout creating C-007 (attempt 1), retrying
INFO  created C-007 (attempt 2)
WARN  timeout creating C-013 (attempt 1), retrying
INFO  created C-013 (attempt 2)
INFO  sync complete: 20 processed, 17 created, 3 skipped, 0 errors

INFO  sync started: 20 records
WARN  timeout creating C-007 (attempt 1), retrying
INFO  created C-007 (attempt 2)
WARN  timeout creating C-013 (attempt 1), retrying
INFO  created C-013 (attempt 2)
INFO  sync complete: 20 processed, 17 created, 3 skipped, 0 errors

And the ticket said:

"The source system had 15 new customers and 5 that already existed in billing. Billing now has 24 rows. It should have 20."

Zero errors. Four duplicates. A green log hiding a red problem. If you've read my story about The CRM Report Bug That Took 3 Weeks to Find, you'll know this feeling: the system insists everything is fine, and the data disagrees.

Why I Asked AI to Explain Instead of Fix

Let's think about what would have happened with "fix it" as the prompt.

The log has two obvious warnings about timeouts and retries. Almost any quick fix, from a human or an AI, would focus there. Add a delay. Increase the timeout. Handle the retry better. Done.

And that would feel like a solution. But it would only address part of the problem. I'll prove that in a moment.

Explaining first forces a different question: "What story fits ALL the evidence?" Not just the loudest clue.

The Explain-Before-Fix Workflow

Here's the simple process I now follow.

Collect evidence: logs, counts, timelines, recent changes, relevant code.
Ask AI to explain: hypotheses only, no code yet.
Rank the hypotheses: which ones fit every fact?
Verify each one: reproduce it, test it, try to disprove it.
Fix every cause: not just the first one you find.
Add a regression test: so it never comes back.

Notice that AI only appears in step 2 and perhaps step 6. Humans' own steps 3, 4 and 5. That split matters, and I'll come back to it.

Round One: AI Root Cause Analysis Using Only Logs

I gave the AI two things: the log above and the ticket text. No code. And I wrote the prompt very deliberately.

The Prompt

Context: A nightly job copies new customers from a CRM to a billing system.

Evidence: [log pasted here] and [ticket text pasted here].

Please do NOT suggest code fixes yet. Instead:
1. Explain what the numbers tell us.
2. List possible causes, ranked by how well they fit the evidence.
3. For each cause, say what would prove it wrong.
4. Tell me what extra information would change your answer.

Context: A nightly job copies new customers from a CRM to a billing system.

Evidence: [log pasted here] and [ticket text pasted here].

Please do NOT suggest code fixes yet. Instead:
1. Explain what the numbers tell us.
2. List possible causes, ranked by how well they fit the evidence.
3. For each cause, say what would prove it wrong.
4. Tell me what extra information would change your answer.

What the AI Said

Here's a condensed version of the response:

What the numbers say:_ The source had 15 new customers, but the log says 17 were created. That's 2 more than expected. Billing also grew by 19 rows (24 minus 5 existing), which is 2 more than the 17 the log claims were created. So there are two separate gaps of 2._

Hypothesis A (strong):_ When a timeout happens, the first write may have actually succeeded before the response was lost. The retry then creates a second copy. This fits the two timeout warnings and explains the 2 extra rows in billing beyond what the log counted._

Hypothesis B (likely, but unproven):_ Two customers were treated as_ new by the job when they already existed, which would explain why the log says 17 created instead of 15. Something in the "does this customer already exist?" check must have failed for two records. Possible reasons include capital letters, extra spaces or different formats.

To disprove A:_ check whether the duplicated rows are exactly the two customers that timed out. To disprove B: find the 2 customers the job created that already existed, and compare their identifying fields._

What I'd need:_ the code of the existence check, and the identifiers of the affected records._

Why This Response Impressed Me

Look at what it did with only a log and a ticket.

It did arithmetic on the evidence. The two gaps of 2 were the clue. A fix-first approach would have seen the timeout warnings, explained two duplicates, and stopped, with two more still unexplained.

It separated what it knew from what it guessed. Hypothesis A was strong. Hypothesis B was flagged as "likely, but unproven."

It told me how to disprove its own ideas. That is what separates an investigator from a guesser.

And it asked for what was missing instead of inventing it.

Next, I pasted the relevant code: the job's loop and the existence check. I asked the same question with the additional evidence.

The AI pointed to two specific things:

First, the retry. The job calls the billing system's create function, and if a timeout occurs, calls it again. But the create function isn't idempotent. In plain language, calling it twice creates two records. If the first call succeeded but the reply got lost, the retry makes a duplicate.

Second, the existence check. The job builds a set of existing emails and compares them using exact text matching. So Priya.Sharma@Acme.com and priya.sharma@acme.com count as different customers. So does raj.k@acme.com with a trailing space.

existing = {r["email"] for r in billing.rows}   # exact match only
if rec["email"] in existing: ...                 # "Raj@x.com " != "raj@x.com"

existing = {r["email"] for r in billing.rows}   # exact match only
if rec["email"] in existing: ...                 # "Raj@x.com " != "raj@x.com"

Two causes. Two mechanisms. Exactly matching the two gaps.

What Is Idempotency? (A Beginner Explanation)

You'll hear this word a lot in integrations, so let me make it simple.

An operation is idempotent if doing it twice has the same effect as doing it once. Pressing the "lift call" button in an elevator is idempotent: pressing it five times doesn't summon five lifts. Transferring money without a unique transaction ID is not idempotent: send the request twice and the money moves twice.

Any time a job can retry, the operation it retries should be idempotent. This is one of the most valuable lessons in integration work, and I'm surprised how rarely it's taught early.

Verifying Before Trusting: The Step AI Can't Do for You

Here's where I want to stop and emphasise something. The AI's explanation sounded convincing. That's exactly why it needed verification.

I wrote about this in I Almost Sent My Manager an AI-Written Lie. A confident explanation isn't evidence. A reproduced result is.

So I ran each hypothesis separately.

This is the real output from the experiment code:

Original code                 -> 24 rows (duplicates: 4)
Fix ONLY the retry problem    -> 22 rows (duplicates: 2)
Fix ONLY the email matching   -> 22 rows (duplicates: 2)
Fix BOTH                      -> 20 rows (duplicates: 0)

Original code                 -> 24 rows (duplicates: 4)
Fix ONLY the retry problem    -> 22 rows (duplicates: 2)
Fix ONLY the email matching   -> 22 rows (duplicates: 2)
Fix BOTH                      -> 20 rows (duplicates: 0)

Read those four lines carefully, because they carry the whole lesson.

If I had fixed only the retry, which is the obvious fix from the logs, two duplicates would remain. The log would still say success. Finance would be unhappy again next week.

If I had fixed only the email matching, two duplicates would remain, caused by the timeouts.

Only fixing both gave the correct result. Each hypothesis explained exactly half of the problem, and the two together explained everything.

The Fixes

Neither fix is complicated.

Fix 1: Make the Retry Safe

Give each customer a stable unique ID and make the write idempotent, so writing the same ID twice does nothing the second time:

if idempotent and any(r.get("id") == rec["id"] for r in rows):
    pass   # already written, do nothing

if idempotent and any(r.get("id") == rec["id"] for r in rows):
    pass   # already written, do nothing

In real systems, this usually means sending an idempotency key or using an "upsert" (update if it exists, insert if it doesn't).

Fix 2: Normalise Before Comparing

Clean the identifying value before matching:

def norm(email):
    return email.strip().lower()

def norm(email):
    return email.strip().lower()

Trim spaces, lowercase, and compare. This sounds trivially obvious, and yet it's behind a huge number of real-world duplicate-record problems.

The Regression Tests

Finally, add tests so the bug can't return silently:

The full fix yields exactly 20 rows
Running the job a second time adds nothing

That second test, re-running must not change anything, is one of my favourite checks for any batch job. It catches a surprising number of bugs.

What AI Got Right (and What It Couldn't Do)

Let me be fair and specific.

What AI Did Well

Spotted that the numbers didn't add up, and used the gaps as clues
Generated several hypotheses quickly, ranked by fit
Suggested ways to disprove each one
Asked for the missing information instead of guessing
Drafted test ideas that I then refined

What AI Couldn't Do

It couldn't verify anything. It reasoned about the code, but only running it proved the explanation.
It didn't know my business rules. Whether raj.k@acme.com and raj.k@acme.com should be the same customer is a business decision, not a technical one. In this case it clearly should be, but in other systems it may not.
It could only see what I shared. In Round One, it correctly said it couldn't identify the second cause without code. Had I pushed it to guess, it might have produced a confident, wrong answer.

This is the same lesson I learned when Half My Selenium Test Cases Were Written by AI. AI is a brilliant first-draft machine and a poor final authority.

Five Rules for Using AI to Debug Production Issues

Rule 1: Ask for Explanations Before Fixes

A fix request makes the AI jump to the nearest answer. An explanation request makes it consider the whole picture.

Rule 2: Give Real Evidence

Paste actual log lines, counts and timelines. Vague descriptions get vague answers. Specific numbers let the AI do the arithmetic that exposed the second bug.

Rule 3: Demand Ways to Disprove

Add this line to every debugging prompt: "For each cause, tell me what would prove it wrong." It turns a list of guesses into a testable plan.

Rule 4: Let It Ask for More

Include: "What extra information would change your answer?" It prevents confident guessing and tells you what to gather next.

Remove passwords, tokens, personal data and confidential customer details before pasting anything into an AI tool. Follow your company's policy. In many workplaces, sharing code or data with external tools needs approval, so check before you paste.

Mistakes to Avoid

Accepting the first plausible explanation. If the explanation doesn't account for every number, it's incomplete.

Fixing one cause and closing the ticket. Always ask: "Does this fix explain all the symptoms?"

Trusting AI output without reproducing it. Run the test. See the result with your own eyes.

Skipping the regression test. A bug without a test is a bug that will visit again.

Giving AI a summary instead of evidence. "The job creates duplicates sometimes" tells it nothing. The log and counts tell it everything.

Forgetting the human context. Ask the business: what is the correct behaviour? The code can't tell you.

How This Connects to Everyday Debugging

You don't need a nightly sync job to benefit from this approach. The same habit works for:

A test that fails only on the build server
A report with totals that don't match
An API that works for some users and not others
A bug that appears only at month-end

In each case, the useful question is the same: what explanation fits all the evidence? AI is excellent at helping you generate and challenge explanations. You supply the facts and the verification.

If you've read my post about How I Found a Production-Critical Bug Hidden Inside a 40,000-Line Log File, you'll remember that the breakthrough came from careful reading and a clear hypothesis, not from a clever tool. AI makes that process faster. It doesn't replace it.

Reproduce the Experiment Yourself

The whole example is a single small Python file with no dependencies.

python customer_sync.py

python customer_sync.py

It prints the production-style log, the 4 duplicates, and the results of fixing each cause separately. Try this:

Paste the log and ticket into your favorite AI tool using the prompt above.
Compare its hypotheses with the two I found.
Change the data, add a third bug, and see whether the AI spots it.
Run the fixes and the regression tests.

Running a small experiment like this is also an excellent portfolio piece. It shows how you think, not just what you know.

The One-Page Cheat Sheet

The Quick-Recall Version

The rule: explain first, fix second. Ask AI for hypotheses, not code.

Give the AI: real logs and counts, the timeline, recent changes, and relevant code with all secrets removed.

AI is good at: spotting numbers that don't add up, listing possible causes, and drafting tests.

AI can't: know your business rules, see what you didn't share, or replace verification.

Always verify: reproduce each hypothesis, test fixes one at a time, and try to disprove before you trust.

After the fix: add a regression test, run the job twice, and write a short post-mortem.

Memory Trick

E.R.V.F.T. gives you the order: Evidence, ranked hypotheses, Verify, fix all causes, Test so it stays fixed.

Final Thoughts: The Real Value Wasn't the Fix. It Was the Explanation.

If I had jumped straight to fixing the timeout issue, I would have solved only half the problem.

The logs looked clean. The obvious clue was visible. The temptation to patch the first thing that looked broken was strong. But production incidents rarely exist to teach us about the first problem we see. They exist to expose the problems we haven't seen yet.

That's why the biggest lesson from this experiment wasn't that AI found a bug.

It was that AI helped me think like an investigator instead of a firefighter.

The duplicate records weren't caused by one mistake. They were caused by two independent failures hiding behind the same symptom. A fix-first mindset would almost certainly have stopped after finding the first one. An explanation-first mindset kept asking:

Does this explain all the evidence?

~Until the answer was yes.

AI didn't debug the system for me. It didn't verify the results, understand the business rules, or take responsibility for the final decision. What it did do was something surprisingly valuable: it helped generate better hypotheses, challenge assumptions, and turn a messy production incident into a structured investigation.

And that's a much more useful skill than getting a quick fix.

So, the next time a production issue lands on your desk, resist the urge to immediately ask:

"How do I fix this?"

~ Try asking:

"What explanation fits all the evidence?"

You might discover that the bug you're looking at is only half the story.

I'd love to hear your experience. What's the most misleading production bug you've ever investigated, where the first explanation turned out to be completely wrong? Share it in the comments.

If you found this useful, please clap 👏, follow, and share it with someone working in production support, testing, integrations, or software engineering.

Contents