I Gave Claude a Legacy Test Suite. It Found Bugs QA Missed for Years.

What happened when I pointed an AI agent at a decade of untouched test cases — the gaps it found, the false positives it created, and the one bug that made me stop and double-check everything QA had signed off on.

A few weeks ago, I wrote about giving an AI agent autonomy on a 20-year-old Siebel system and the five places it broke. That post was about production code. This one is about something quieter, and honestly more uncomfortable: our test suite.

💥 Master Any Skills in 3 Months 📚 Up to 50% OFF Premium Courses ⏰ Limited-Time Offer 👉 Enroll Now & Start Learning

Test suites don't get the same attention production code does. Nobody schedules a refactor sprint for test cases. They accumulate for years, get copy-pasted for new features, and mostly get trusted by default — if it's green, it's fine. Nobody asks why it's green.

So, I ran an experiment. I pointed Claude at a legacy regression suite — a few hundred test cases, some over eight years old, covering core CRM workflows in our Siebel environment — and asked it to review, not rewrite. Just look at what we had and tell me what it thought.

I expected cleanup suggestions. Maybe some dead test cases. What I got instead was a list of coverage gaps that had been sitting in production-critical workflows for years, silently assumed to be tested. QA hadn't missed them because QA was careless. They'd missed them because the suite told everyone they were covered.

This is the honest version of that week — what actually happened, not the highlight reel.

Why Legacy Test Suites Are a Trust Problem, not a Coverage Problem

Here's the thing nobody says out loud: a passing test suite doesn't mean your system is tested. It means your system matches whatever assumptions existed when those tests were written.

Our suite had grown the way most enterprise test suites grow — reactively. A bug ships, someone writes a test to catch that exact bug next time, and the suite gets one line longer. Repeat for eight years and you get excellent coverage of bugs that already happened and close to nothing for the logic paths nobody has broken yet.

That's not a QA failure. That's just how manual, incremental test-writing works under deadline pressure. The problem is the suite looks comprehensive. Green checkmarks don't distinguish between "this path is genuinely safe" and "nobody has hit this path with a bug yet."

Illustration of a reactive testing workflow in which test cases are primarily added after production bugs are discovered. Each identified defect results in a new regression test, causing the test suite to grow incrementally. When new features are introduced, developers often adapt existing tests rather than systematically designing coverage for new execution paths. Consequently, untested paths may remain undiscovered, leading to a "green" build status that can create a false sense of confidence in software quality and test coverage.

That gap between "tested" and "assumed safe" is exactly where Claude turned out to be useful — not because it's a better tester than a human, but because it doesn't inherit the suite's history. It has no memory of which bugs "already happened," so it doesn't default to trusting a path just because nothing's broken there recently.

The Setup: What I Actually Gave It

To keep this honest, here's exactly what I did — no cherry-picking the setup after the fact.

• Input: ~340 existing test cases across four core Siebel modules (case management, contact merge, workflow escalation, and reporting exports), exported as structured text with test steps and expected outcomes
• Task: Review the suite for gaps, contradictions, and untested edge cases — not rewrite or "improve" existing tests
• Guardrails: Read-only access. Claude could flag and suggest, but every recommendation went through manual QA review before anything changed
• What I didn't do: I didn't tell it which modules had known issues, and I didn't prime it with our bug-tracker history. I wanted to see what it found cold.

What It Actually Found

1. The contact-merge edge case nobody had tested in either direction

Our contact-merge tests all assumed Contact A merges into Contact B cleanly. Every single test case checked the forward merge. Not one checked what happens when a merge is triggered on a contact that's mid-update in another session — a timing condition that's rare, but not impossible in a system used by hundreds of agents simultaneously.

Claude flagged it not because it "knew" this was risky, but because it noticed every test case shared the same underlying assumption and none of them varied it. That's a pattern-recognition strength, not a domain-expertise one — and it's exactly the kind of thing that's easy for a human to miss when you've read the same 40 test cases a dozen times.

2. A reporting export bug that had been "passing" for three years

This one stung. A scheduled export test checked that a report generated correctly — but the expected output file it compared against had a stale date field baked in from whenever the test was written. The test wasn't validating correctness anymore. It was validating that the export matched a snapshot from three years ago, and any date-related regression since then would have silently passed as long as the format stayed superficially similar.

Nobody wrote that test badly on purpose. It made sense in 2023. It just never got revisited as an assumption worth re-checking.

3. Escalation logic that only worked if you followed the "expected" path

The workflow escalation tests all assumed cases move through statuses in the documented order. Claude asked, in its output, what happens if an agent skips a status — say, escalates directly from "New" to "Critical" without passing through "In Review," which the UI technically allows even if it's not the intended flow. We didn't have a single test for that. Turns out, when we manually checked, the escalation notification logic did have a gap there. Rare in practice. Not zero.

Where Claude Got It Wrong — Because This Isn't a Success Story Without This Part

Not everything it flagged was a real gap. About a third of its "issues" were false positives:

• It flagged several tests as "redundant" because they looked structurally similar — but the near-duplicate tests were intentionally covering different user roles with different permission sets. Claude didn't have enough context on our role hierarchy to see why they weren't actually redundant.
• It suggested a "missing" null-check test for a field that's enforced as mandatory at the database level, making the scenario it worried about literally impossible to reach.
• On two occasions it recommended a fix to test logic that would have quietly weakened the assertion rather than strengthened it — technically making the test pass more easily, not more correctly. If we'd accepted those without review, we'd have shipped a worse test suite while feeling like we'd improved it.

That last point is the one I'd want any team to sit with before doing this themselves. An AI reviewing your tests will sound confident about all of its suggestions, correct and incorrect alike. The false positives didn't look any less certain than the real findings. Manual QA review wasn't a formality here — it was loading bearing.

What This Actually Changes About How We Test

I'm not going to tell you AI review replaces manual QA, because that's not what happened here. What happened is narrower and, I think, more useful:

Claude is good at finding where your test suite's assumptions cluster. Humans who've maintained a suite for years stop seeing those assumptions — they've read past them so many times they've become invisible. An agent with no history has no reason to skip over them.

Claude is not good at knowing which assumptions are load bearing on purpose. It doesn't know your role hierarchy, your database constraints, or why a test that looks redundant isn't. That context has to come from someone who's lived in the system.

The workflow that actually worked wasn't "AI finds bugs." It was AI surfaces candidates, human decides which candidates are real. Cut either half out and you either miss the gaps that matter or drown your suite in noise that doesn't.

If You're Going to Try This Yourself

A few things I'd tell anyone trying this on their own legacy suite, based purely on what actually happened here — not what I hoped would happen:

1. Don't prime it with your bug history first. Let it review cold. Priming it with "here's what we already know is broken" biases it toward finding what you already expect, which defeats the point.
2. Budget real review time. A third false-positive rate isn't a failure; it's the cost of using a tool that has no domain memory. Plan for it instead of being surprised by it.
3. Start with your oldest, least-touched test modules. That's where assumption drift compounds the most — nobody's questioned those tests in years, which is exactly where Claude's lack of history becomes an advantage instead of a limitation.
4. Treat every "this is redundant" suggestion with extra suspicion. That was our highest false-positive category by far, almost always because of missing role or permission context.

We didn't rewrite our test suite. We added eleven new test cases covering real gaps, fixed one badly-stale export test, and — maybe more importantly — stopped assuming a green build means a safe system. That last shift is the one that's actually stuck.

If you're maintaining a test suite old enough to have its own legacy problems, I'd be curious what an honest review turns up for you too.

This is part of an ongoing series on what agentic AI actually does — not what it demos — inside old enterprise systems. If you're running AI against your own legacy stack, I'd genuinely like to hear what broke.

Thank you for being a part of the community

Before you go:

👉 Be sure to clap and follow the writer ️👏️️

👉 Follow us: Linkedin| Medium

👉 CodeToDeploy Tech Community is live on Discord — Join now!

Disclosure: This post includes affiliate and partnership links.