Evaluating AI-Generated Test Cases: What Happened When I Introduced 10 Deliberate Defects

I compared manual test design with AI-assisted testing on a realistic enterprise scenario, using 10 planted bugs. Each approach missed things the other caught.

The Question Everyone in QA Is Quietly Asking

Open any AI assistant, paste in a requirements document, and type "write test cases." Within seconds you get a tidy list of thirty of them, numbered, named and confident.

It feels like magic. It also raises an uncomfortable question that every tester, developer and QA lead is thinking about:

AI generated 30 test cases in seconds. But how many would actually catch bugs in a real enterprise application?

I wanted a real answer instead of an opinion. So I built a small experiment you can run yourself. The results did surprise me, though maybe not in the direction you'd expect. AI didn't "win," and neither did the human approach. What happened is far more useful to know, especially if you're just starting your career.

In this article you'll learn:

A realistic enterprise testing scenario (purchase order approvals)
How a human tester thinks about it
How an AI assistant generates test cases
Exactly which bugs each approach missed, and why
A practical human-plus-AI workflow you can use tomorrow
A one-page cheat sheet

First, Some Honest Caveats

Before the results, let me be upfront about what this experiment is and isn't. Good testers don't oversell their evidence.

It's small. One program, ten bugs, two test suites. This is a learning experiment, not a scientific benchmark.

The bugs were planted by me. I chose them to represent common real-world defects. A different set of bugs would give different numbers.

The AI suite was written from the specification alone. I gave the AI assistant only the written rules and asked it to generate tests. I wrote this suite before planting the bugs, so it couldn't be tuned to them.

The human-designed suite was written afterwards. It follows how an experienced QA engineer thinks: risk first, then rules, then combinations. Because the bugs already existed when I wrote it, this suite probably has a small advantage. Keep that in mind when reading the scores.

Your results may differ. The whole code is included at the end so you can swap in your own AI tool, your own tests, and your own bugs. That's the real value here.

With that said, let's look at the scenario.

The Testing Scenario: A Purchase Order Router

Almost every enterprise application has some version of an approval workflow. Someone requests something, the system decides who needs to approve it, and rules check the request along the way. Expense claims, leave requests, loan applications, purchase orders. Same idea, different clothes.

I chose a purchase order router. It decides what happens to a new order.

The Rules in Plain English

The system checks each order, in this exact sequence:

Valid input: The amount must be a real number above zero, and the currency must be USD, INR or EUR.
Vendor check: The vendor's status must be exactly "approved."
Budget check: The order's value in USD must not exceed the department's remaining budget.
Auto-approval: Orders up to and including $1,000 are approved automatically.
Approval level: Orders above $1,000 up to and including $10,000 need a manager. Anything above $10,000 needs a director.
Self-approval check: If someone is both the requester and the approver, the order is rejected.

Currencies convert to USD first. In my version, 1 USD equals 80 INR, and 1 EUR equals 1.25 USD.

Why This Scenario Is Realistic

It looks simple, but it has the things that make enterprise software tricky: money, thresholds, multiple currencies, permissions, and rules that run in a specific order. Real bugs love to hide exactly here.

What Is Seeded-Bug Testing? (A Beginner-Friendly Explanation)

Here's a trick testers use to measure how good a test suite really is.

Take a program that works correctly. Deliberately break it in one small way, such as changing a "less than or equal to" into a "less than." Then run your tests. If at least one test fails, your suite caught the bug. If everything still passes, your suite missed it.

Repeat this with many different small bugs, and you get a score for how good your tests are at catching real mistakes. Professionals call this mutation testing, and each broken version is called a mutant.

Think of it like a fire drill. You don't wait for a real fire to check whether the alarm works. You create a small, safe fire.

This approach is brilliant for our question because it measures what matters: not how many tests you have, but how many bugs they catch.

The 10 Bugs Waiting to Escape into Production

To see whether AI-generated tests could spot real defects, I deliberately planted 10 realistic bugs into the system.

None of these are flashy crash-the-app failures. They're the kind of subtle mistakes that slip through code reviews, survive testing, and quietly create expensive problems in production.

🚨 Bug #1: An order of exactly $1,000 incorrectly requires approval. 🚨 Bug #2: An order of exactly $10,000 incorrectly requires director approval. 🚨 Bug #3: INR amounts are multiplied by 80 instead of divided during currency conversion. 🚨 Bug #4: The budget check runs before the vendor-status check, producing the wrong rejection reason. 🚨 Bug #5: An order equal to the available budget is incorrectly rejected. 🚨 Bug #6: Users can approve their own purchase orders at the director level. 🚨 Bug #7: Vendors with a "pending" status are mistakenly treated as approved. 🚨 Bug #8: Unsupported currencies are silently accepted. 🚨 Bug #9: Decimal values are lost during currency conversion, causing incorrect approval decisions. 🚨 Bug #10: NaN (Not a Number) amounts pass validation.

At first glance, most of these look minor. They're not.

A few can trigger incorrect approvals. A few can create audit findings. Others can quietly undermine financial controls. None would make headlines, but every one of them could cost a business money, trust, or both.

And that's exactly why they're interesting: these are the kinds of bugs real testers get paid to find.

How a Human Tester Approaches It

A human tester usually doesn't start by writing test cases. They start by asking questions.

Step 1: Ask "What Would Hurt the Business?"

Before thinking about inputs, a good tester thinks about consequences. What's the worst thing this system could do?

Approve an order that should have needed a director (money leak)
Let someone approve their own purchase (fraud and audit trouble)
Show the user a confusing rejection reason (support tickets)

Risk decides where to spend effort. Not every rule deserves equal attention.

Step 2: Look at Boundaries

Humans learn early that bugs love edges. If the limit is $1,000, test exactly $1,000, a cent above it, and a cent below it. This habit is called boundary value analysis, and it's the single most productive technique in testing.

Step 3: Think About Rule Interactions

This is where experience shines. A tester asks: what if two rules apply at once?

What if the vendor is blocked and the order is over budget? The spec says the vendor check comes first, so the user should see a vendor rejection. A tester will deliberately build a case where both rules fail, to see which one wins.

Step 4: Think About How the System Will Really Be Used

A tester picturing a real user thinks about money in decimals ($1,000.50 is a very different thing from $1,000), about each approval level, and about who exactly is blocked from approving what.

A Few Human-Designed Test Cases

Here are three from my human-style suite, so you can see the flavour:

# Rule interaction: vendor blocked AND over budget. Which message wins?
route(9000, "USD", "blocked", 100, "amy")      # expect REJECTED_VENDOR

# Money detail: just above the auto-approval limit
route(1000.50, "USD", "approved", 9999, "amy") # expect NEEDS_MANAGER

# Permission check at the highest level
route(50000, "USD", "approved", 99999, "amy", "amy")  # expect REJECTED_SELF_APPROVAL

# Rule interaction: vendor blocked AND over budget. Which message wins?
route(9000, "USD", "blocked", 100, "amy")      # expect REJECTED_VENDOR

# Money detail: just above the auto-approval limit
route(1000.50, "USD", "approved", 9999, "amy") # expect NEEDS_MANAGER

# Permission check at the highest level
route(50000, "USD", "approved", 99999, "amy", "amy")  # expect REJECTED_SELF_APPROVAL

Notice what these tests have in common. Each one targets a specific way the business could be harmed, not just a line in a document.

What the AI Did Well

It was systematic. It walked through the rules one by one and wrote tests for each.

It remembered boundaries. It tested exactly $1,000, $1,001, exactly $10,000 and $10,001, which is textbook boundary analysis.

It covered invalid inputs a human might not bother with. It tried a missing amount, a text amount instead of a number, a NaN value, an empty currency, an unsupported currency and an empty vendor status.

It was consistent. No fatigue, no skipped rules, no typos in the test names.

A Few AI-Generated Test Cases

("tc02 exactly 1000 auto-approves",
    route(1000, "USD", "approved", 50000, "amy")   # expect AUTO_APPROVED
("tc13 NaN amount invalid",
    route(float("nan"), "USD", "approved", 50000, "amy")  # expect REJECTED_INVALID
("tc14 unsupported currency GBP invalid",
    route(100, "GBP", "approved", 50000, "amy")    # expect REJECTED_INVALID

("tc02 exactly 1000 auto-approves",
    route(1000, "USD", "approved", 50000, "amy")   # expect AUTO_APPROVED
("tc13 NaN amount invalid",
    route(float("nan"), "USD", "approved", 50000, "amy")  # expect REJECTED_INVALID
("tc14 unsupported currency GBP invalid",
    route(100, "GBP", "approved", 50000, "amy")    # expect REJECTED_INVALID

These are good tests. Any tester would be happy to receive them as a starting point.

The Hidden Pattern

Here's something I noticed when I read all 30. They were mostly one rule at a time. Almost every test checked a single rule in isolation. Very few tried to make two rules collide, and none of them asked "what's the worst thing that could happen here?"

Thirty test cases sound like thirty different ideas. In reality it was closer to a dozen ideas, repeated with different numbers. Keep that in mind, because it explains the results.

The Results: What Actually Happened

I ran both suites against the correct program first. Both passed everything, so there were no false alarms. Then I ran them against each of the 10 broken versions.

The AI-generated suite (30 tests) caught 6 of 10 bugs.
The human-designed suite (18 tests) caught 8 of 10 bugs.
Together, the two suites caught all 10.

Read that again, because there are three separate lessons hidden in those numbers.

Lesson 1: More Tests Is Not Better Tests

The AI suite had about 70% more tests than the human suite, yet it caught fewer bugs. Test count is a vanity metric. What matters is how many different kinds of failure you're probing.

Lesson 2: Neither Approach Was Enough Alone

If I'd stopped at either suite, I'd have shipped bugs. This is the strongest argument against both "AI will replace testers" and "AI tests can't be trusted."

Lesson 3: The Overlap Is Smaller Than You'd Think

Each approach caught bugs the other one completely missed. They aren't two versions of the same thing. They have different blind spots. That's exactly why combining them works.

The Defect Line-Up: Who Caught What?

Four bugs were caught by both. Four were caught only by the human suite. Two were caught only by the AI suite.

Bugs Both Approaches Caught

The two boundary bugs ($1,000 and $10,000), the rupee conversion bug, and the pending-vendor bug. These are the "textbook" bugs: clear rules, clear values. Both approaches handle them well, which tells you something comforting. The fundamentals are covered either way.

What the AI Suite Missed (and Why)

Missed: Rule Order (Vendor vs Budget)

The bug: the system checked the budget before the vendor status. For a blocked vendor with an over-budget order, the user would see "over budget" instead of "vendor blocked."

Why the AI missed it: every AI test checked one rule at a time. A blocked vendor with a healthy budget. An approved vendor with a tiny budget. The AI never created a situation where both rules failed together, so the order never mattered.

Why it matters in real life: the wrong error message sends users down the wrong path. They request more budget for an order that's going to be rejected anyway.

Missed: Budget Exactly Equal to Order Value

The bug: an order exactly equal to the budget was rejected, when the spec says only orders that exceed the budget are rejected.

Why the AI missed it: it tested budgets just above and just below the order amount, but not the exact equality. It did apply boundary thinking to the approval thresholds but didn't carry it over to the budget rule.

Lesson: boundary analysis has to be applied to every comparison, not just the obvious ones.

Missed: Self-Approval on Director-Level Orders

The bug: the "you can't approve your own order" check was skipped for director-level orders.

Why the AI missed it: it tested self-approval once, at the manager level, and ticked that rule off. A human tester immediately asks, "Does this hold at every approval level?" because permission bugs tend to hide at the high end, where the most money moves.

This is the kind of bug that causes real audit findings.

Missed: Dropped Cents After Conversion

The bug: after converting to USD, cents were truncated. So $1,000.50 became $1,000 and was auto-approved when it should have gone to a manager.

Why the AI missed it: its test values were mostly round numbers. It did test $1,001 and $0.01, but never a value just above a threshold by a small fraction. A human who has seen money bugs knows that rounding is where financial software breaks.

What the Human Suite Missed (and Why)

Humans weren't perfect either.

Missed: Unsupported Currencies

The bug: if someone entered a currency the system didn't support, it was silently accepted instead of rejected.

Why the human suite missed it: the human tester thought like a user of the screen. On a real screen, the currency is a dropdown with three choices. Why test GBP? The AI, reading only the spec, didn't know about the dropdown and tested it anyway.

In real life, though, APIs, imports and integrations don't have dropdowns. Data arrives from other systems, and that's where unexpected values appear.

Missed: NaN Amounts

The bug: a "not a number" amount slipped through.

Why the human suite missed it: nobody types NaN into a form. It only appears when another system produces bad data, such as a broken spreadsheet import or a divide-by-zero in an upstream calculation. Humans test what feels plausible. AI tests what's possible.

Step 1: Human Identifies the Risks

Start with the question: what would hurt the business most? List the top three to five risks. This takes ten minutes and steers everything afterwards.

Step 2: AI Drafts a Wide First Set

Paste the specification into your AI tool and ask for test cases. You'll get breadth fast. Treat this as raw material, not a finished suite.

Step 3: Human Reviews and Prunes

Read every generated test. Delete duplicates. Fix wrong expected results. Mark which risks they cover and which they don't. This review step is where your professional judgement matters most.

Step 4: Human Adds the Gaps

Now add what the AI tends to miss: rule combinations, rule order, exact equality, permissions at every level, and domain-specific details like money rounding.

Step 5: AI Expands and Automates

Go back to the AI with a more focused request: "Generate unusual and invalid inputs for this function," or "Convert these test cases into automated test code." It's excellent at this kind of mechanical scaling.

Step 6: Run, Learn and Feed Back

When a bug escapes to production, don't just fix it. Add a test for it, and ask why your process missed it. Over time, your human-plus-AI process gets sharper.

Prompts That Give Better AI Test Cases

The quality of your results depends heavily on how you ask. Here are a few prompts worth trying.

Ask for combinations explicitly.

"Generate test cases where two or more rules fail at the same time, and state which rule should win."

Ask for boundaries on every comparison.

"List every comparison in this specification (greater than, at most, equal to) and give boundary test values for each."

Ask for the evil user.

"What inputs would an attacker or a careless integration send that a normal user never would?"

Ask for risk ranking.

"Rank these test cases by business impact if they fail."

Ask it to challenge itself.

"What rules or behaviours are NOT covered by these tests?"

Notice that every good prompt pushes the AI beyond its default habit of "one test per rule."

Mistakes Freshers Make with AI-Assisted Testing

Mistake 1: Trusting the Count

"The AI gave me 50 test cases, so we're covered." Not necessarily. Count what kinds of risks the tests cover.

Mistake 2: Skipping the Review

Never run AI-generated tests without reading them. Some may assert the wrong thing, and a wrong test that passes gives false confidence.

Mistake 3: Giving the AI No Context

A prompt with only "write test cases for this" gets generic tests. Give it the business context, past bugs and risk areas.

Mistake 4: Never Checking Your Own Tests

How do you know your tests are any good? Try the seeded-bug trick. Break something on purpose and see whether a test fails. If none does, you've learned something valuable.

Reproduce the Experiment Yourself

The whole experiment is a tiny Python project with no dependencies. It has four files:

order_router.py is the program under test, with the ten seeded bugs and the specification
ai_suite.py holds the 30 AI-generated tests
human_suite.py holds the 18 human-style tests
run_experiment.py runs everything and prints the results

To run it:

python run_experiment.py

python run_experiment.py

Make It Your Own

Give the specification to your favourite AI tool and replace ai_suite.py with its output.
Write your own tests before looking at the bug list, then replace human_suite.py.
Add bugs of your own to order_router.py.
Compare. You may get very different numbers, and that's the point.

This is also a great portfolio piece. Running a small experiment and sharing honest results says more about your testing mindset than any certificate.

The One-Page Cheat Sheet

Short on time? Here's the whole article on one page.

The Quick-Recall Version

AI is great at: fast first drafts, boundary and invalid-input checks, test data and automation scripts.

Humans are great at: business context, risky rule combinations, money and permissions, and knowing what really matters.

AI often misses: rule order, exact-equality edge cases, and checks that must hold at every level.

Humans often miss: odd inputs like NaN or unsupported values, and anything tedious to write by hand.

Best workflow: human sets the risks, AI drafts wide, human reviews and adds the gaps, AI scales and automates, then learn from every escaped bug.

Seeded-bug test: break a working program on purpose to see whether your tests notice.

Numbers to Remember

AI-generated suite: 30 tests, 6 of 10 bugs caught
Human-designed suite: 18 tests, 8 of 10 bugs caught
Combined: 10 of 10 bugs caught

Memory Trick

AI widens the net. Humans aim it.

Final Thoughts: The Real Winner Wasn't AI or Humans

I started this experiment with a simple question: Can AI-generated test cases catch real bugs?

The answer is yes, but that's only half the story.

What surprised me most wasn't what AI caught. It was where AI and human testers failed differently. The AI-generated suite caught issues the human-designed suite missed, while the human-designed suite found bugs the AI never thought to look for. Together, they caught all 10.

That's the real lesson here.

AI is excellent at speed, consistency, and generating broad test coverage. Humans are still better at understanding business risk, spotting subtle interactions between rules, and asking, "What could go wrong in the real world?"

The future of software testing isn't manual testing vs AI.

It's manual testing plus AI. If you're early in your QA or testing career, that's good news. The skills that make great testers valuable, understanding the business, thinking critically, challenging assumptions, and identifying risk, are still the skills that matter most.

Use AI as a fast, tireless assistant. Just don't outsource your judgment to it.

If you found this useful, please clap 👏, follow, and share it with a tester or developer friend.

Contents