Can AI Distinguish a Product Bug from a Broken Test?

Something went wrong.

But it does not tell us what.

The failure could come from:

• a real product defect,
• a broken locator,
• unstable test data,
• an unavailable environment,
• a flaky test,
• an intentional application change,
• or simply an outdated expected result.

Yet most automation frameworks treat all of these situations in exactly the same way:

FAILED.

And then a human starts investigating.

With Agentic AI, we can potentially go much further.

A failed test is not a diagnosis

Imagine a Playwright test expects a payment to be accepted.

The execution fails.

A basic automation system reports the error.

A basic self-healing system may try to modify the test.

But an intelligent system should first ask:

What exactly failed?

Did the button disappear?

Did the API return a 500?

Did the business rule change?

Was the test data invalid?

Or did the application genuinely behave incorrectly?

Those are completely different problems.

And they should lead to completely different actions.

The system needs evidence before making a decision

Instead of:

Execution → Failure → Fix test

we can think about:

Execution → Evidence → Analysis → Classification → Decision

The AI can analyze different signals:

UI behavior What happened on the screen?

API responses Did a backend service fail?

Logs and traces Where did the execution actually break?

Requirements What was the expected business behavior?

Application changes Was the UI or feature intentionally modified?

Previous executions Has this failure happened before?

Only after combining this information should the system decide what to do.

Different failures need different decisions

For example:

Locator changed

→ likely an automation problem → self-healing may be appropriate.

Environment unavailable

→ infrastructure issue → retry or escalate.

Business behavior no longer matches the requirement

→ potential product defect → do not modify the test.

Requirement was officially changed

→ test may need to be updated → validate the new expected behavior.

Intermittent failure

→ possible flaky test → analyze execution history before changing anything.

The important point is simple:

The same "FAILED" status should not trigger the same response.

This is where multiple agents become useful

One agent can analyze the execution.

Another can inspect logs and API traces.

Another can compare the observed behavior with the requirement.

Another can look at application changes and previous executions.

And a decision agent can combine those signals before recommending an action.

The workflow becomes:

Test fails → collect evidence → identify probable root cause → classify the failure → decide whether to heal, retry, report or escalate

This is much more powerful than simply teaching AI how to repair broken scripts.

Confidence matters

Of course, AI will not always know the answer with certainty.

And it should not pretend to.

A mature system should be able to say:

"I believe this is an automation issue with 92% confidence."

or:

"The available evidence is insufficient. Human validation is required."

That uncertainty is not a weakness.

It is part of making AI decisions explainable and safe.

This is the direction we are exploring at TuringOne

The objective is not just to automate failure handling.

It is to make failure analysis more structured, contextual and explainable.

Because the real question after a failed test should not be:

"How do we make it pass again?"

It should be:

"What actually broke?"

And maybe that is one of the most important steps toward truly intelligent testing systems:

moving from test execution to automated diagnosis.