Can AI Hack Applications Without a Security Expert?

AI is changing the way we approach cybersecurity. We are already using AI to write code, review vulnerabilities, analyse logs, and automate security testing.

But there is one important question:

Can AI perform a complete hacking operation without the help of a security expert?

A recent report from the OWASP GenAI Security Project, LLM Exploit Generation — Phase 2, explores this question through practical experiments.

The researchers compared traditional, human-guided LLM usage with autonomous AI agents using penetration-testing tools. Their findings offer some interesting lessons for anyone working in Application Security, Penetration Testing, or AI Security.

Let's break it down.

1. Traditional LLMs vs. AI Agents

First, let's understand the difference.

With a traditional LLM, we ask questions and receive instructions. We run the commands, review the results, and ask the model what to do next.

With an AI agent, the system can use tools, execute tasks, analyse the output, and decide on subsequent actions with less human involvement.

For example, a traditional LLM might suggest how to enumerate services. An agentic system can run the relevant tools, collect the results, and continue its investigation.

This sounds like a big improvement. But does automation automatically mean better results?

The report found that the answer is more complicated.

2. AI Can Make Reconnaissance Easier

One of the most interesting findings was how AI helped with reconnaissance and vulnerability research.

In the experiments, AI systems could discover exposed services, identify an additional staging environment, recognise software versions, and research potentially relevant vulnerabilities.

These tasks usually require knowledge of different tools and techniques.

More importantly, one experiment showed that a user with very little offensive security knowledge could follow AI-generated instructions and reach an account-takeover outcome in a controlled lab.

Think about what this means.

A person who doesn't know where to start can now get guidance from an AI model. The model can help identify what to investigate and explain the next steps.

This lowers the entry barrier for certain offensive security activities.

However, it doesn't mean that anyone can automatically compromise any application. The experiment was conducted in a controlled environment, and the results should be understood in that context.

3. AI Agents Can Get Stuck in Loops

Here is where things get interesting.

We often assume that an autonomous agent will continuously learn from its actions and improve its decisions. But the report found that agents sometimes repeated the same actions without making meaningful progress.

For example, an agent might discover that a service is available but fail to understand why another test is unsuccessful. Instead of changing its approach, it repeats scans or tries similar commands. The researchers described this behaviour as going down a rabbit hole.

The problem is not always a lack of knowledge. Sometimes the model knows the relevant concept but fails to apply it at the right moment.

This is an important limitation.

Executing more commands does not necessarily mean making more progress.

For security teams, this also raises questions about tool efficiency, API costs, unnecessary traffic, and how reliably an agent can recognise a dead end.

4. AI Can Give Incorrect Technical Guidance

Another important finding was that models sometimes generated incorrect commands, fabricated vulnerability references, or gave misleading troubleshooting advice.

A command may look correct but fail in the actual environment. A suggested CVE may not apply to the target. An agent may also misunderstand whether a failure comes from the target, the tool, or the environment.

Anyone who has worked in penetration testing or application security will understand why this matters.

Security testing is not just about running tools. We need to interpret the output, verify the findings, understand the application's behaviour, and distinguish a genuine vulnerability from a false positive.

If a person blindly trusts AI output, the result can be misleading.

This is why validation remains a critical part of security testing, even when AI is involved.

5. Does Agentic AI Eliminate the Need for Security Experts?

The report's answer is more nuanced than a simple yes or no.

The agentic systems progressed further than many standalone, human-guided sessions, especially when the agents were well designed and coordinated.

However, setting up these systems required technical knowledge. During execution, researchers still had to resolve logical loops, fix environment issues, correct assumptions, and intervene when the agents could not proceed.

In the tested configurations, no agent completed the final exploitation step autonomously.

The researchers also observed that the framework surrounding a model could influence the outcome substantially. Tool integration, context management, and coordination affected how effectively the model used its capabilities.

In simple terms, having a powerful model is not enough. The system around it also matters.

For now, I see AI as a way to reduce repetitive work and help security professionals investigate faster, rather than as a complete replacement for security expertise.

6. What Should AppSec Teams Do Now?

From an Application Security perspective, I believe there are a few important takeaways.

First, start evaluating AI-assisted security testing. Understand where AI can help with reconnaissance, vulnerability analysis, code review, and repetitive investigation.

Second, validate every important finding. AI-generated results should not automatically be treated as confirmed vulnerabilities. Evidence and reproducibility still matter.

Third, monitor automated behaviour. Repeated scans, unnecessary requests, and unusual tool usage can create operational noise. Security monitoring should account for the behaviour of automated systems as well as traditional tools.

Fourth, maintain human oversight. High-impact security decisions should have appropriate validation and authorisation. Do not rely entirely on an AI model's own assurance that an action is safe.

Finally, keep building security expertise. Understanding application architecture, authentication, authorisation, business logic, and deployment environments becomes even more valuable when working with AI-based tools.

Final Thoughts

My takeaway is simple:

AI can make security testing faster, but speed without validation is not security.

The goal should not be to replace security professionals with AI. It should be to build systems where AI handles suitable tasks and security professionals provide the judgement, verification, and control that the technology still needs.

As AI agents continue to improve, understanding both their capabilities and their limitations will become an important part of modern application security.

Source: OWASP GenAI Security Project, LLM Exploit Generation — Phase 2: Agentic vs. Classic Offensive Approaches (October 2026).