How I Built an AI Pentester Skill Instead of Another Prompt File

When I first started trying to make AI useful for offensive web testing, I kept running into the same failure mode: the model could sound smart for a few turns, but the quality fell apart once the target got large, the workflow got messy, or the evidence needed to stay clean across multiple steps.

That made one thing clear very quickly. I did not need another big prompt. I needed a real skill.

A prompt can tell a model to "hunt for bugs." A proper skill has to do more than that. It has to define how the model should think about scope, what evidence matters, when to retrieve deeper knowledge, how to avoid wasting context, and how to keep notes so the session does not turn into a pile of half-tested ideas.

That is the problem this skill was built to solve.

The failure I kept seeing with prompt-only setups was not that the model knew nothing. The model usually knew enough to sound useful. The problem was consistency. After a few turns, it would forget what had already been tested, retrieve too much irrelevant material, mix candidates with confirmed findings, or jump from one interesting idea to another without closing the loop. That is not a knowledge problem only. It is an operating problem.

The first design decision was about sources. I did not want the skill to depend on vague blog knowledge or random payload collections with no structure. I wanted a corpus that could actually support professional reasoning. So I built it from open and defensible material:

• RFCs for the protocol layer: URI handling, cookies, WebSocket, OAuth 2.0, bearer tokens, JWT, JSON, TLS, and HTTP semantics.

• OWASP material for security methodology and control language: ASVS, WSTG, Top 10, API Security Top 10, Cheat Sheet Series, Developer Guide, and Secure Coding Practices.

• NIST and CISA material for assessment discipline and authentication assurance: SP 800–115, SP 800–95, SP 800–63B, and Secure by Design guidance.

• Local disclosed report corpora for pattern analogs, proof styles, and attack-shape ideas.

• The project's own playbooks, curated notes, and report-pattern cards for faster routing during real use.

That source choice mattered because I was not trying to teach the model trivia. I was trying to give it a better operating environment.

The second decision was about structure. If I dumped everything into one long file, the skill would become heavy and hard to use. So the skill was split into layers.

The top layer is the operator contract. That is the part that tells Codex how to behave: build an application model first, treat proof as a hard boundary, keep a durable assessment record, and only retrieve more knowledge when it will change the next action.

The second layer is routing. Once the current surface is understood, the skill needs to know whether it should pull web fundamentals, OWASP control guidance, pentest methodology, standards material, local playbooks, or disclosed analog reports. That is where most weak AI skills fail. They either retrieve too much, or they retrieve the wrong thing at the wrong time.

The third layer is the local body around the skill. This is where the AI Pentester project helps. The model is not operating in a vacuum. It can use the existing project body for recon, workspace state, browser-backed testing, Burp-backed request truth, and RAG lookups. The skill is there to make those components usable by the model in a disciplined way, not to replace them.

A real piece of the routing contract looks like this:

Start with:

1. references/routing.md

2. references/base-hunt-contract.md

Then load only the reference that matches the current signal:

- engagement-loop.md for run hygiene

- fundamental-observation-contract.md for a new page/workflow/object/role

- pentest-decision-map.md for evidence-to-bug-family routing

- retrieval-contract.md when local playbooks or report analogs improve the next test

Start with:

1. references/routing.md

2. references/base-hunt-contract.md

Then load only the reference that matches the current signal:

- engagement-loop.md for run hygiene

- fundamental-observation-contract.md for a new page/workflow/object/role

- pentest-decision-map.md for evidence-to-bug-family routing

- retrieval-contract.md when local playbooks or report analogs improve the next test

That rule matters because context is not free. A large model can hold more material, but more material is not automatically better reasoning. The skill should help the model pull the smallest useful piece of knowledge that changes the next action.

The extraction step was important too. I used book-to-skill to convert the open source and standards material into structured extraction artifacts, then grouped those artifacts into four practical libraries:

• web fundamentals

• OWASP web security

• web pentest methodology

• cybersecurity standards

Those are useful categories because they map to real decisions during a hunt. If the question is "what does the browser or protocol actually do here," the model should go to fundamentals. If the question is "what control failed and what does valid proof look like," it should go to OWASP or NIST-aligned material. If the question is "how should this exact surface be tested," it should go to methodology and playbooks.

That sounds obvious once it exists, but getting the routing right is the hard part. The skill only became useful once I stopped treating retrieval as a default action and started treating it as a targeted tool. The model should not retrieve just because knowledge exists. It should retrieve because the next move would be better with a sharper method, a stronger proof shape, or a more exact bug-family route.

That is also why I kept the skill intentionally strict about evidence. The model is still free to be creative. It should invent target-shaped hypotheses, custom payload variants, sibling-route checks, and bug chains. But it should not be free to blur suspicion and confirmation. Creativity is valuable in attack generation. It is dangerous in proof language.

So the skill enforces a few things very hard:

• map the application before attacking it

• reduce every meaningful surface to actor, object, state, parser or sink, and authority boundary

• write down what was tested

• separate candidate signals from confirmed findings

• only promote a finding when the proof is reproducible and attacker-relevant

That combination turned out to be much more valuable than simply adding more methodology text.

The other practical piece is the assessment record. The model should not keep the whole hunt only in chat memory. It should maintain durable files like:

Assessment.md

AttackSurface.md

Graph.md

Endpoints.md

JSAnalysis.md

Hypotheses.md

Coverage.md

Findings.md

Retrospective.md

Assessment.md

AttackSurface.md

Graph.md

Endpoints.md

JSAnalysis.md

Hypotheses.md

Coverage.md

Findings.md

Retrospective.md

This is boring on purpose. During a long hunt, the valuable thing is not just the next payload. It is knowing what has already been mapped, what was tested, what is blocked, what remains interesting, and what has proof.

The other lesson I learned while building this is that a good pentesting skill should not try to replace the model's own judgment. If a skill becomes too rigid, it turns the model into a checklist runner. If it is too loose, the model drifts and wastes time. The right balance is to give the model a strong frame and then leave room for target-specific reasoning inside it.

That is the part I care about most in this design. I do not want Codex to become a static exploit macro. I want it to behave more like a disciplined operator: understand the target, use the right layer of knowledge, form a concrete hypothesis, test it, validate it, record it, and move on.

The practical takeaway from building this is simple. If you want an AI model to do serious web security work, do not start by writing a bigger prompt. Start by deciding what knowledge it should trust, what artifacts it should maintain, and what exact boundary separates "interesting" from "proved." Once those are clear, the skill becomes a real tool instead of a clever instruction block.

The skill is still meant to leave room for creativity. That part matters. I do not want the model to become a checklist runner. The goal is to give it strong guardrails around scope, evidence, and retrieval, then let it form target-shaped hypotheses, test sibling routes, chain findings, and use its own reasoning where the target demands it.