Post cover image

August 23, 2026

Building AI Pentester, Week 2: Scope Parsing Is More Important Than It Looks

This week I worked on the Program layer of AI Pentester, and it turned out to be much more important than I first thought.

By Krishn Babariya

2 min read

When people talk about security tooling, they usually jump to recon tricks, browser automation, payloads, or validation logic. But before any of that happens, the system needs to know what the target actually is. In real programs, that input is messy. You rarely get one clean hostname and a neat set of rules. You get wildcard domains, root domains, full URLs, copied policy text, exclusions, notes, rewards, and tech hints all mixed into one blob of text.

A human can scan that and mentally separate the important parts. A system should not be expected to do that again and again in later stages.

That is why the Program layer now exists as a real normalization boundary.

This is the Program tab where that boundary became visible:

None

The flow for this stage looks like this:

None

The parser currently extracts domains, wildcards, URLs, in-scope lines, out-of-scope lines, rules, reward notes, and technology hints. It also tries to guess a primary domain, wildcard, and suggested scope URL so the next stage does not begin with a blank state.

The first part of that code path looks like this:


domains, wildcards, urls = _extract_domains_and_urls(raw_text)

sections = _extract_sections(raw_text)

result.domains = domains

result.wildcards = wildcards

result.urls = urls

result.in_scope_lines = sections["in_scope"]

result.out_of_scope_lines = sections["out_of_scope"]

result.rules_lines = sections["rules"]

result.reward_notes = sections["rewards"]

result.tech_hints = sections["tech_hints"]

The concrete bug this stage is meant to prevent is inconsistent scope handling. A wildcard like *.example.com, a root domain like example.com, and a full URL like https://app.example.com/login are not the same input. If the parser flattens them too early, recon loses breadth or hunt loses the correct starting path. If exclusions are mixed with in-scope text, later automation can make decisions it should never make.

The practical reason this matters is simple. If the tool keeps passing around raw text, every later layer has to keep reinterpreting that text. Recon has to guess whether a line represents a wildcard or a URL. Hunt has to guess whether a rule was hard scope or just a comment. That is a bad use of complexity. The clean version is to interpret once and store the result in a stable workspace.

This is also where a lot of subtle future bugs come from. If the parser handles a wildcard inconsistently, recon can miss valid hosts. If out-of-scope data is not preserved clearly, the hunt layer can make decisions it should never even be allowed to make. If a full URL is flattened into a bare host too early, later crawling and evidence correlation become weaker.

The main lesson from week 2 is that scope parsing is not setup work in the boring sense. It is attack-surface preparation. If the front door of the system is careless, every downstream component is working with compromised context.

For anyone building something similar, the method that helped most here was to treat the program input like a data-ingestion problem, not just a form field. Normalize once. Preserve the original meaning. Write the workspace so the next stage can move immediately.

The practical pattern I would reuse is:


raw policy text -> structured scope object -> saved workspace package -> recon input

That gives the AI less ambiguity later. The model can still reason over the program rules, but it does not have to rediscover the basic target shape every time.

That is what I wanted this week to accomplish, and it changed how solid the rest of the pipeline feels already.