Post cover image

July 21, 2026

Python Web Penetration Testing — Day 3

Web Crawling with Scrapy — Mapping the Application Like a Digital Cartographer

By Aman Sharma

8 min read

Free Link

I spent weeks manually clicking through web applications, bookmarking every page, and drawing mental maps of how everything connected. Then I discovered Scrapy — and realized I'd been doing the work of a machine. Here's how to automate application mapping and never miss a hidden endpoint again.

The Hook: The Day I Realized Manual Mapping Was Holding Me Back

I was three hours into a penetration test, manually clicking through a web application, documenting every page, every parameter, every form. My browser had fifty tabs open. My notes were a mess. I was exhausted, and I'd barely scratched the surface.

Then I looked at what I'd actually accomplished: I'd mapped maybe 30 pages. The application had hundreds.

My senior colleague walked by, glanced at my screen, and said, "You're doing the computer's job. Let the computer do it."

He introduced me to Scrapy. Within minutes, I had a script that crawled the entire application, discovered pages I'd never found, and extracted forms, emails, and comments — all while I sat back and watched.

That day, I learned a crucial lesson: Reconnaissance is about efficiency. The best hunters automate the boring stuff so they can focus on the interesting stuff.

None

Why This Matters

Web application mapping is the second phase of the penetration testing methodology — and it's where most beginners fall short.

Here's why it's critical:

  • You can't test what you can't find. Hidden pages, forgotten endpoints, and unlinked directories are often the most vulnerable.
  • Manual mapping is incomplete. You'll miss pages that aren't linked from the homepage, or that require specific parameters.
  • Automation scales. One application might have hundreds of pages. A company might have dozens of applications. Manual mapping doesn't scale.
  • Crawlers find what humans miss. JavaScript-generated links, AJAX-loaded content, and deeply nested pages are all fair game for a good crawler.

The difference between a good pentester and a great one is knowing the application's entire attack surface. And the best way to do that is with a crawler.

What Is Web Application Mapping?

In the penetration testing methodology, mapping is the second phase. It's where we build a comprehensive map of the application's resources and functionalities.

What we're looking for:

None

The goal: Identify every component and entry point in the application so we can test them all.

Why Crawlers? (And Their Limitations)

Crawlers automate the mapping process. They start at a URL, extract all links, follow them, extract more links, and repeat until the entire site is mapped.

But they're not perfect:

  • JavaScript-heavy sites — Traditional crawlers don't execute JavaScript. They'll miss dynamically loaded content.
  • Session-dependent content — If you need to be logged in, the crawler won't see authenticated pages.
  • Forms — Crawlers can submit forms, but they need to know what data to send.

The workaround: For JavaScript-heavy sites, use a headless browser like PhantomJS or Selenium alongside your crawler. For authenticated content, log in first and keep the session.

What Is Scrapy?

Scrapy is the industry-standard web crawling framework for Python. It's designed for large-scale web scraping and crawling, making it perfect for security testing.

Key features:

  • Fast and efficient — Handles thousands of requests per minute
  • Extensible — Easy to add custom parsing logic
  • Built-in link extraction — Automatically finds and follows links
  • Item pipelines — Process and store extracted data
  • Middleware — Handle cookies, user-agents, and proxies

The Scrapy workflow:

  1. Start URLs — The spider begins here
  2. Parse response — Extract links and data
  3. Follow links — Request new pages
  4. Repeat — Until the entire site is mapped
None

Setting Up Your First Scrapy Project

First, let's create a Scrapy project. This is the foundation for all our crawling work.

# Install Scrapy
pip install scrapy

# Navigate to your examples directory
cd ~/Desktop/Examples/

# Create a new Scrapy project
scrapy startproject basic_crawler

What this creates:

basic_crawler/
├── scrapy.cfg
└── basic_crawler/
    ├── __init__.py
    ├── items.py          # Data models
    ├── middlewares.py    # Custom middleware
    ├── pipelines.py      # Data processing
    ├── settings.py       # Project settings
    └── spiders/          # Your spiders go here
        └── __init__.py

Creating Your First Spider

A spider is a class that defines how to crawl a website and extract data.

# File: basic_crawler/basic_crawler/spiders/spiderman.py

import scrapy
from basic_crawler.items import BasicCrawlerItem

class MySpider(scrapy.Spider):
    name = "basic_crawler"
    allowed_domains = ["www.scruffybank.com"]
    start_urls = ["http://www.scruffybank.com/"]

    def parse(self, response):
        # Create an item to store extracted data
        item = BasicCrawlerItem()

        # Extract page title
        title = response.css('title::text').get()
        if title:
            item['title'] = title.strip()
            yield item

        # Extract all links
        links = response.css('a::attr(href)').getall()
        for link in links:
            # Make sure the link is absolute
            absolute_link = response.urljoin(link)
            # Only follow links within the same domain
            if 'scruffybank.com' in absolute_link:
                yield scrapy.Request(absolute_link, callback=self.parse)

What this does:

  1. Starts at http://www.scruffybank.com/
  2. Extracts the page title
  3. Finds all links on the page
  4. Follows links within the same domain
  5. Repeats the process recursively

Defining Items (Data Models)

In Scrapy, items define the data structure we want to extract.

# File: basic_crawler/basic_crawler/items.py
import scrapy
class BasicCrawlerItem(scrapy.Item):
    # Page information
    url = scrapy.Field()
    title = scrapy.Field()
    content_length = scrapy.Field()
    # Interesting data
    emails = scrapy.Field()
    forms = scrapy.Field()
    comments = scrapy.Field()
    hidden_fields = scrapy.Field()
    # Links
    links_found = scrapy.Field()
    external_links = scrapy.Field()

Why this matters: Defining items upfront makes your code cleaner and your data more structured.

Running Your First Crawl

Now let's run the spider and see what it finds.

cd basic_crawler
scrapy crawl basic_crawler

What you'll see:

2024-01-15 12:00:00 [scrapy.core.engine] INFO: Spider opened
2024-01-15 12:00:00 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://www.scruffybank.com/> (referer: None)
2024-01-15 12:00:01 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://www.scruffybank.com/login.php> (referer: http://www.scruffybank.com/)
2024-01-15 12:00:01 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://www.scruffybank.com/dashboard.php> (referer: http://www.scruffybank.com/)
2024-01-15 12:00:02 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://www.scruffybank.com/users.php> (referer: http://www.scruffybank.com/)
2024-01-15 12:00:02 [scrapy.core.engine] INFO: Closing spider (finished)

What I found: The crawler discovered pages I hadn't even noticed during manual mapping — including a users.php page that turned out to be vulnerable to SQL injection.

Saving Results to JSON

Scrapy makes it easy to save results in different formats.

scrapy crawl basic_crawler -o results.json -t json

What results.json looks like:

[
  {
    "url": "http://www.scruffybank.com/",
    "title": "Scruffy Bank - Welcome"
  },
  {
    "url": "http://www.scruffybank.com/login.php",
    "title": "Scruffy Bank - Login"
  },
  {
    "url": "http://www.scruffybank.com/dashboard.php",
    "title": "Scruffy Bank - Dashboard"
  }
]
None

Building a Recursive Security Crawler

Now let's build something practical — a crawler that discovers the entire application structure and extracts security-relevant information.

Step 1: Create the Recursive Spider

# File: basic_crawler/basic_crawler/spiders/security_crawler.py
import scrapy
import re
from urllib.parse import urlparse
from basic_crawler.items import BasicCrawlerItem
class SecurityCrawler(scrapy.Spider):
    name = "security_crawler"
    allowed_domains = ["www.scruffybank.com"]
    start_urls = ["http://www.scruffybank.com/"]
    # Track visited URLs to avoid duplicates
    visited_urls = set()
    def parse(self, response):
        # Check if we've already visited this URL
        if response.url in self.visited_urls:
            return
        self.visited_urls.add(response.url)
        # Create item for this page
        item = BasicCrawlerItem()
        item['url'] = response.url
        item['title'] = response.css('title::text').get()
        # Extract interesting information
        item['emails'] = self.extract_emails(response.text)
        item['forms'] = self.extract_forms(response)
        item['comments'] = self.extract_comments(response.text)
        item['hidden_fields'] = self.extract_hidden_fields(response)
        yield item
        # Extract and follow all links
        links = response.css('a::attr(href)').getall()
        for link in links:
            absolute_link = response.urljoin(link)
            if self.is_allowed_url(absolute_link):
                yield scrapy.Request(absolute_link, callback=self.parse)
    def extract_emails(self, text):
        """Extract email addresses from page content."""
        email_pattern = r'[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}'
        return re.findall(email_pattern, text)
    def extract_forms(self, response):
        """Extract form information (action, method, inputs)."""
        forms = []
        for form in response.css('form'):
            form_data = {
                'action': form.attrib.get('action', ''),
                'method': form.attrib.get('method', 'GET'),
                'inputs': []
            }
            for input_elem in form.css('input'):
                input_data = {
                    'name': input_elem.attrib.get('name', ''),
                    'type': input_elem.attrib.get('type', 'text'),
                    'value': input_elem.attrib.get('value', '')
                }
                form_data['inputs'].append(input_data)
            forms.append(form_data)
        return forms
    def extract_comments(self, text):
        """Extract HTML comments."""
        comment_pattern = r'<!--(.*?)-->'
        return re.findall(comment_pattern, text, re.DOTALL)
    def extract_hidden_fields(self, response):
        """Extract hidden form fields (potential IDOR targets)."""
        hidden_fields = []
        for input_elem in response.css('input[type="hidden"]'):
            hidden_fields.append({
                'name': input_elem.attrib.get('name', ''),
                'value': input_elem.attrib.get('value', '')
            })
        return hidden_fields
    def is_allowed_url(self, url):
        """Check if URL is within the allowed domain."""
        parsed = urlparse(url)
        domain = parsed.netloc
        return domain in self.allowed_domains or domain == ''

What this does:

  • Extracts emails, forms, comments, and hidden fields
  • Follows links recursively
  • Avoids duplicates with a visited set
  • Only crawls within the allowed domain

Step 2: Run the Security Crawler

scrapy crawl security_crawler -o security_results.json -t json

What you'll see in the output:

2024-01-15 12:05:00 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://www.scruffybank.com/>
2024-01-15 12:05:01 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://www.scruffybank.com/login.php>
2024-01-15 12:05:02 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://www.scruffybank.com/dashboard.php>
2024-01-15 12:05:03 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://www.scruffybank.com/users.php?id=1>
2024-01-15 12:05:04 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://www.scruffybank.com/admin/>

Step 3: Analyze the Results

What I found in security_results.json:

[
  {
    "url": "http://www.scruffybank.com/",
    "title": "Scruffy Bank - Welcome",
    "emails": [],
    "forms": [],
    "comments": [" Scruffy Bank - A vulnerable banking application for training "]
  },
  {
    "url": "http://www.scruffybank.com/login.php",
    "title": "Scruffy Bank - Login",
    "emails": [],
    "forms": [
      {
        "action": "check_login.php",
        "method": "POST",
        "inputs": [
          {"name": "username", "type": "text", "value": ""},
          {"name": "password", "type": "password", "value": ""}
        ]
      }
    ],
    "comments": [],
    "hidden_fields": []
  },
  {
    "url": "http://www.scruffybank.com/users.php?id=1",
    "title": "User Profile",
    "emails": ["johnny@scruffybank.com"],
    "forms": [],
    "comments": [],
    "hidden_fields": []
  }
]

What this tells me:

  • Login page — Has a POST form with username and password fields (potential for brute force)
  • Users page — Has an id parameter (potential for IDOR)
  • Emails — johnny@scruffybank.com is a valid email (potential for phishing)
  • Comments — A developer comment in the homepage (potential information disclosure)
None

Real Bug Bounty Perspective

Here's a true story from a bug bounty engagement.

I was testing a large SaaS application. The public-facing pages were heavily locked down — no obvious vulnerabilities. But I ran my Scrapy crawler against the application and discovered a /dev/ directory that wasn't linked anywhere.

Inside that directory, I found a backup.sql file containing database credentials, API keys, and customer data. The developer had left it there for "testing" and forgotten about it.

The vulnerability: Sensitive information disclosure via unlinked directory.

The bounty: $2,500.

The lesson: The most critical vulnerabilities are often hidden in unlinked directories. Crawlers find what humans miss.

The Hacker's Workflow

Here's how I use Scrapy in real engagements:

  1. Initial crawl — Map the entire application
  2. Extract forms — Identify every form and its parameters
  3. Extract emails — Build a list of valid email addresses
  4. Extract comments — Look for developer notes, TODOs, and hidden endpoints
  5. Extract hidden fields — Identify potential IDOR targets
  6. Analyze URL patterns — Look for parameters like ?id=, ?user=, ?file=
  7. Test interesting findings — Manually test each discovery

Pro tip: I often run multiple crawls with different configurations — one for the public site, one for authenticated pages, and one focused on specific directories.

Final Thoughts

The day I learned to automate web application mapping was the day I stopped being a beginner. Suddenly, I wasn't spending hours clicking through applications — I was spending minutes writing spiders and hours finding vulnerabilities.

Scrapy changed everything for me. It turned a tedious, error-prone manual process into a systematic, automated workflow. And it revealed hidden pages and endpoints I would have never found on my own.

The best part? The same skills you learned today apply to every web application you'll ever test. Once you know how to build a crawler, you can map any application in minutes.

you can check this article too…

Python Web Penetration Testing Day 2: The Heart of HTTP — Interacting with Web Applications Like a Pro

https://medium.com/the-first-digit/web-penetration-testing-with-python-144863bbcaab?sharedUserId=amannsharmaa

Liked this guide? Smash that clap button 50 times (it's free therapy), drop a comment .

Your engagement keeps the chaos alive.

— Your friendly neighborhood account takeover artist 🕵️‍♂️💥