Post cover image

May 28, 2026

CONTENT DISCOVERY-TRYHACKME WALKTHROUGH

Discover hidden web content using manual techniques, OSINT, and Gobuster enumeration.

By Girija

7 min read

TASK 1-INTRODUCTION:

In the world of web application security, not everything is meant to be visible to the public. Many web servers host hidden files, forgotten directories, backup archives, admin panels, configuration files, and development resources that are not directly linked from the main website. These hidden resources can expose sensitive information and become easy entry points for attackers.

Content discovery is the process of identifying these hidden or unlinked resources within a web application. It plays a critical role in penetration testing because even a single exposed backup file or administration portal can lead to severe security breaches.

During security assessments, testers often search for:

  • Hidden directories and files
  • Admin login panels
  • Backup and old version files
  • Configuration files
  • Development environments
  • Exposed APIs and dashboards

SET UP YOUR VIRTUAL ENVIRONMENT:

Before starting the content discovery lab, the virtual environment must be configured properly. This includes starting both the AttackBox and the Target Machine. The AttackBox provides the necessary security tools for testing, while the lab machine acts as the target web application for performing content discovery tasks.

None

TASK 2 — MANUAL DISCOVERY-COMMON FILES:

robots.txt and sitemap.xml

During content discovery, certain default files exposed by web servers can reveal hidden or sensitive information. Two of the most important files are robots.txt and sitemap.xml.

robots.txt

The robots.txt file is used to tell search engine crawlers which parts of a website should not be indexed. However, it is not a security mechanism because anyone can still access the listed directories directly through the browser.

None

sitemap.xml

The sitemap.xml file helps search engines discover important pages on the website. It often contains URLs that may not be easily accessible through normal browsing.

In this lab, the sitemap revealed multiple endpoints, including hidden and sensitive paths such as:

/s3cr3t-area

The sitemap also exposed URLs with parameters like id=, which may become useful testing points during web application security assessments.

Both robots.txt and sitemap.xml are valuable sources of information during the reconnaissance and content discovery phase of penetration testing.

What is the directory in robots.txt that isn't allowed to be viewed by web crawlers?
ANSWER: /staff-portal
What is the path of the secret area found in sitemap.xml?
ANSWER: /s3cr3t-area

TASK 3-MANUAL DISCOVERY — HEADERS & FRAMEWORKS STACK:

HTTP Headers

When a web server responds to a request, it includes HTTP headers that can reveal useful technical details. Headers like Server and X-Powered-By often expose the web server software and the language or framework the application runs on.

Run the following command against the Acme IT Support web server. The -v flag enables verbose output, which includes the response headers:

Using the following command helps display the response headers:

curl http://MACHINE_IP -v
None 
What is the flag value from the X-FLAG header?
ANSWER: THM{HEADER_FLAG}

The page source code also contained comments mentioning the framework version and a link to the framework documentation. By reviewing the framework documentation, the administration portal path was identified. Accessing the admin portal with the default credentials:

Username: admin
Password: admin

allowed successful login and retrieval of the hidden flag from the administration dashboard.

This demonstrates how HTTP headers and source code comments can accidentally expose sensitive information useful during reconnaissance and content discovery.

None None 
What is the flag from the framework's administration portal?
ANSWER: THM{CHANGE_DEFAULT_CREDENTIALS}

TASK 4-OSINT — SEARCH ENGINES & WEB TOOLS:

OSINT Techniques

Open-Source Intelligence (OSINT) refers to publicly available tools and resources used to gather information about a target website. These techniques help security testers discover hidden information, exposed files, technologies, and possible attack surfaces.

Google Hacking / Dorking

Google Dorking uses advanced search operators to find sensitive information indexed by search engines. These operators help filter search results to discover admin pages, documents, login portals, and other exposed content.

Some commonly used operators include:

Operator & Description

site:Limits results to a specific website

inurl:Searches for specific words in the URL

filetype:Finds specific file types such as PDF or TXT

intitle:Searches for words in page titles

intext:Searches for words inside page content

Example:

site:tryhackme.com filetype:pdf

Wappalyzer

Wappalyzer is a browser extension and online tool used to identify the technologies running on a website. It can detect:

  • Web frameworks
  • CMS platforms
  • Programming languages
  • Analytics tools
  • CDN services
  • Server technologies
What Google dork operator limits results to a specific site?
ANSWER: site:
What online tool and browser extension identifies what technologies a website is running?
ANSWER: Wappalyzer

TASK 5-OSINT REPOSITORIES & ARCHIVES:

Wayback Machine

The Wayback Machine(opens in new tab) is an archive of the Internet dating back to the late 1990s. Search for a domain, and you'll see every snapshot captured over time. This is useful for finding pages that have been removed from the live site but may still be accessible: old login forms, forgotten API endpoints, or content that was published briefly before being taken down.

GitHub

Git(opens in new tab) is a version control system that tracks changes to files over time. GitHub is the most widely used cloud-hosted platform for Git repositories. Developers sometimes accidentally commit sensitive data: API keys, credentials, configuration files, and .env files, before realising the repository is public.

Search GitHub for the company name or domain you're targeting. Once you find a relevant repository, look through the commit history, not just the current files. Sensitive data is often removed in a later commit, but remains in the history.

S3 Buckets

Amazon S3(Simple Storage Service) is a cloud storage platform that many organisations use to host files and static website content. The URL format for an S3 bucket is https://{name}.s3.amazonaws.com. Bucket owners set permissions, but misconfigurations are common: a publicly accessible bucket can expose files that were never meant to be seen.

Common naming patterns include {company}-assets, {company}-backup, {company}-www, and {company}-dev. Try these patterns against your target's company name. You can also find bucket URLs in the website's page source or in GitHub repositories.

What is the website address for the Wayback Machine?
ANSWER: https://web.archive.org/
What URL format do Amazon S3 buckets end in? (Answer starts with a .)
ANSWER: .s3.amazonaws.com

TASK 6-AUTOMATED DISCOVERY-GOBUSTER FUNDAMENTALS:

Automated Content Discovery with Gobuster

Manual testing and OSINT techniques are useful during reconnaissance, but automated tools help discover hidden content much faster. Automated content discovery works by sending thousands of requests to a target web server using predefined wordlists containing common directory and file names.

Gobuster

Gobuster is a fast and open-source enumeration tool written in Go. It is commonly used for:

  • Directory and file enumeration
  • DNS subdomain discovery
  • Virtual host enumeration

The tool is pre-installed on both Kali Linux and the TryHackMe AttackBox.

Basic command syntax:

gobuster dir -u http://MACHINE_IP -w /usr/share/wordlists/SecLists/Discovery/Web-Content/common.txt

Important Flags

Flag & Description

-uSpecifies the target URL

-wSpecifies the wordlist path

-tSets the number of threads

-xSearches for specific file extensions

-oSaves output to a file

-rFollows redirects

Scan Results

The Gobuster scan revealed several accessible files and directories on the target web application, including:

  • /contact
  • /news
  • /customers
  • /development.log
  • /private
  • /robots.txt
  • /sitemap.xml

Interesting findings included:

  • /customers redirected to a login portal
  • /development.log may contain sensitive information
  • /private appeared to be a restricted directory

These discovered endpoints help security testers understand the application's structure and identify potential areas for further testing during penetration testing engagements.

What is the name of the directory beginning with /mo that was discovered?
ANSWER: /monthly
What is the name of the log file that was discovered?
ANSWER: /development.log

TASK 7-AUTOMATED DISCOVERY-SUBDOMAINS & VIRTUAL HOSTS:

DNS and Virtual Host Enumeration

Gobuster also supports dns and vhost modes, which help discover hidden subdomains and virtual hosts during web application reconnaissance.

Subdomains vs Virtual Hosts

Before enumeration, it is important to understand the difference between subdomains and virtual hosts:

  • Subdomains are resolved using DNS records.
  • Example: blog.example.thm
  • Virtual Hosts (vhosts) are configured on the web server and identified using the Host HTTP header.
  • Multiple websites can run on the same IP address.

Gobuster provides separate modes for both:

  • dns → Subdomain enumeration
  • vhost → Virtual host enumeration

dns Mode

The dns mode brute-forces possible subdomains using a wordlist.

Example command:

gobuster dns -d example.thm -w /usr/share/wordlists/SecLists/Discovery/DNS/subdomains-top1million-5000.txt --wildcard

Important Flags

Flag & Description

-dSpecifies the target domain

-wSpecifies the wordlist

--wildcardForces enumeration even with wildcard DNS

-iDisplays resolved IP addresses

Discovered Subdomains

The scan identified several subdomains, including:

These discovered subdomains may contain separate applications or vulnerable services not available on the main website.

vhost Mode

The vhost mode discovers hidden virtual hosts configured on the target web server.

Example command:

gobuster vhost -u http://MACHINE_IP --domain example.thm -w /usr/share/wordlists/SecLists/Discovery/DNS/subdomains-top1million-5000.txt --append-domain --exclude-length 250-320

Important Flags

Flag & Description

--domainSpecifies the target domain

--append-domainAppends the domain to each wordlist entry

--exclude-lengthFilters common response sizes

During the virtual host enumeration, no additional valid virtual hosts were discovered. This indicates that no hidden web applications or virtual hosts were configured beyond the main application.

DNS and virtual host enumeration are important during penetration testing because organizations often host development, staging, or forgotten applications on subdomains that may contain vulnerabilities.

None 
Apart from dns and -w, which shorthand flag is required for dns mode?

ANSWER: -d
How many virtual hosts on acmeitsupport.thm respond with status code 200?
ANSWER: 3

TASK 8-CONCLUSION:

Content discovery is one of the most important phases of web application reconnaissance. The techniques in this room work together: manual checks surface quick wins, OSINT finds information the target has already shared publicly, and automated tools cover the breadth that neither approach can do alone.

Here's a quick recap of what was covered:

Method Techniques Manual robots.txt, sitemap.xml, favicon fingerprinting, HTTP headers, framework stack OSINT Google dorking, Wappalyzer, Wayback Machine, GitHub, S3 buckets Automated Gobuster dir, dns, and vhost modes A good content discovery workflow runs all three methods against a target before moving to exploitation. The directories and files you find here feed directly into later stages of a penetration test.

HAPPY HACKINGGG!!!!!!