Google Dorking

Room Goal

The primary objective of this room is to understand the inner workings of search engines — specifically how they crawl, index, and rank web pages. Building on this foundational knowledge, the room guides us through leveraging advanced search operators (known as 'Google Dorks') to uncover exposed sensitive files, administrative panels, and security misconfigurations that have been publicly indexed on the internet.

Google is arguably the most famous example of "Search Engines", I mean who remembers Ask Jeeves? shudders

Now it might be rather patronising explaining how these "Search Engines" work, but there's a lot more going on behind the scenes then what we see. More importantly, we can leverage this to our advantage to find all sorts of things that a wordlist wouldn't. Researching as a whole — especially in the context of Cybersecurity encapsulates almost everything you do as a pentester. MuirlandOracle has created a fantastic room on learning the attitudes towards how to research, and what information you can gain from it exactly.

"Search Engines" such as Google are huge indexers — specifically, indexers of content spread across the World Wide Web.

These essentials in surfing the internet use "Crawlers" or "Spiders" to search for this content across the World Wide Web, which I will discuss in the next task.

Deconstructing the Web: How Search Engines Crawl and Index the Internet

When you type a query into a search engine, the results appear almost instantly. But have you ever wondered how the search engine knew those websites existed in the first place? Before we can master advanced recon techniques like Google Dorking, we must first understand the foundational mechanics operating under the hood of modern search engines: Crawling and Indexing.

Let's dive into how these automated bots map out the internet and look at the answers to this TryHackMe task.

What are Web Crawlers and How Do They Work?

At the core of every search engine is a tool known as a Web Crawler (often called a spider or a bot). A crawler's primary purpose is to discover content across the vast expanse of the World Wide Web. They achieve this discovery through two primary methods:

1. Pure Discovery: A crawler visits a specific URL directly, analyzes the website, and returns crucial data regarding the site's content type back to the search engine.
2. Link Traversal: Crawlers behave dynamically. When they scrape a page, they discover and follow any and all URLs embedded within that website. Much like a digital mapmaker, the bot traverses from one link to the next, spreading outwards to map everything it can possibly find.

Visualizing the Mechanics: From Scraping to Querying

Consider a basic abstraction of the process:

• Scraping Keywords: If a crawler visits a domain like mywebsite.com and identifies the keywords "Apple", "Banana", and "Pear", it logs this text into a local dictionary.
• Persistence & Storage: This collected dictionary is sent back and saved within the search engine's massive database. If a user subsequently queries the search engine for "Pears", the search engine cross-references its database and displays mywebsite.com as the matching result.

[Search Engine] ←-> [Crawler] — -> Visits: mywebsite.com — [Extracts Content] — [Keywords Dictionary] (Apple, Banana, Pear)

The Power of Link Interconnection

The real scale of a crawler becomes apparent when websites reference external domains. If mywebsite.com contains a hyperlink pointing to anotherwebsite.com, the crawler finishes scraping the first site and immediately follows the trail to the second.

If anotherwebsite.com contains keywords like "Tomatoes", "Strawberries", and "Pineapples", the search engine's dictionary updates to hold data for both domains:

mywebsite.comApples, Bananas, Pears

anotherwebsite.comTomatoes, Strawberries, Pineapples

Now, if a user searches for "Strawberries", the search engine instantly maps it to the referenced trail and yields anotherwebsite.com.

The Next Challenge: Hierarchy and Ranking

While this setup works perfectly when only a few sites exist, real-world scenarios involve millions of websites sharing similar keywords. If a user searches for "Tomatoes" and hundreds of pages contain that word, how does the engine decide the order of what gets displayed?

This introduces the concept of Ranking Algorithms, which evaluate domains based on relevance, authority, and popularity — a topic that forms the bedrock of modern Search Engine Optimization (SEO) and advanced intelligence gathering.

TryHackMe Task Solutions

Here are the direct solutions to the practical questions presented at the end of this task section:

Question 1

Name the key term of what a "Crawler" is used to do. This is known as a collection of resources and their locations.

Answer: Index

Question 2

What is the name of the technique that "Search Engines" use to retrieve this information about websites?

Answer: Crawling

Question 3

What is an example of the type of contents that could be gathered from a website?

Answer: Keywords

The Secret Point System: How Google Decides Who Wins the Internet

In our last section, we discovered how search engine bots crawl and index the web. But what happens when thousands of websites share the exact same keywords? Who gets the coveted #1 spot on Google's front page, and who gets buried on page 10?

This is where Search Engine Optimisation (SEO) comes into play. Think of it as a hidden digital point-scoring system where websites compete for Google's attention.

The Digital Scorecard: How Do You Win Points?

Google keeps its exact algorithms a closely guarded secret, but we know the core factors that boost a website's ranking. Search engines naturally prioritize domains that are "optimal" and easy to read.

Here is how you score points on the SEO leaderboard:

• Device & Browser Responsiveness: Does your website load beautifully on Google Chrome, Firefox, and — most importantly — mobile phones? If it looks broken on a smartphone, you lose points.
• Crawler-Friendliness (Sitemaps): Is your website easy for a bot to navigate? Using a "Sitemap" (a digital map of your website's pages) tells the crawler exactly where to go, earning you major bonus points.
• Keyword Relevance: The keywords must actually match what users look for. If a user searches for "Colours" and Google hasn't indexed any domain with that specific keyword, no results will show up.

Auditing the Web: Let's Look at TryHackMe's Score

To see this in action, web developers use optimization tools (like Google's Lighthouse/Site Analyser) to audit their websites. These tools look at four main metrics: Performance, Accessibility, Best Practices, and SEO.

During an audit of tryhackme.com, the platform scored an impressive 85/100 for SEO! This means TryHackMe has done an excellent job making sure search engines can easily read and rank their content.

TryHackMe Task Solutions

For this specific section, TryHackMe is simply asking you to explore the concept and use the SEO checkup tool to see how it works.

Question 1

Use the same SEO checkup tool… to see how their results compare…Answer: No answer needed

Demystifying Robots.txt: The Gateway to Web Crawling and Directory Security

When search engine bots or automated web scrapers visit a website, they don't explore blindly. They look for a specific guide map that dictates exactly where they are welcome and where doors are tightly locked. This roadmap is known as the robots.txt file.

Whether you are building websites, optimizing for search engines (SEO), or conducting a security assessment, understanding how robots.txt works is essential. In this article, we break down the core keywords, analyze structural rules, and walk through real-world configuration challenges to explain how this simple file governs web visibility.

What Exactly is Robots.txt?

The robots.txt file is a plain text file placed at the root directory of a web server. It acts as the initial point of interaction for search engine "Crawlers" or "Spiders" (like Googlebot or Bingbot). Before a crawler indexes any sub-page or media file on your domain, it explicitly requests this file to determine its crawling permissions.

Important Note: The robots.txt file works on a voluntary system. While reputable web crawlers strictly follow these directives, malicious actors or aggressive scrapers can choose to completely ignore them. Therefore, it should never be used as a primary security mechanism to hide truly confidential data.

Core Keywords and Syntax

The layout of a robots.txt file relies on a small set of straightforward directives. Let's look at the foundational keywords:

User-agentSpecifies the exact name of the crawler to which the rule applies. Using an asterisk (*) acts as a wildcard targeting all crawlers.

AllowIndicates the directories, file paths, or individual files that the specified crawler is permitted to index.

DisallowSpecifies the paths or structural extensions that the crawler is forbidden from indexing.

SitemapProvides an absolute URL pointing to the site's XML sitemap, making it easier for engines to discover new content.

Example 1: A Fully Open Architecture

If you want all crawlers to freely explore every corner of your website, a standard configuration looks like this:

User-agent: *
Allow: /
Sitemap: http://mywebsite.com/sitemap.xml

User-agent: *
Allow: /
Sitemap: http://mywebsite.com/sitemap.xml

Example 2: Blacklisting Sensitive Folders

By default, if a folder isn't explicitly blocked, a crawler assumes it is allowed to read it. To prevent indexation of private folders, we use the Disallow keyword:

User-agent: *
Disallow: /super-secret-directory/
Disallow: /not-a-secret/but-this-is/
Sitemap: http://mywebsite.com/sitemap.xml

User-agent: *
Disallow: /super-secret-directory/
Disallow: /not-a-secret/but-this-is/
Sitemap: http://mywebsite.com/sitemap.xml

In this case, crawlers will index the broad contents of /not-a-secret/, but will strictly drop out when hitting the sub-directory /but-this-is/.

Practical Breakdown: Question and Answer Walkthrough

Let's solidify these concepts by solving standard web architecture and configuration scenarios commonly found in development and security challenges.

Question 1: Where would "robots.txt" be located on the domain "ablog.com"?

• Answer: ablog.com/robots.txt
• Explanation: Web crawlers are programmatically coded to look for the robots file at the absolute root level of a hostname. Placing it anywhere else (e.g., [ablog.com/assets/robots.txt](https://ablog.com/assets/robots.txt)) will result in crawlers failing to find it, causing them to default to full indexation.

Question 2: If a website was to have a sitemap, where would that typically be located natively?

• Answer: /sitemap.xml
• Explanation: The standard root relative path for a website's map is /sitemap.xml. This file gives structured links to the crawler to ensure hidden or deep pages are easily indexed.

Question 3: How would we only allow "Bingbot" to index the website?

• Answer: User-agent: Bingbot
• Explanation: To restrict actions to specific search engines, replace the generic wildcard * with the exact identifier of the targeted crawler.

Question 4: How would we prevent a "Crawler" from indexing the specific directory "/dont-index-me/"?

• Answer: Disallow: /dont-index-me/
• Explanation: The Disallow directive tells the matched user-agent to skip parsing the directory path provided. The trailing slash ensures that everything inside that specific folder is restricted.

Question 5: What is the extension of a Unix/Linux system configuration file that we might want to hide from "Crawlers" to prevent data leakage?

• Answer: .conf
• Explanation: On Linux/Unix environments, systems and applications heavily rely on .conf (or .ini / .cfg) configuration files. If these are exposed under the public web directory, attackers can discover database credentials or server configurations. You can use regex pattern matching like Disallow: /*.conf$ to hide them uniformly.

Final Thoughts

Managing your robots.txt correctly is a subtle balancing act. While it is perfect for saving server bandwidth and controlling SEO visibility, exposing sensitive file paths inside a public robots.txt file acts as a map for malicious scouts looking for hidden directories. Keep your permissions clean, ensure proper placement, and always back up your public files with rigorous server-side authentication!

Understanding Sitemaps: The Digital Roadmap for Search Engines

Just as physical maps help travelers navigate unfamiliar territories, websites utilize Sitemaps to guide search engine crawlers through their structural layout. A sitemap is a specialized indicative resource that defines the essential pathways to discover content across a web domain.

Whether you are optimizing a site for SEO or conducting directory enumeration during a security assessment, understanding the blueprint of a website's structure is a core skill.

Why Are Sitemaps So Favourable for Search Engines?

Search engine crawlers (like Googlebot or Bingbot) process massive amounts of data daily. To optimize their resource consumption and efficiency, they prefer pre-mapped instructions over brute-force exploration.

Think of a sitemap as a pre-compiled wordlist containing valid URLs. Instead of blindly guessing directory paths, the crawler reads this file and hits the exact target pages immediately. Simply put: the easier a website is to crawl, the more optimized it is for the search engine.

Anatomy of a Sitemap: Visual vs. Code

Conceptually, a sitemap can be visualized as a hierarchical tree:

• Routes (Nested Content): Represented by directory paths (e.g., /Products/Groceries/Fruits).
• Actual Pages: The final destination content where users interact (e.g., individual blog posts or product pages).

However, in the real world, sitemaps don't look like graphic charts. They are strictly formatted in XML (Extensible Markup Language). An XML sitemap explicitly lists the location (<loc>) of each URL and the date it was last modified (<lastmod>), ensuring search engines index fresh content efficiently.

Technical Walkthrough: Challenge Solutions

When interacting with technical training platforms like TryHackMe, identifying the exact strings expected by automated grading systems is crucial. Let's analyze the correct structural answers based on the site mapping text:

1. What is the typical file structure of a "Sitemap"?

• Answer: xml
• Explanation: Sitemaps leverage the standard 3-letter Extensible Markup Language (.xml) format, enabling spiders to easily parse hierarchical tree tags.

2. What real life example can "Sitemaps" be compared to?

• Answer: map
• Explanation: As defined in the introductory text, web sitemaps serve the exact same spatial guidance purpose for digital bots as real-life geographical maps do for physical navigation.

3. Name the keyword for the path taken for content on a website

• Answer: route
• Explanation: The specific 5-letter technical keyword representing a path structurally taken to point crawlers toward nested sub-directories or categories is a route.

Summary

A sitemap is the perfect companion to a robots.txt file. While robots.txt tells crawlers where they cannot go, the sitemap gives them an explicit, efficient map of exactly where they should go. Keeping this file updated guarantees better indexing, clean architecture visibility, and robust web presence management.

Google Dorking: Unleashing the Full Power of Advanced Search Operators

Most web users interact with Google on a very basic level — typing a few keywords and scrolling through the top results. However, under the hood, Google's crawler indexes an astronomical amount of public data. With the right commands, you can filter out the noise and extract precise, hidden intelligence. This technique is known as Google Dorking.

In this article, we'll explore how advanced search operators work and how security professionals leverage them for Open Source Intelligence (OSINT).

Refining Queries with Essential Operators

Google allows the use of specific programming-like operators to narrow down search criteria. For instance, putting exact phrases inside quotation marks tells Google to drop arbitrary matches and only return that specific phrase.

But it gets much more powerful when you target specific domains. By using the site: operator, you can force Google to display results strictly from one web platform.

As seen above, a standard search for a term might bring up millions of mixed results. If we want to refine this to only scan a trusted news portal, we restructure the query using site:bbc.co.uk.

What Makes Google Dorking So Appealing?

First and foremost: it is entirely legal. Google Dorking simply discovers information that is already publicly available and indexed on the open web. However, it often uncovers sensitive directories because system administrators forget to configure their server permissions correctly.

Here are the primary terms utilized in advanced dorking:

**TermActionfiletype:**Search for a specific file by its extension (e.g., PDF, DOCX, TXT).**cache:**View Google's cached version of a specified URL.**intitle:**Forces the specified phrase to appear directly in the webpage's title.

For example, if an analyst wants to discover every publicly exposed PDF file on a specific corporate domain, they can chain these operators together seamlessly: site:bbc.co.uk filetype:pdf.

The Dark Side: Directory Traversal

When combined cleverly, dorks can reveal internal index layouts — often referred to as Directory Traversal. Searching for terms like intitle:"index of" can lead directly to exposed server directories where private downloads, backups, or configuration setups reside completely unprotected.

Challenge Walkthrough & Solutions

Let's apply these concepts to practical questions often encountered in web architecture and OSINT rooms:

1. What would be the format used to query the site bbc.co.uk about flood defences?

• Answer: site:bbc.co.uk flood defences
• Walkthrough: To target the domain, we initialize the query with site:bbc.co.uk and append the explicit search phrase flood defences right after it.

2. What term would you use to search by file type?

• Answer: filetype:
• Walkthrough: As documented in the core operator syntax table, appending the filetype: selector followed by an extension (like pdf or xml) restricts findings exclusively to that format.

3. What term can we use to look for login pages?

• Answer: intitle:
• Walkthrough: Since login interfaces almost universally contain the word "Login" or "Signin" inside their HTML head titles, forcing a search with intitle:login is the most efficient way to map authentication portals.

Conclusion :)

Google Dorking highlights a critical lesson in web management: obscurity is not security. Just because you don't link a page on your homepage doesn't mean Google won't find it. Always ensure your sensitive paths are protected by strict access controls rather than just hoping they stay unindexed!