Post cover image
Photo by Andriyko Podilnyk on Unsplash

July 23, 2026

One Command Pulled URLs from four different sources at once; I was not ready for the results.

Why Gau replaced three separate tools in my recon workflow, and every command you need actually to use it right

By Yamini Yadav_369

7 min read

I used to run Wayback Machine URLs, then separately check a couple of other archive sources by hand, then merge everything myself before I even started filtering. It worked, but it was slow, and I was repeating the same manual merge step on every target.

Then someone in a bug bounty community mentioned gau to me, short for get all URLs. I ran it on a target I had already tested with my old manual process, expecting maybe a few extra results at best. Instead, I got URLs from sources I was not even checking before, all combined into one clean list, in one command. That was the moment I stopped doing recon the slow way.

None

If you are still pulling archived URLs from one source at a time and merging them by hand, this tool fixes that problem completely. Let me walk you through exactly what gau is, how it works, and every command you need to use it properly.

What Is GAU

Gau, which stands for get all URLs, is a command-line tool built by lc that fetches known URLs for a domain from multiple sources at the same time. Instead of checking just the Wayback Machine as some tools do, gau pulls from the Wayback Machine, Common Crawl, AlienVault's Open Threat Exchange, and URLScan, all in a single run.

Think of it like this. If waybackurls is one person checking one archive, gau is a small team checking four different archives simultaneously and handing you back one combined, deduplicated report instead of four separate messy lists you would have to merge yourself.

This matters because different archive sources catch different things. A URL that never got picked up by the Wayback Machine might show up in Common Crawl's dataset instead. By checking multiple sources at once, you genuinely see more of a domain's history than any single source alone can give you.

Why This Tool Actually Matters

The core idea behind this whole category of tools is the same one I have said before: old forgotten URLs often reveal things the current live site does not. But the real gap Gau solves is coverage.

Relying on just one archive source means you are trusting that one dataset caught everything worth catching, and it did not. Common Crawl indexes the web on its own schedule and scope. URLScan captures URLs from actual scans people submit. AlienVault's OTX pulls from threat intelligence data. Each source has blind spots the others fill in.

For a bug bounty hunter or pentester, this means running GAU instead of a single source tool is almost always going to surface more old endpoints, more forgotten parameters, and more leftover files than checking just one archive would. More coverage means more chances at finding something that matters.

How GAU Actually Works

Let us slow down and look at what is actually happening when you run this tool.

It queries multiple APIs in parallel: When you give GAU a domain, it sends requests out to each of its configured sources at the same time rather than one after another. This is part of why it feels fast even though it is checking several places.

Each source returns its own archived URL data: The Wayback Machine returns what it has crawled historically. Common Crawl returns URLs from its own massive web crawling dataset. URLScan returns URLs from scans submitted by users and automated systems. OTX returns URLs tied to threat intelligence records.

Gau merges and deduplicates the results: Once all four sources respond, Gau combines everything into a single list and removes duplicate entries so you are not stuck scrolling through the same URL five times just because different sources happened to catch it too.

No direct interaction with the target: Just like waybackurls, GAU never sends a single request to the actual target domain during this process. Everything comes from third-party sources that already did the crawling. This makes it a completely passive and safe technique to run on any domain.

Installing GAU

GAU is written in Go, so make sure Go is installed first.

go install github.com/lc/gau/v2/cmd/gau@latest

Confirm it is installed correctly:

gau --version

If a version number comes back, you are ready to run it.

Let us go through each command properly, one at a time, so you understand exactly what you are asking the tool to do.

1. Basic domain lookup

Command:

echo example.com | gau

What it means in plain English: "Take this domain, check all four sources you know about, and give me every URL you can find for it."

When to use it: This is your default starting command for any target. Run this first before anything more specific.

2. Running it on multiple domains

Command:

cat domains.txt | gau

What it means: "Here is a list of domains in this file, one per line. Go check all sources for each one and give me the combined results."

When to use it: Great when a target company owns several domains, and you want one unified list covering everything at once.

3. Saving output to a file

Command:

echo example.com | gau > urls.txt

What it means: "Do the lookup, but save every result into urls.txt instead of just printing it to the screen."

When to use it: Always do this. These lists get large fast, and you need them saved so you can filter through properly afterwards.

4. Including subdomains automatically

Command:

echo example.com | gau --subs

What it means: "Check the main domain, but also pull in URLs for any subdomains of example.com that the sources have on record."

When to use it: This gives you much wider coverage in one shot without needing to run a separate subdomain enumeration tool first.

5. Filtering by specific sources only

Command:

echo example.com | gau --providers wayback,otx

What it means: "Only check the Wayback Machine and AlienVault OTX, skip the other two sources."

When to use it: Useful if you already know one source tends to give noisy or irrelevant results for a certain type of target, or if you want a faster run by narrowing the sources checked.

6. Filtering out certain file extensions

Command:

echo example.com | gau --blacklist png,jpg,gif,svg,css

What it means: "Get me all the URLs, but skip anything ending in these file types since they are usually just images or styling files, not useful for testing."

When to use it: This cleans up your results immediately by removing a ton of noise from static assets that rarely matter for security testing.

7. Setting a threads limit

Command:

echo example.com | gau --threads 5

What it means: "Run the lookup using 5 threads at a time instead of the default, to control how much load this puts on your own system or the sources being queried."

When to use it: Helpful if you are running this on a lower-powered machine or if you want to be more conservative with your request rate to the source APIs.

8. Filtering results by date

Command:

echo example.com | gau --from 202001 --to 202312

What it means: "Only give me URLs that were archived between January 2020 and December 2023."

When to use it: Useful when you want to focus specifically on a certain time window, for example right before or after a known company event like a rebrand, merger, or major platform migration.

9. Saving output in JSON format

Command:

echo example.com | gau --json > urls.json

What it means: "Give me the results, but format them as JSON and save into this file instead of a plain text list."

When to use it: Best when you plan to process the output using a script or feed it into another automation pipeline that expects structured data.

10. Combining with subfinder for full subdomain coverage

Command:

subfinder -d example.com -silent | gau

What it means: "First find every subdomain for example.com, then run gau against that entire list so you get archived URLs across the whole domain footprint, not just the main site."

When to use it: This combination gives you the widest possible coverage, since you are feeding gau a much bigger list of targets than the single root domain alone.

11. Filtering for sensitive keywords in the output

Command:

cat urls.txt | grep -Ei "admin|api|token|secret|config|backup"

What it means: "From the saved list, only show me lines that contain any of these sensitive-sounding words, ignoring capital letters."

When to use it: A fast way to cut through a massive list and jump straight to the URLs most worth checking manually.

12. Checking which old URLs are still alive

Command:

cat urls.txt | httpx -silent -mc 200

What it means: "Take this list of old URLs and check each one right now, only show me the ones that actually respond with a 200 OK status."

When to use it: Filters your huge list down to the handful of paths that are still live today, which is usually the most immediately actionable subset of results.

Here is how I actually run gau on a real engagement, step by step.

  1. Run subfinder first to get every subdomain for the target.
  2. Pipe that subdomain list straight into gau to pull archived URLs across the entire domain footprint at once.
  3. Blacklist common static file extensions immediately to cut out noise from the start.
  4. Save the full clean output to a file and remove duplicates with sort -u.
  5. Filter for sensitive keywords first, since this usually surfaces the highest value leads fastest.
  6. Separately filter for parameters and JavaScript files for manual review.
  7. Run the whole cleaned list through httpx to confirm what is still live today.
  8. Manually verify anything flagged before including it in a report.

This exact flow has consistently turned up more results for me than running any single archive source tool on its own, purely because gau is checking more places at the same time.

None

Common Mistakes Beginners Make

The biggest mistake is not blacklisting static file extensions from the start. Without this, a huge chunk of your output is just image and stylesheet files that add nothing to your testing and just slow down manual review later.

Another common mistake is skipping the subdomain step entirely and only checking the root domain. Since gau supports pulling subdomains directly with a flag, there is no good reason to miss this wider coverage on any real engagement.

People also tend to trust old URLs without verifying anything. Just because gau returned a URL does not mean it still works or leads anywhere useful today. Always confirm live status and manually review before treating anything as a real finding.

Gau earns its place in a recon toolkit by solving a very specific problem well, checking multiple archive sources at once instead of forcing you to run several tools separately and merge the mess yourself. It is not doing anything mysterious under the hood; it is simply combining what already exists across different datasets into one clean output.

Blacklist the noise early, always include subdomains, filter smart, and verify before reporting. Stick to that process, and gau will consistently give you more coverage than relying on a single archive source ever could.

If this helped you understand gau better, follow The First Digit for more practical security tool breakdowns written the way they should be: simple, honest, and based on real testing experience.