__ _______________________ _________._________________________ \_ _____/ \______ \ / _ \ / _____/ / _____/ | | \_ _____/ | __) | _/ / /_\ \ / \ ___ / \ ___ | | | __)_ | \ | | \ / | \ \ \_\ \ \ \_\ \ | |___ | \ \___ / |____|_ / \____|__ / \______ / \______ / |_______ \ /_______ / \/ \/ \/ \/ \/ \/ \/

gau Cheatsheet

← Back to cheatsheets

← Home

gau (Get All URLs) fetches known URLs from AlienVault's Open Threat Exchange, the Wayback Machine, Common Crawl, and URLScan for a given domain. Great for discovering endpoints and parameters.

Basic Usage

  • gau <domain> - Fetch URLs for domain
  • echo "example.com" | gau - Pipe domain
  • cat domains.txt | gau - Multiple domains

Options

  • --blacklist - Comma-separated list of extensions to skip
  • --fc - Filter by status code (comma-separated)
  • --from - Fetch URLs from date (YYYYMM)
  • --ft - Filter to status code (show only these)
  • --fp - Remove query strings
  • --json - Output as JSON
  • --mc - Match status code (comma-separated)
  • --mt - Match MIME type
  • --o - Output file
  • --providers - Comma-separated providers (wayback, otx, commoncrawl, urlscan)
  • --retries - Number of retries
  • --subs - Include subdomains
  • --threads - Number of threads
  • --to - Fetch URLs to date (YYYYMM)
  • --verbose - Verbose output

Providers

  • wayback - Wayback Machine (web.archive.org)
  • otx - AlienVault Open Threat Exchange
  • commoncrawl - Common Crawl
  • urlscan - URLScan.io

Common Examples

Basic Fetch

gau example.com

Get all known URLs.

Include Subdomains

gau --subs example.com

Include all subdomain URLs.

Save to File

gau example.com --o urls.txt

Save results to file.

Exclude Extensions

gau --blacklist png,jpg,gif,css,js example.com

Skip static assets.

Filter Date Range

gau --from 202301 --to 202312 example.com

URLs from 2023 only.

Specific Providers

gau --providers wayback,otx example.com

Use only Wayback and OTX.

JSON Output

gau --json example.com

Output in JSON format.

Multiple Domains

cat domains.txt | gau --threads 5

Process multiple domains.

Remove Query Strings

gau --fp example.com | sort -u

Unique paths without parameters.

Pipeline Examples

Find Parameters

gau example.com | grep "=" | sort -u

Find URLs with parameters.

Find JS Files

gau example.com | grep "\.js$" | sort -u

Extract JavaScript files.

Find API Endpoints

gau example.com | grep -E "(api|v1|v2)" | sort -u

Find potential API endpoints.

With httpx

gau example.com | httpx -silent -status-code

Check which URLs are live.

Find Interesting Files

gau example.com | grep -E "\.(json|xml|config|sql|backup|bak)$"

Find potentially sensitive files.

Configuration

Config File Location

  • ~/.gau.toml - Configuration file

Example Config

threads = 5
retries = 3
blacklist = ["png", "jpg", "gif"]
providers = ["wayback", "otx", "urlscan"]

Tips

  • Use --blacklist to filter out static files
  • Combine with grep to find interesting patterns
  • Pipe to httpx to check live URLs
  • Look for parameters for potential injection points
  • Historical URLs may reveal removed functionality
  • Great for bug bounty recon
  • Combine with waybackurls for comprehensive coverage
  • Check JS files for API keys and secrets

← Back to cheatsheets

← Home