210 lines
7.5 KiB
HTML
210 lines
7.5 KiB
HTML
<!DOCTYPE html>
|
|
<html lang="en">
|
|
<head>
|
|
<meta charset="UTF-8">
|
|
<meta name="viewport" content="width=device-width, initial-scale=1.0">
|
|
<meta http-equiv="Content-Security-Policy" content="default-src 'self'; script-src 'self'; style-src 'self'; font-src 'self' data:; img-src 'self' data:; connect-src 'self'; base-uri 'self'; form-action 'self' https://defcon.social https://bsky.app;">
|
|
<meta http-equiv="X-Content-Type-Options" content="nosniff">
|
|
<link rel="stylesheet" href="../assets/css/style.css">
|
|
<link rel="icon" type="image/x-icon" href="../favicon.ico">
|
|
<script>
|
|
// Apply theme immediately to prevent flash
|
|
(function() {
|
|
const theme = localStorage.getItem('theme') ||
|
|
(window.matchMedia && window.matchMedia('(prefers-color-scheme: dark)').matches ? 'dark' : 'light');
|
|
document.documentElement.setAttribute('data-theme', theme);
|
|
})();
|
|
</script>
|
|
<title>gau Cheatsheet - Cheatsheets - Launch Pad</title>
|
|
</head>
|
|
<body>
|
|
<button class="theme-toggle" id="themeToggle" aria-label="Toggle dark mode">
|
|
<svg class="theme-icon theme-icon-moon" xmlns="http://www.w3.org/2000/svg" width="24" height="24" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><path d="M21 12.79A9 9 0 1 1 11.21 3 7 7 0 0 0 21 12.79z"></path></svg>
|
|
<svg class="theme-icon theme-icon-sun" xmlns="http://www.w3.org/2000/svg" width="24" height="24" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" style="display: none;"><circle cx="12" cy="12" r="5"></circle><line x1="12" y1="1" x2="12" y2="3"></line><line x1="12" y1="21" x2="12" y2="23"></line><line x1="4.22" y1="4.22" x2="5.64" y2="5.64"></line><line x1="18.36" y1="18.36" x2="19.78" y2="19.78"></line><line x1="1" y1="12" x2="3" y2="12"></line><line x1="21" y1="12" x2="23" y2="12"></line><line x1="4.22" y1="19.78" x2="5.64" y2="18.36"></line><line x1="18.36" y1="5.64" x2="19.78" y2="4.22"></line></svg>
|
|
</button>
|
|
<br/><br/>
|
|
<div class="name">
|
|
__ _______________________ _________._________________________
|
|
\_ _____/ \______ \ / _ \ / _____/ / _____/ | | \_ _____/
|
|
| __) | _/ / /_\ \ / \ ___ / \ ___ | | | __)_
|
|
| \ | | \ / | \ \ \_\ \ \ \_\ \ | |___ | \
|
|
\___ / |____|_ / \____|__ / \______ / \______ / |_______ \ /_______ /
|
|
\/ \/ \/ \/ \/ \/ \/
|
|
</div>
|
|
<div class="blog-page-header">
|
|
<div class="blog-header-content">
|
|
<a href="/cheatsheets" class="back-link" title="Back to Cheatsheets">
|
|
<svg xmlns="http://www.w3.org/2000/svg" width="42" height="42" viewBox="0 0 24 24" class="home-icon"><path fill="currentColor" d="M10 20v-6h4v6h5v-8h3L12 3 2 12h3v8z"/></svg>
|
|
</a>
|
|
<h1 class="blog-page-title">gau Cheatsheet</h1>
|
|
</div>
|
|
</div>
|
|
<div class="blog-post-container">
|
|
<div class="blog-posts-container" style="max-width: 900px; margin: 0 auto;">
|
|
<div class="blog-post">
|
|
<div class="blog-post-content">
|
|
<p><a href="index.html">← Back to cheatsheets</a></p>
|
|
<p><a href="../index.html">← Home</a></p>
|
|
<hr>
|
|
<p>gau (Get All URLs) fetches known URLs from AlienVault's Open Threat Exchange, the Wayback Machine, Common Crawl, and URLScan for a given domain. Great for discovering endpoints and parameters.</p>
|
|
<hr>
|
|
<h2>Basic Usage</h2>
|
|
<ul>
|
|
<li>gau <domain> - Fetch URLs for domain</li>
|
|
</ul>
|
|
<ul>
|
|
<li>echo "example.com" | gau - Pipe domain</li>
|
|
</ul>
|
|
<ul>
|
|
<li>cat domains.txt | gau - Multiple domains</li>
|
|
</ul>
|
|
<hr>
|
|
<h2>Options</h2>
|
|
<ul>
|
|
<li>--blacklist - Comma-separated list of extensions to skip</li>
|
|
</ul>
|
|
<ul>
|
|
<li>--fc - Filter by status code (comma-separated)</li>
|
|
</ul>
|
|
<ul>
|
|
<li>--from - Fetch URLs from date (YYYYMM)</li>
|
|
</ul>
|
|
<ul>
|
|
<li>--ft - Filter to status code (show only these)</li>
|
|
</ul>
|
|
<ul>
|
|
<li>--fp - Remove query strings</li>
|
|
</ul>
|
|
<ul>
|
|
<li>--json - Output as JSON</li>
|
|
</ul>
|
|
<ul>
|
|
<li>--mc - Match status code (comma-separated)</li>
|
|
</ul>
|
|
<ul>
|
|
<li>--mt - Match MIME type</li>
|
|
</ul>
|
|
<ul>
|
|
<li>--o - Output file</li>
|
|
</ul>
|
|
<ul>
|
|
<li>--providers - Comma-separated providers (wayback, otx, commoncrawl, urlscan)</li>
|
|
</ul>
|
|
<ul>
|
|
<li>--retries - Number of retries</li>
|
|
</ul>
|
|
<ul>
|
|
<li>--subs - Include subdomains</li>
|
|
</ul>
|
|
<ul>
|
|
<li>--threads - Number of threads</li>
|
|
</ul>
|
|
<ul>
|
|
<li>--to - Fetch URLs to date (YYYYMM)</li>
|
|
</ul>
|
|
<ul>
|
|
<li>--verbose - Verbose output</li>
|
|
</ul>
|
|
<hr>
|
|
<h2>Providers</h2>
|
|
<ul>
|
|
<li>wayback - Wayback Machine (web.archive.org)</li>
|
|
<li>otx - AlienVault Open Threat Exchange</li>
|
|
<li>commoncrawl - Common Crawl</li>
|
|
<li>urlscan - URLScan.io</li>
|
|
</ul>
|
|
<hr>
|
|
<h2>Common Examples</h2>
|
|
<h3>Basic Fetch</h3>
|
|
<pre><code>gau example.com</code></pre>
|
|
<p>Get all known URLs.</p>
|
|
<h3>Include Subdomains</h3>
|
|
<pre><code>gau --subs example.com</code></pre>
|
|
<p>Include all subdomain URLs.</p>
|
|
<h3>Save to File</h3>
|
|
<pre><code>gau example.com --o urls.txt</code></pre>
|
|
<p>Save results to file.</p>
|
|
<h3>Exclude Extensions</h3>
|
|
<pre><code>gau --blacklist png,jpg,gif,css,js example.com</code></pre>
|
|
<p>Skip static assets.</p>
|
|
<h3>Filter Date Range</h3>
|
|
<pre><code>gau --from 202301 --to 202312 example.com</code></pre>
|
|
<p>URLs from 2023 only.</p>
|
|
<h3>Specific Providers</h3>
|
|
<pre><code>gau --providers wayback,otx example.com</code></pre>
|
|
<p>Use only Wayback and OTX.</p>
|
|
<h3>JSON Output</h3>
|
|
<pre><code>gau --json example.com</code></pre>
|
|
<p>Output in JSON format.</p>
|
|
<h3>Multiple Domains</h3>
|
|
<pre><code>cat domains.txt | gau --threads 5</code></pre>
|
|
<p>Process multiple domains.</p>
|
|
<h3>Remove Query Strings</h3>
|
|
<pre><code>gau --fp example.com | sort -u</code></pre>
|
|
<p>Unique paths without parameters.</p>
|
|
<hr>
|
|
<h2>Pipeline Examples</h2>
|
|
<h3>Find Parameters</h3>
|
|
<pre><code>gau example.com | grep "=" | sort -u</code></pre>
|
|
<p>Find URLs with parameters.</p>
|
|
<h3>Find JS Files</h3>
|
|
<pre><code>gau example.com | grep "\.js$" | sort -u</code></pre>
|
|
<p>Extract JavaScript files.</p>
|
|
<h3>Find API Endpoints</h3>
|
|
<pre><code>gau example.com | grep -E "(api|v1|v2)" | sort -u</code></pre>
|
|
<p>Find potential API endpoints.</p>
|
|
<h3>With httpx</h3>
|
|
<pre><code>gau example.com | httpx -silent -status-code</code></pre>
|
|
<p>Check which URLs are live.</p>
|
|
<h3>Find Interesting Files</h3>
|
|
<pre><code>gau example.com | grep -E "\.(json|xml|config|sql|backup|bak)$"</code></pre>
|
|
<p>Find potentially sensitive files.</p>
|
|
<hr>
|
|
<h2>Configuration</h2>
|
|
<h3>Config File Location</h3>
|
|
<ul>
|
|
<li>~/.gau.toml - Configuration file</li>
|
|
</ul>
|
|
<h3>Example Config</h3>
|
|
<pre><code>threads = 5
|
|
retries = 3
|
|
blacklist = ["png", "jpg", "gif"]
|
|
providers = ["wayback", "otx", "urlscan"]</code></pre>
|
|
<hr>
|
|
<h2>Tips</h2>
|
|
<ul>
|
|
<li>Use --blacklist to filter out static files</li>
|
|
</ul>
|
|
<ul>
|
|
<li>Combine with grep to find interesting patterns</li>
|
|
</ul>
|
|
<ul>
|
|
<li>Pipe to httpx to check live URLs</li>
|
|
</ul>
|
|
<ul>
|
|
<li>Look for parameters for potential injection points</li>
|
|
</ul>
|
|
<ul>
|
|
<li>Historical URLs may reveal removed functionality</li>
|
|
</ul>
|
|
<ul>
|
|
<li>Great for bug bounty recon</li>
|
|
</ul>
|
|
<ul>
|
|
<li>Combine with waybackurls for comprehensive coverage</li>
|
|
</ul>
|
|
<ul>
|
|
<li>Check JS files for API keys and secrets</li>
|
|
</ul>
|
|
<hr>
|
|
<p><a href="index.html">← Back to cheatsheets</a></p>
|
|
<p><a href="../index.html">← Home</a></p>
|
|
</div>
|
|
</div>
|
|
</div>
|
|
</div>
|
|
<script async type="text/javascript" src="../blog/analytics.js"></script>
|
|
<script src="../theme.js"></script>
|
|
</body>
|
|
</html>
|