robots.txt generator · free

Tell every AI crawler it's welcome.

robots.txt is the front door AI engines knock on first. Explicitly welcome the crawlers that produce citations, make a deliberate call on training bots, declare your sitemap, and keep utility paths private — built to RFC 9309 with the policies OpenAI, Anthropic, Google, and Perplexity document.

Prefill from your website

Forge

We read your existing robots.txt and sitemap declaration, then import the settings you can tune below.

Site origin *

Protocol + host only. Paths live in the rules below.

Sitemap URL

Defaults to /sitemap.xml. The Sitemap: line is the single most valuable line for discovery.

Crawler policy

AI search & assistant crawlers10 botsRecommended

OAI-SearchBot, ChatGPT-User, Claude-SearchBot, PerplexityBot, DuckAssistBot, and friends — the fetches that become live citations in AI answers.

AI training crawlers8 bots

GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot — models learn from your content. Separate decision from citations; most brands allow.

Web search engines3 bots

Googlebot, Bingbot, Applebot — foundation visibility and AI Overviews grounding.

Private pathsblocked for unlisted crawlers

Also apply these blocks to the explicit AI/search groups (off = AI groups stay fully open)

Crawl-delay

Google ignores it and several AI vendors don't document support — it can only throttle you.

1 issue to fix

21 bots welcomed

1.1 KB

robots.txt

# robots.txt for 
# AI search, assistant, and training crawlers are explicitly welcomed.
# Generated with AEO Surge — https://www.aeosurge.com/tools/robots-txt-generator

# Default policy for every crawler not listed below
User-agent: *
Disallow: /api/
Allow: /

# AI search & assistant crawlers — these fetches become live citations in ChatGPT, Perplexity, Copilot, and Claude
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: DuckAssistBot
User-agent: Amazonbot
User-agent: YouBot
User-agent: MistralAI-User
Allow: /

# AI training crawlers — models learn from this content (a separate decision from citations; remove this group to opt out)
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
User-agent: meta-externalagent
User-agent: cohere-ai
User-agent: Bytespider
Allow: /

# Web search engines — foundation visibility and AI Overviews grounding
User-agent: Googlebot
User-agent: Bingbot
User-agent: Applebot
Disallow: /api/
Allow: /

Sitemap: /sitemap.xml

Best-practice checklist

Wildcard group present — every unlisted crawler gets an explicit default policy.
Sitemap URLs must be absolute, e.g. `Sitemap: https://example.com/sitemap.xml`.
All 10 AI search & assistant crawlers evaluated as allowed — your pages can be cited.
18/20 tracked AI crawlers are explicitly named in the file — explicit groups signal intent and survive future wildcard edits.
1.1 KB — well within crawler limits.

Publish at your site root as /robots.txt with Content-Type: text/plain. Rules apply per host and port — a www ↔ apex redirect means one canonical robots.txt answers for both.