AI Search & GEO
AI robots.txt generator
Pick a policy — welcome all AI crawlers, allow AI search while blocking model training, or block AI entirely — and this tool generates the exact robots.txt rules, with each of the 16 major AI bots handled by name. Fine-tune any individual crawler, add your sitemap, and paste the result into your robots.txt.
Free · no signup
Stay citable in ChatGPT, Perplexity and Claude while opting out of model training.
Your robots.txt
# AI crawler policy — generated with spronta.com/tools # OpenAI: crawls pages to train openai models (also feeds search grounding) User-agent: GPTBot Disallow: / # Anthropic: crawls pages to improve claude models User-agent: ClaudeBot Disallow: / # Google: opt-out token for gemini training (does not affect google search rankings) User-agent: Google-Extended Disallow: / # Common Crawl: open web archive used as training data by many ai labs User-agent: CCBot Disallow: / # Apple: opt-out token for apple intelligence training User-agent: Applebot-Extended Disallow: / # Meta: crawls pages to train meta ai models User-agent: meta-externalagent Disallow: / # ByteDance: crawls pages to train bytedance/doubao models User-agent: Bytespider Disallow: / # Explicitly allowed: OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, DuckAssistBot, Amazonbot # Everyone else User-agent: * Disallow:
Merge these rules into your existing /robots.txt, then verify with the AI crawler access checker.
How to use the AI robots.txt generator
-
Pick a preset
The three presets encode the three coherent strategies. "Allow search, block training" is the most common choice for brands that want citations without contributing training data.
-
Fine-tune per bot
Tick or untick individual crawlers — each is labelled with its vendor and whether it feeds training, search indexes, or live fetches.
-
Merge and verify
Add the generated rules to your existing robots.txt, deploy, and verify the live file with the AI crawler access checker.
Which robots.txt policy should you choose for AI?
There are three coherent positions. Welcome all AI maximises visibility: every crawler may fetch your pages, your content can appear in AI answers and future models. Allow search, block training keeps you citable in ChatGPT, Perplexity, and Claude answers while opting out of model training — the middle path most commercial sites want. Block all AI opts out entirely; choose it knowingly, because AI answers will route around you to competitors.
What rarely makes sense is the accidental fourth position: a blanket block copy-pasted in 2023 that catches search crawlers too. If you blocked "AI bots" more than a year ago, re-check what you're actually blocking — the crawler landscape has split into training and search since then.
How the generated rules work
robots.txt is grouped by User-agent: each blocked bot gets its own group with Disallow: / (block everything), and a final User-agent: * group with an empty Disallow keeps normal crawlers unaffected. Rules are voluntary — reputable crawlers honour them, and every bot this tool manages has publicly committed to doing so.
One thing robots.txt cannot do: remove content from models that already trained on it. It controls future fetches only.
Frequently asked questions
How do I block GPTBot but stay in ChatGPT search?
Add "User-agent: GPTBot / Disallow: /" while leaving OAI-SearchBot and ChatGPT-User unblocked — that's exactly what the "Allow AI search, block training" preset generates. GPTBot trains models; OAI-SearchBot indexes for citations; ChatGPT-User fetches pages live during chats.
Will blocking AI crawlers hurt my Google rankings?
No — as long as you don't block Googlebot or Bingbot, which this tool never includes in its block lists. Google-Extended (Gemini training) and Applebot-Extended are pure opt-out tokens with no effect on search rankings.
Do AI companies actually respect robots.txt?
The major ones do — OpenAI, Anthropic, Google, Apple, Meta, and Perplexity all document their user-agents and state they honour robots.txt for the listed crawlers. robots.txt won't stop disreputable scrapers, but it is the standard, legally-referenced mechanism for stating your policy.