AI Search & GEO
AI crawler access checker
This tool fetches your live robots.txt and tests it against 18 AI and search crawlers — including OpenAI's GPTBot, Anthropic's ClaudeBot, PerplexityBot, and Google-Extended — showing exactly which ones may fetch your pages and which rule blocks the rest. If AI crawlers can't read your site, AI search can't cite it.
Free · no signup Checks your live site
How to use the AI crawler access checker
-
Enter your domain
The tool fetches your live /robots.txt — the same file every crawler reads first.
-
Read the verdict
Each crawler shows allowed or blocked, with the exact robots.txt line that decided it.
-
Decide your policy
Blocking training bots is a legitimate choice; blocking AI search bots silently removes you from AI answers. The table tells you which is which.
Which AI crawlers matter?
They fall into three groups, and the difference matters. Training crawlers (GPTBot, ClaudeBot, CCBot, Google-Extended, Applebot-Extended, meta-externalagent, Bytespider) collect content to train future models. Search-index crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot) build the indexes that let AI assistants cite and link to you. On-demand fetchers (ChatGPT-User, Claude-User, Perplexity-User) retrieve a page live when a user asks about it.
Many sites blanket-blocked "AI bots" in 2023–24 and unknowingly blocked the search group too — which means ChatGPT search and Perplexity can't cite them today. Blocking Googlebot-adjacent tokens is riskier still: Google's AI Overviews are fed by the normal Googlebot, and Google-Extended only controls Gemini training, not Search.
Does blocking AI crawlers protect my content?
robots.txt is a voluntary convention: reputable crawlers honour it, but it is not access control. Blocking training bots reduces your content's use in future model training; it does not remove what models already learned, and it won't stop bad actors who ignore robots.txt.
The practical trade-off in 2026: blocking AI search crawlers costs you visibility in an answer channel your competitors are being cited in. Most brands are better served by allowing search and assistant fetchers, and making a deliberate choice about training bots.
Frequently asked questions
How do I check if ChatGPT can crawl my website?
Enter your domain above. The tool reads your live robots.txt and tests OpenAI's three crawlers separately: GPTBot (model training), OAI-SearchBot (ChatGPT search index), and ChatGPT-User (live page fetches during a chat). Each shows allowed or blocked with the rule that decided it.
Should I block GPTBot?
It depends on what you sell. Blocking GPTBot opts your content out of OpenAI model training, but GPTBot also feeds search grounding — many publishers now allow it precisely to stay visible in AI answers. If you want citations without contributing training data, allow OAI-SearchBot and ChatGPT-User while blocking GPTBot, and accept some visibility trade-off.
What's the difference between Googlebot and Google-Extended?
Googlebot indexes your site for Google Search, including AI Overviews — block it and you disappear from Google entirely. Google-Extended is an opt-out token that only controls whether your content trains Gemini models; blocking it does not affect your Search rankings or AI Overviews.
Is my robots.txt data stored?
No. The file is fetched, evaluated in your browser, and discarded. Nothing about your site is saved.