Technical SEO
Robots.txt tester & validator
Paste a robots.txt (or fetch any site's live file), choose a crawler, and test any URL path: the tool tells you allowed or blocked, shows the exact rule and line number that decided it, and flags syntax problems — using the same longest-match logic Google documents for its own crawlers.
Free · no signup Checks your live site
How to use the robots.txt tester & validator
-
Load your robots.txt
Paste the file, or type a domain and fetch the live version — useful for checking what's actually deployed.
-
Choose a crawler and a path
Test as Googlebot, Bingbot, GPTBot, or any custom user-agent, against any path including query strings.
-
Read the deciding rule
The verdict names the Allow/Disallow line that won, so you can fix the actual rule rather than guessing.
How robots.txt matching actually works
Three rules explain almost every surprise. First, a crawler uses only one group: the most specific User-agent match (Googlebot-Image beats Googlebot beats *) — groups don't combine. Second, within that group the longest matching path wins, regardless of order in the file. Third, on a tie between Allow and Disallow of equal length, Allow wins.
Wildcards do the rest: * matches any characters, $ anchors the end of the URL. So "Disallow: /*?sort=" blocks any URL containing ?sort=, and "Disallow: /*.pdf$" blocks PDFs specifically. This tester implements exactly these semantics and shows its working.
Common robots.txt mistakes this catches
The classics: rules placed before any User-agent line (silently ignored), paths that don't start with / (ignored by most crawlers), "noindex" directives in robots.txt (unsupported by Google since 2019 — use a meta robots tag), and Crawl-delay (ignored by Googlebot). The validator flags each with the line number.
The most expensive mistake is subtler: a Disallow that's broader than intended. "Disallow: /app" also blocks /apple-watch-review — test your important URLs, not just the ones you meant to block.
Frequently asked questions
Does blocking a URL in robots.txt remove it from Google?
No — it can do the opposite. robots.txt stops crawling, not indexing: a blocked URL that other sites link to can still appear in results, just with no snippet. To remove a page from the index, allow crawling and add a noindex meta tag or X-Robots-Tag header.
What does "Disallow:" with an empty value mean?
It means allow everything for that user-agent group. An empty Disallow matches nothing, so nothing is blocked — it's the conventional way to write an explicit "this bot is welcome" group.
Do Allow rules work in robots.txt?
Yes — all major crawlers support Allow. It's how you open a subfolder inside a blocked area: "Disallow: /admin/" plus "Allow: /admin/public/" blocks the admin area except the public part, because the longer Allow rule wins for those URLs.
Why does Google's tester disagree with my expectation?
Usually one of three reasons: your crawler matched a more specific group than you thought, a longer rule elsewhere in the group won, or your path pattern needs a wildcard (patterns are prefix matches, not substring matches). This tool names the winning rule so you can see which happened.