Robots.txt Tester & Validator

Check Whether Your robots.txt Actually Blocks What You Think It Does

Writing a robots.txt is easy; getting the rule precedence right is not. Between overlapping Allow/Disallow lines, wildcard patterns, and per-crawler user-agent groups, it's common to block a page you meant to allow (or the reverse) without noticing until Search Console flags it weeks later. Paste your file here, pick a crawler, and test as many URL paths as you want against the exact same longest-match-wins logic that Google's own robots.txt parser uses.

How to use it

• Paste your robots.txt content into the box, or click "Load an example" to see how the tool behaves first.
• Pick the crawler user-agent you want to test against — Googlebot, Bingbot, Googlebot-Image, or a custom bot name. The tool matches it against the most specific applicable group, falling back to the wildcard User-agent: * group.
• List the URL paths you want to check, one per line — a homepage, a blocked admin section, a specific post, whatever you need to verify.
• Each result shows Allowed or Blocked, plus exactly which rule and line number decided it.

Real-world use cases

Confirming that a new Disallow: /checkout rule doesn't accidentally also block /checkout-guide, a blog post you actually want indexed. Debugging why Googlebot-Image isn't picking up product photos, by testing the image-specific user-agent group separately from the general one. Reviewing a robots.txt before deploying it to production so a typo doesn't silently deindex the whole site.

Frequently asked questions

What's the difference between a robots.txt generator and a robots.txt tester?

A generator builds a robots.txt file from scratch by picking rules. A tester does the opposite: it takes a robots.txt you already have (or are drafting) and tells you exactly how a real crawler would interpret it against specific URLs — which is essential once a file has more than a couple of lines.

Which rule wins when Allow and Disallow overlap?

The longest matching pattern wins, regardless of whether it's Allow or Disallow. If two rules match with exactly the same length, Allow takes priority. That's the same precedence logic this tool applies.

Does robots.txt support wildcards?

Yes — * matches any sequence of characters within a path, and $ anchors a rule to the end of the URL. For example, Disallow: /*.pdf$ blocks any URL ending in .pdf, wherever it lives on the site.

Why isn't my Disallow rule blocking a page?

The most common cause is a more specific Allow rule elsewhere in the same group overriding it, or the rule sitting under the wrong user-agent group entirely. Run the exact path through this tool with the crawler you care about to see which rule is actually deciding the outcome.

Does robots.txt block a page from ever appearing in search results?

Not by itself. Disallowing a URL stops crawlers from fetching its content, but the URL can still be indexed (with no snippet) if other pages link to it. To fully keep a page out of search results, use a noindex meta tag or header instead, and don't block it in robots.txt so the crawler can actually see that tag.