Skip to main content

Home / Tools / Robots.txt Tester

Free robots.txt tester

Fetch a site's robots.txt, see every group and rule parsed out, and test whether a specific path is allowed or blocked for the crawler you care about.

What this tool checks

The tester downloads /robots.txtfrom the site's origin, parses it into user-agent groups with their Allow and Disallow rules, and lists any Sitemap: declarations. If you supply a path, it evaluates that path the way Google does: it picks the group whose user-agent token best matches the crawler (longest match, with * as the fallback), then applies the most specific rule — longest pattern wins, allow beats disallow on ties. Wildcards (*) and end-anchors ($) are supported.

Why robots.txt matters

robots.txt is the first file every well-behaved crawler requests, and it is one of the few places where a one-character mistake can deindex-in-effect an entire business. A stray Disallow: /stops all crawling; a malformed group header silently detaches the rules beneath it; blocking your CSS and JavaScript directories prevents Google from rendering pages properly, which hurts rankings even though the HTML itself is crawlable. The file has also become the control point for AI: OpenAI's GPTBot, Anthropic's ClaudeBot, PerplexityBot and Google-Extended all read it to decide whether your content can be fetched for training or answering. Whether you allow or block them should be a decision, not an accident.

How to fix common problems

Everything blocked: look for Disallow: / under User-agent: * — usually a leftover staging config. Delete it and resubmit your sitemap in Search Console. Rules not applying: rules only attach to the group above them; a blank User-agent line or a typo like User Agent: orphans everything that follows. Blocked assets: never disallow /wp-includes/, /assets/ or similar folders that hold CSS/JS — Google needs them to render the page. Case sensitivity: paths are case-sensitive; /Admin/ and /admin/ are different rules. Missing sitemap line: adding Sitemap: https://example.com/sitemap.xml is free crawl-discovery — do it even if you have submitted the sitemap manually.

Remember that robots.txt is public — anyone can read it, so never use it to "hide" sensitive URLs. Auth and noindex are for privacy; robots.txt is only for crawl budget.

Frequently asked questions

Does robots.txt remove pages from Google?

No — it only stops crawling, not indexing. A blocked URL can still appear in results (as a bare link with no snippet) if other pages link to it. To remove a page from the index, allow crawling and add a noindex meta tag, or use Search Console removals.

Which rule wins when several match?

Google uses the most specific match: the rule with the longest matching pattern applies, and if an allow and a disallow tie in length, allow wins. This tester implements exactly that logic, including * wildcards and $ end-anchors.

What happens if a site has no robots.txt at all?

Crawlers treat a 404 on /robots.txt as "everything allowed" and carry on. That is completely fine for most small sites. A 5xx error on robots.txt is worse — Google may pause crawling the whole site until the file is reachable again.

Should I block AI crawlers like GPTBot or ClaudeBot?

It is a trade-off. Blocking them keeps your content out of AI training and answer engines; allowing them makes it possible for assistants to cite and recommend your site. If AI referrals matter to your funnel, blocking every AI bot is usually a mistake.

Why is my whole site blocked by "Disallow: /"?

That single line under User-agent: * tells every crawler to skip the entire site. It is commonly shipped by accident when a staging configuration reaches production — CMS "discourage search engines" settings generate it. Remove the line and request recrawling.

Want the full picture?

This tool checks one thing. A full RankVitals audit runs a Lighthouse speed test plus 50+ SEO and AI-readiness checks on your whole site — free, no signup.

Run a full site audit →