What this tool checks
The checker downloads /robots.txtfrom the URL's origin, parses it into user-agent groups, and evaluates the path you pasted against each of the five AI crawlers separately. It applies the same matching rules Google documents: the group whose user-agent token best matches the bot wins (longest token, with * as the fallback), and within that group the most specific pattern applies — longest Allow/Disallow pattern wins, and Allow beats Disallow on a tie. Wildcards (*) and end-anchors ($) are supported. Each result also tells you whether the verdict came from a rule that names the bot or from the catch-all group.
Why AI crawler access matters
Answer engines are becoming a real referral channel, and unlike classic search they do not need to send a click to have already decided whether you exist. If ChatGPT cannot fetch your pricing page, it cannot recommend your product when someone asks for options. If Perplexity cannot read your comparison article, it will cite a competitor's instead. The gate for all of this is one public text file that almost nobody re-reads after launch.
The failure mode is rarely a considered decision. It is a Disallow: /left over from staging, a CMS "discourage search engines" toggle, a security plugin that added a blanket AI-bot block, or a CDN bot-management rule that someone enabled to stop scrapers and never revisited. Each of those looks identical from the outside: your content simply stops appearing in AI answers, with no error and no notification.
How to fix what you find
To allow an engine, remove the Disallow from its group, or add an explicit group that permits it — a named group always beats the catch-all, so User-agent: ClaudeBot followed by Allow: / re-opens access even while User-agent: * stays restrictive. To block training but keep citations, block GPTBot and Google-Extended while leaving OAI-SearchBot allowed. Check a real content URL, not just the homepage: rules are path-based, so /blog/post and /can have opposite verdicts. And remember robots.txt is only half of it — a page that is crawlable but returns thin server-rendered HTML still gives an AI engine nothing to quote, which is what a full audit's AI-readiness checks measure.