Robots.txt lets a site apply different crawling rules to different crawlers, and increasingly, sites are using that capability to draw a deliberate line between search-indexing crawlers and AI-training or AI-retrieval crawlers. This tool fetches a domain's live robots.txt and checks it against the current major named AI crawlers, showing exactly which ones are allowed and which are blocked for a given path.
Why AI crawlers deserve a different policy conversation than search crawlers
For most of the web's history, "should I block this crawler" was a fairly simple question with a fairly simple default answer: allow search engine crawlers, since search visibility is almost universally something a site wants. AI crawlers complicate that default, because they're not all doing the same job a search crawler does. Some — like OpenAI's GPTBot, Anthropic's ClaudeBot and anthropic-ai, Google's Google-Extended, ByteDance's Bytespider — exist specifically to gather content as training data for a large language model, a use case that has nothing to do with sending traffic back to the site and raises real, legitimate questions about content licensing and compensation that search indexing never did. Others — like OpenAI's ChatGPT-User, Perplexity's Perplexity-User, or Perplexity's own indexing crawler PerplexityBot — exist to retrieve and cite specific pages in response to a live user's query, functioning more like a modern analogue to search crawling, potentially driving genuine referral interest even without a traditional clickable search result. These are meaningfully different tradeoffs, and treating every AI crawler identically — either blocking all of them reflexively, or allowing all of them without ever having actually decided to — means a site's owner isn't actually making the choice they might want to make if they understood the distinction.
The crawlers this tool checks, and what each one is actually for
This tool checks access for the current major named AI crawlers, each labeled with what it's actually used for: OpenAI's GPTBot (model training), ChatGPT-User (fetching a page a live user references), and OAI-SearchBot (powering ChatGPT's search results); Google's Google-Extended (a deliberately separate crawler controlling Gemini and AI Overview training, independent of Googlebot's own search indexing); Anthropic's ClaudeBot, anthropic-ai, and Claude-Web; Perplexity's PerplexityBot and Perplexity-User; CCBot, which powers the open Common Crawl dataset a large share of the AI industry trains on regardless of which specific company's crawler visits directly; ByteDance's Bytespider; Apple's Applebot-Extended (again, deliberately separate from the Applebot used for Siri and Spotlight search); Amazon's Amazonbot; Meta's Meta-ExternalAgent; and a few others including Diffbot and Cohere's crawler. Each entry in this tool's results shows not just whether it's allowed or blocked, but who operates it and what it's specifically used for — since "blocked" or "allowed" only becomes a meaningful decision once you know what you're actually opting into or out of.
Reading the result correctly: precedence matters
Robots.txt access for a given crawler and path is determined by finding the most specific matching rule across every group that applies to that crawler — its own named group if one exists, or the general wildcard * group if it has no dedicated entry of its own — using the same longest-match-wins precedence logic real crawlers implement (also available as its own dedicated tool: the Robots.txt Generator & Tester). A crawler showing as blocked doesn't necessarily mean an explicit rule was written specifically naming it — it commonly means a blanket Disallow: / under the wildcard * group is catching it by default, because no more specific, more permissive rule exists for that particular crawler's own name. Understanding which rule actually produced a given result — this tool shows the matched rule directly — is what turns a blocked/allowed result from a mystery into something you can deliberately adjust.
There's no universally correct policy here
This is worth stating plainly: there is no single right answer to which AI crawlers a site should allow, and reasonable, well-informed site owners land in different places. Some deliberately allow every named AI crawler, on the theory that AI-driven discovery and citation is an emerging traffic and visibility channel worth being present in early. Others deliberately block AI-training crawlers specifically (GPTBot, Google-Extended, ClaudeBot, Bytespider, CCBot) while still allowing on-demand retrieval crawlers (ChatGPT-User, Perplexity-User), on the theory that training-data use provides no direct benefit back to the site while on-demand citation at least carries some chance of driving a genuinely interested visitor. Others block everything AI-related on principle, prioritizing content control over any potential AI-driven visibility. This tool is built to make the current state of a site's policy visible and auditable — it deliberately doesn't recommend one policy over another, since that's a judgment call specific to each site's actual goals and values around content licensing and AI.
Auditing regularly, not just once
A robots.txt file is easy to set up once and then forget about, but the list of active AI crawlers worth having an opinion on keeps growing as new companies launch new products, and a policy decided a year ago may no longer reflect what a site's owner would choose today, or may simply be missing rules for crawlers that didn't exist when the file was last touched. Revisiting this check periodically — alongside other routine technical SEO maintenance like verifying sitemap accuracy or checking for broken internal links — is a low-effort way to make sure a site's AI-crawler policy stays a deliberate, current choice rather than an accidental artifact of whatever the file happened to say when it was last edited.
How this complements the Robots.txt Generator & Tester
This tool answers "what's the current state of this domain's policy toward AI crawlers, right now, live?" — a read-only audit against a real, fetched file. The companion Robots.txt Generator & Tester tool on this site answers the follow-up question once you've decided a change is needed: building the actual rule additions or edits, and testing them against any crawler and path before they go live. Used together, they cover the full loop — audit the current state here, then make and verify the actual change there.
If a domain has no robots.txt file at all, every crawler — including every AI crawler this tool checks — is allowed access to every path by default, since there's no file present instructing otherwise. If a site's owner wants a deliberate, specific policy toward any of these crawlers, an actual robots.txt file with explicit rules has to exist; silence isn't neutral, it's full access.