Robots.txt Generator & Tester

Robots.txt is the first file most crawlers check before fetching anything else from a site — a plain-text set of rules, sitting at a fixed, well-known location, telling each named crawler which parts of the site it may and may not fetch. This tool both builds a robots.txt from scratch and tests an existing one against a specific crawler and path, using the same precedence rules real crawlers follow.

What robots.txt actually controls, and what it doesn't

The single most consequential misunderstanding about robots.txt is treating it as an indexing control, when it's actually a fetching control — a subtle but important distinction. Disallowing a path tells a compliant crawler "don't request this URL's content," which is genuinely effective at keeping a crawler from spending time and bandwidth on parts of a site that don't need crawling (admin areas, internal search results, duplicate parameter-driven URL variants). What it does not reliably do is keep a URL out of a search index — if Google discovers a disallowed URL through an external link, it can still list that bare URL in search results, just without a snippet, since it was explicitly told not to read the page's content to generate one. Reliable index control requires a noindex directive (a meta tag or HTTP header), which has to actually be fetched and read to take effect — meaning a page can't simultaneously be disallowed in robots.txt and reliably noindexed, since a blocked crawler never sees the noindex instruction living inside the page it was told not to fetch.

How the matching algorithm actually decides a verdict

Robots.txt syntax looks simple, but the rule for resolving conflicting or overlapping directives surprises people used to "first match wins" logic from other config formats. Per the specification most major crawlers (including Googlebot) follow, the most specific matching rule wins — specificity measured by the length of the path pattern itself, in characters — regardless of which order the rules physically appear in the file. When an Allow and a Disallow rule both match a given path with exactly equal length, Allow wins. Wildcards (*, matching any sequence of characters) and the end-of-string anchor ($) both participate in pattern matching and therefore in this specificity comparison. This tool's tester implements that exact algorithm — parsing the file into per-user-agent groups, then finding the single longest matching rule for a given path — so you can check the actual effective outcome for any combination of user-agent and path rather than working through overlapping rules by hand, which is exactly the kind of thing that's easy to get subtly wrong under time pressure.

Why per-crawler rules matter more than they used to

For most of robots.txt's history, the practical concern was almost entirely about search-engine crawlers, and a single User-agent: * block covering every crawler identically was usually sufficient. That's changed: the rise of AI systems that crawl the web for model training data, retrieval, and on-demand page fetching has introduced a whole new category of crawler with a meaningfully different purpose than search indexing — and a site owner may reasonably want a different policy for each. Allowing Googlebot to index your content for search while disallowing GPTBot from using it to train a model is a completely coherent, increasingly common position, and robots.txt supports it directly: separate User-agent: groups let each named crawler get its own distinct rule set, rather than forcing an all-or-nothing decision that applies identically to every bot that visits.

The AI crawler landscape, briefly

Major AI companies have published named, identifiable crawlers specifically so sites can make this distinction: OpenAI's GPTBot and OAI-SearchBot, Anthropic's ClaudeBot, Google's Google-Extended (which is deliberately separate from Googlebot itself, controlling AI training use without affecting search indexing), Perplexity's PerplexityBot, and several others, each documented with a specific, named user-agent string a robots.txt file can target individually. Whether to allow or block each one is a genuine policy decision with real tradeoffs on both sides — allowing them can mean visibility in AI-generated answers and citations; blocking them withholds content from model training and AI retrieval — and there's no universally correct answer, only a decision that should be made deliberately rather than left as an accidental default. This site's AI Crawler Access Checker tool is built specifically to audit an existing robots.txt against the current, real list of named AI crawlers.

A robots.txt file is a request, not a lock

It's worth being direct about this: robots.txt has zero technical enforcement mechanism behind it. It works purely because major, reputable crawlers — search engines and legitimate AI companies alike — have chosen, as a matter of established web convention and their own public commitments, to respect it. Nothing stops a crawler that doesn't care about that convention from ignoring the file entirely and fetching whatever it wants; robots.txt simply doesn't have a way to prevent that at a technical level. For content that genuinely must never be accessed by anyone or anything without authorization — not "shouldn't be indexed," but "must never be fetched" — the correct tool is real access control (authentication, IP restriction, a paywall), not a polite text-file request that only well-behaved crawlers choose to honor.

Common mistakes worth checking for

Blocking an entire site accidentally with a stray Disallow: / left in from a staging environment that got copied to production unmodified — this happens more often than it should, and is worth double-checking after any deployment pipeline change. Disallowing a path while also expecting a noindex tag on that same path to take effect, not realizing the two conflict as described above. Forgetting the Sitemap directive, which — while not strictly required — is a low-cost, widely-supported way to point every crawler directly at your sitemap without them needing to guess or discover it some other way.