llms.txt Generator & Validator

llms.txt is a proposed convention — a plain-text, markdown-formatted file at a site's root — meant to give AI systems a concise, curated map of a site's most important content, in a format that's directly readable without needing to parse HTML or infer structure from navigation menus. This tool builds a spec-compliant file from your site's actual sections and links, and validates an existing one's structure.

Where llms.txt came from, and what it's actually trying to fix

The convention was proposed by Jeremy Howard (co-founder of Answer.AI and fast.ai) in 2024, motivated by a specific, practical observation: AI systems that answer questions using live web content — retrieval-augmented chat systems, AI search engines, coding assistants that browse documentation — generally have to work with an entire site's raw HTML, inferring what's actually important from navigation structure, headings, and page hierarchy the same way a search engine crawler does. That's a lossy, imprecise process, and it's particularly wasteful for AI systems working within a limited context window, where fetching and parsing an entire page's HTML — navigation chrome, footer links, cookie banners, and all — to extract a handful of genuinely relevant facts is expensive and often unreliable. llms.txt proposes a direct fix: a short, deliberately curated, plain-markdown index sitting at a predictable, well-known location (/llms.txt), giving any AI system that chooses to look for it a fast, low-noise map of what actually matters on the site.

The file's actual structure, and why each part exists

The convention specifies a fairly minimal shape. A single # heading names the site or project. A blockquote (>) immediately after gives a one-to-two-sentence summary — the single most important thing an AI system reading only that far should understand. An optional paragraph of additional context can follow, for anything a reader needs to know before diving into the linked sections. Then one or more ## sections organize links into logical groups, each link written as a standard markdown link followed by a colon and a short description: - [Getting Started](https://example.com/docs/start): Setup and first steps. An optional final ## Optional section is reserved specifically for lower-priority links — a convention that lets an AI system with a very tight context budget skip that section entirely and still get the most important content from everything above it. Every part of this structure exists to serve the same goal: letting a system with limited time or context budget get maximum value from reading as little of the file as possible, front-loading the highest-value information first.

Curated, not exhaustive — the most important design principle

The temptation, especially for a larger site, is to treat llms.txt as "the sitemap, but markdown" and simply list every page. This defeats the entire purpose. A sitemap.xml is deliberately exhaustive — every indexable URL, with no editorial judgment, because its job is ensuring nothing gets missed during crawling. llms.txt is deliberately the opposite: a short, hand-curated selection of the pages and sections that actually matter most, written with enough editorial judgment to be genuinely useful to something reading it under time or context pressure. A 500-page site with an llms.txt listing all 500 pages has produced something functionally identical to a sitemap and has thrown away the one property that made llms.txt worth creating in the first place — its selectivity.

Being honest about how much this actually helps right now

It would be misleading to present llms.txt as a proven, guaranteed lever for AI visibility — no major AI company has published a clear, verifiable commitment to prioritizing or even consistently checking for it, and adoption across the AI ecosystem remains uneven and largely unconfirmed at the level of any specific system's actual behavior. The honest, grounded case for maintaining one is about asymmetric cost and risk rather than a guaranteed payoff: the file costs very little to create and keep updated, causes no downside whatsoever if a given AI system happens to ignore it, and stands a real, non-zero chance of being genuinely useful as more systems adopt the convention over time — which is exactly the calculation that led this site to publish and maintain its own llms.txt as a working example, not because its effect has been definitively measured, but because the downside risk of maintaining one is close to zero.

Keeping it accurate as a site changes

An llms.txt file that quietly falls out of sync with what's actually on the site — describing a section that's been restructured, linking to a page that's moved or been removed — is arguably worse than not having one at all, since it actively misleads whatever reads it rather than simply providing nothing. Treating llms.txt as a living file that gets updated alongside major content or navigation changes, rather than a one-time artifact created once and forgotten, is what keeps it trustworthy for the systems it's meant to help.

How this differs from robots.txt and sitemap.xml, side by side

robots.txt controls crawling permission — what a crawler is and isn't allowed to fetch — and says nothing about content or priority. sitemap.xml is an exhaustive, uncurated list of every indexable URL, meant to ensure complete crawl coverage. llms.txt is neither a permission mechanism nor an exhaustive index — it's a curated, editorially-selected content map, written in a format optimized for a language model or AI retrieval system to read directly and efficiently. The three files serve genuinely different, complementary purposes, and a site that maintains all three is covering crawling permission, complete discoverability, and AI-optimized content prioritization as three distinct, non-overlapping concerns.