Technical SEO: crawlability, robots.txt, and XML sitemaps
Part 1 covered crawling as the first stage of the whole pipeline — nothing else in this series matters for a page a crawler can't reach or was explicitly told not to enter. This part covers the two files that most directly control that: robots.txt and the XML sitemap.
robots.txt: a real, minimal example
User-agent: *
Allow: /
Sitemap: https://www.example.com/sitemap.xmlThis is a real, live robots.txt — deliberately simple: allow every crawler, everywhere, and point directly at the sitemap. Allow: / isn't strictly required (crawling is allowed by default absent a Disallow), but stating it explicitly removes any ambiguity for a crawler parsing the file. The Sitemap: line is what lets a crawler discover the sitemap without it having to be manually submitted through every individual search engine's own webmaster tools.
Disallowing what shouldn't be crawled
User-agent: *
Disallow: /admin/
Disallow: /api/
Allow: /Disallow is for paths with no reason to appear in search results at all — an admin panel, a JSON API endpoint (the Next.js series covers exactly this pairing for a Route Handler). This is a request, not an enforced technical lock — a well-behaved crawler like Googlebot respects it; it does nothing to actually prevent access by a browser or a script that ignores it, so it's not a security mechanism.
A single misplaced Disallow: / — no path after the slash — blocking the entire site from every crawler at once. This is one of the most common, and most damaging, real technical SEO mistakes: an entire site silently deindexing itself because of one overly broad rule, often left over from a staging environment's robots.txt that shipped to production by accident.
A real sitemap, and a real sitemap-index architecture
<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
<url>
<loc>https://www.example.com/tutorials/python-requests-library</loc>
<lastmod>2026-08-13</lastmod>
<changefreq>monthly</changefreq>
<priority>0.7</priority>
</url>
</urlset>A plain sitemap is exactly this shape — one <url> entry per page, each with a <lastmod> (part 1's ranking discussion touches on why this matters: it tells a crawler whether a page is worth re-fetching). A single sitemap file has a practical limit of 50,000 URLs; a larger site outgrows one file and needs a sitemap index instead:
<?xml version="1.0" encoding="UTF-8"?>
<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
<sitemap><loc>https://www.example.com/sitemap-pages.xml</loc></sitemap>
<sitemap><loc>https://www.example.com/sitemap-tutorials.xml</loc></sitemap>
<sitemap><loc>https://www.example.com/sitemap-blog.xml</loc></sitemap>
<sitemap><loc>https://www.example.com/sitemap-news.xml</loc></sitemap>
</sitemapindex>This is a real, live architecture — this site's own /sitemap.xml is exactly this index shape, splitting URLs by content type into separate sub-sitemaps rather than one flat file. Each sub-sitemap is generated dynamically from the site's own content — every tutorial, automatically included the moment it's published, with no manually maintained list anywhere (the Next.js series covers the App Router version of this pattern directly).
Submitting it
A sitemap referenced in robots.txt is discoverable automatically, but submitting it directly through Google Search Console (part 20 covers the full tool) triggers a faster initial crawl and surfaces any parsing errors Google's own sitemap validator finds — worth doing on top of, not instead of, the robots.txt reference.
Maintaining a sitemap by hand — a static file updated occasionally when someone remembers — instead of generating it from the same data source the site itself is built from. A hand-maintained sitemap drifts out of date within weeks; a generated one, built directly from live content, structurally can't.
Next: canonical tags — the fix for when the same content is genuinely reachable at more than one URL.