Generating a sitemap and robots.txt

Part 9 and 10 covered how one page gets found and understood by a search engine. This part covers how every page gets discovered in the first place — a sitemap listing every URL, and a robots.txt telling crawlers what they're allowed to visit.

app/sitemap.ts

// app/sitemap.ts
import type { MetadataRoute } from "next";
import { getAllPosts } from "@/lib/posts";

export default function sitemap(): MetadataRoute.Sitemap {
  const posts = getAllPosts().map((post) => ({
    url: `https://devnotes.example.com/posts/${post.slug}`,
    lastModified: post.date,
    changeFrequency: "monthly" as const,
    priority: 0.7,
  }));

  return [
    {
      url: "https://devnotes.example.com",
      changeFrequency: "weekly",
      priority: 1,
    },
    {
      url: "https://devnotes.example.com/posts",
      changeFrequency: "weekly",
      priority: 0.9,
    },
    ...posts,
  ];
}

sitemap.ts is a reserved filename, the same reserved-convention pattern as page.tsx and route.ts — Next.js automatically serves whatever this default export returns at /sitemap.xml, generating the actual XML for you. Building the post entries from getAllPosts() — the exact same function every page in this series already calls — means the sitemap can never drift out of sync with what posts actually exist; adding a post to the data source adds it here automatically, with no separate list to maintain.

app/robots.ts

// app/robots.ts
import type { MetadataRoute } from "next";

export default function robots(): MetadataRoute.Robots {
  return {
    rules: {
      userAgent: "*",
      allow: "/",
      disallow: "/api/",
    },
    sitemap: "https://devnotes.example.com/sitemap.xml",
  };
}

Same reserved-filename pattern, serving /robots.txt. disallow: "/api/" keeps crawlers out of the Route Handlers from part 13 — JSON endpoints have no business showing up in search results — while allow: "/" permits everything else, including every post page. The sitemap field here is what points a crawler at the exact file generated above, so it doesn't have to guess or rely on it being submitted manually.

Why lastModified matters

lastModified: post.date on each sitemap entry is a real signal, not a formality — it's what tells a crawler whether it's worth re-fetching a specific page versus one it already has an up-to-date copy of. A blog that never updates this field (or hardcodes today's date on every post regardless of when it actually changed) gives crawlers no useful information to prioritize re-crawling with.

Splitting a very large sitemap

A single sitemap file has a practical size limit (50,000 URLs, per the sitemap protocol) — a site that grows well past what one post array can reasonably represent eventually needs a sitemap index instead: multiple smaller sitemaps, one per content type or date range, referenced from a top-level index file. That's a scale problem worth solving when it's actually reached, not before; app/sitemap.ts returning one array, as built here, covers every realistic blog for a long time before that split becomes necessary.

Last part: getting this from npm run dev on a laptop to a real, deployed site — plus the tests worth having before it ships.