XML Sitemap Best Practices for Faster, Cleaner Indexing

A sitemap is a hint, not a guarantee

Search engines treat a sitemap as a strong discovery signal, not a command. It won't force indexing of low-quality pages — what it does is remove any doubt about which URLs you consider canonical and worth crawling, which matters most on large or newer sites.

What to include

• Only canonical URLs — the exact version you want indexed.
• Only URLs that return a 200 status and are actually indexable (no noindex tag, not blocked by robots.txt).
• An accurate lastmod date — only update it when the page's content meaningfully changes, not on every deploy.

What to exclude

• Pages with a noindex tag — including one in a sitemap sends a contradictory signal.
• Redirecting or 404'ing URLs — these get reported as sitemap errors instead of helping discovery.
• Duplicate or parameterized variants of the same page (sort orders, session IDs, tracking parameters) — list the one canonical version only.
• Thin, auto-generated pages with little unique content — these are a common cause of "Crawled - currently not indexed" and including hundreds of them dilutes the sitemap's signal for pages that matter.

Size limits

A single sitemap file is capped at 50,000 URLs and 50MB uncompressed. Larger sites need a sitemap index file — one file that lists multiple child sitemaps — rather than one giant file. Most frameworks (including this site's) generate this automatically, but it's worth confirming if you're hand-rolling one.

Submitting and maintaining it

Submit it once in both Google Search Console and Bing Webmaster Tools, and add a Sitemap: line to your robots.txt pointing at it — that way any crawler that finds your site independently, without a prior submission, can locate it too. After that, the sitemap should regenerate automatically as content changes; a hand-maintained sitemap almost always goes stale.

Common mistakes that quietly waste crawl budget

• Generating it once at launch and never updating it as pages are added or removed.
• Setting every URL's lastmod to the current date on every build — this trains crawlers to stop trusting the field as a real change signal.
• Forgetting to resubmit after a major URL restructure (e.g. a domain migration or permalink change) — Google will eventually notice on its own, but resubmitting speeds up the transition.