Sitemap strategy for a large content site: split, index, lastmod

By the end of this you will have a sitemap that scales past 50,000 URLs, tells search engines which pages actually changed, and gets new posts crawled in hours instead of weeks. This is the setup we run on content sites that publish most days and expect to cross a thousand posts.

A sitemap is a file that lists the URLs on a site and, optionally, when each one last changed. On a small site it barely matters. Google will find fifty well-linked pages without help. On a large content site the sitemap stops being a nicety and becomes the map crawlers use to decide what to fetch and how often. Get it wrong and fresh posts sit undiscovered for weeks while the crawler re-reads pages that have not moved.

What you need before starting

• A list of every canonical, indexable URL on the site, generated from the database rather than hand-maintained.
• A reliable last-modified timestamp per URL that reflects real content change, not every row write.
• Access to Google Search Console and Bing Webmaster Tools to submit and monitor.
• A framework that renders XML at request or build time. The examples here use Next.js, but the rules apply anywhere.

Step 1: Know the two hard limits

Google caps a single sitemap file at 50,000 URLs or 50MB uncompressed, whichever comes first (Google Search Central). You can gzip the file to save bandwidth, but the 50MB ceiling is measured uncompressed. Cross either limit and Google rejects the file.

A sitemap index solves this. It is a file that lists other sitemaps instead of URLs, and it can reference up to 50,000 of them. That is a theoretical ceiling of 2.5 billion URLs across one index. No blog will reach it. The point is narrower: once you pass 50,000 URLs, you stop shipping one file and start shipping an index that points at shards.

Step 2: Shard by content type, not by arbitrary number

The naive split is by count: 50,000 URLs per file, chunked in publish order. It works, but it throws away a debugging signal. Shard by content type instead. Blog posts in one sitemap, category pages in another, static marketing pages in a third. Each shard stays under the limit, and Google Search Console reports coverage per sitemap. When indexing drops, you see which content type is affected instead of staring at one number for the whole site. Segmenting by type is the standard practice for exactly this reason (Google Search Central).

If a single type still exceeds 50,000 URLs, chunk that type by number: blog-1.xml, blog-2.xml, and so on, all listed in the index.

Step 3: Include only URLs you want indexed

A sitemap is a list of your best URLs, not every URL that resolves. Include a page only if it returns 200, is not marked noindex, is not blocked by robots.txt, and is its own canonical. Never list redirects, 404s, parameter variants, or pages that canonicalize elsewhere. Mixing non-indexable URLs into a sitemap sends a contradictory signal: you ask Google to index a page you have told it to ignore. On a large site those contradictions add up and erode trust in the whole file.

For a multi-locale site, declare each language variant with hreflang alternates inside the sitemap entry rather than shipping the same URL three times. That keeps the locale relationships explicit and the URL count honest.

Step 4: Make lastmod honest, and drop the rest

Three optional tags exist: lastmod, changefreq, and priority. Only one earns its place. Google ignores priority and changefreq outright (Google Search Central), and they add bytes to a file you are trying to keep small. Cut them.

lastmod is different. It is the one sitemap attribute modern search engines actively use, and Google reads it only when it is consistently accurate. John Mueller has been blunt: faking the date does not help rankings, and once Google decides your lastmod is unreliable it stops trusting the tag at all (Search Engine Journal). Most sites keep it so badly that Google discounts it by default.

The rule is simple. Set lastmod to the timestamp of a real content change, and nothing else. Do not bump it on every database write, every metadata migration, or every deploy. If your CMS touches a row for reasons the reader never sees, that must not move lastmod. We anchor ours on the same signal we show readers as an Updated date, which keeps the machine-facing and human-facing freshness in sync. More on that discipline in our piece on content freshness and the 90-day rule.

Step 5: Notify on change, because ping is dead

Submitting a sitemap once tells search engines where it lives. It does not tell them the moment a new post goes live. The old nudge, the sitemap ping endpoint, is gone. Google removed its ping API in 2023 and Bing deprecated its own in 2022 (Ctrl blog).

The replacement is IndexNow, a protocol that pings participating engines the instant a URL is added, changed, or removed. Bing, Yandex, Naver, and Seznam use it. Google tested it and, as of 2026, still does not. Bing reports that 22% of clicked URLs in its results now come from IndexNow submissions. The official guidance is not either-or: keep the sitemap as the full inventory and add IndexNow for instant change notifications (IndexNow.org). For Google, a fresh and accurate sitemap plus solid internal linking remains the discovery path.

Step 6: Generate it dynamically in Next.js

Hand-maintained XML rots the day someone forgets to update it. Generate the sitemap from the same data that renders the pages. In the Next.js App Router, generateSitemaps returns an array of ids, and Next serves each shard at /sitemap/[id].xml (Next.js docs). Count total posts, divide by a chunk size under 50,000, and map each id to a database range. Cache the query for 24 hours so a daily-publishing site does not rebuild the whole map on every crawler hit.

Verifying it works

1. Submit the index URL in Google Search Console and Bing Webmaster Tools, not the child shards. Google discovers children through the index.
2. Open the Sitemaps report a few days later. Each shard should show Success with a discovered-URL count close to what you expect.
3. Spot-check a child sitemap in the browser. It should return valid XML, list only 200-OK canonical URLs, and carry lastmod dates that match real edits.
4. Run the URL Inspection tool on a brand-new post. If Google found it through the sitemap, the report says so.

Common failures and fixes

• Child sitemaps in the wrong directory. A sitemap listed in an index must sit in the same directory or lower (Google Search Central). An index at the root cannot reference a child in a sibling folder. Keep them at or below the index path.
• lastmod that never stops moving. If every deploy touches every timestamp, Google learns the tag is noise and ignores it. Anchor lastmod on content change only.
• Submitting each shard by hand. Adding every child sitemap separately clutters the report and gains nothing. Submit the index alone.
• noindex pages inside the sitemap. The most common contradiction. Filter the list at generation time so a page cannot be both listed and suppressed.
• Treating the sitemap as a crawl command. It is a hint, not an order. It helps discovery; it does not force indexing. Thin or duplicate pages will not rank just because they sit in a sitemap.

Going further

A sitemap is one piece of the discovery layer. Pair it with the freshness discipline in our 90-day freshness guide, the crawler-access rules in Cloudflare and AI crawlers, and the rest of the pre-launch floor in our technical SEO checklist. If AI engines are part of your traffic plan, llms.txt covers the file that does for language models what the sitemap does for crawlers.

Sources

• Google Search Central: Manage your sitemaps with sitemap index files
• Google Search Central: Build and submit a sitemap
• Search Engine Journal: Mueller on updating sitemap dates
• Ctrl blog: Bing stops accepting sitemap pings, switch to IndexNow
• Bing Webmaster Tools: Why IndexNow
• IndexNow.org
• Next.js docs: generateSitemaps
• Bing Webmaster Blog: keeping content discoverable with sitemaps in AI-powered search