When a site publishes dozens of pages per week, search engines do not find them all by themselves. The sitemap is the file that provides them with the complete list of URLs to explore. For anyone monitoring the evolution of the web, understanding this mechanism changes the way information is collected.
The sitemap as a freshness signal for monitoring
Most guides present the sitemap from an SEO perspective. Its usefulness for monitoring is less documented, even though it is straightforward.
Related reading : Everything You Need to Know About the Women's Ranking and Biathlon Points System for the Season
An XML sitemap contains, for each URL, a lastmod tag that indicates the date of the last significant modification. By monitoring this tag on the sites you follow, you can identify updated pages without having to visit them one by one.
Google has also clarified that lastmod should reflect a real change in the main content, structured data, or links, and not just a simple formatting adjustment. This tightening transforms the tag into a reliable indicator: when it changes, the content has actually changed.
Related reading : Everything You Need to Know About the SNCF Carmillon Pass for Retirees: Benefits and Conditions
By regularly checking the sitemap of Our Web Monitoring, you can visualize, for example, all the recently published or updated pages on the site, sorted by date.
XML sitemap file: structure and technical limits
A sitemap is a text file in XML format. It begins with a character set declaration, then lists blocks each containing a URL and its metadata. Here’s what you can find in it:
- The loc tag, which contains the full address of the page (for example https://monsite.fr/article-veille).
- The lastmod tag, which specifies the date of the last significant change in ISO format (year-month-day).
- The changefreq tag, which suggests a frequency of updates (daily, weekly, monthly). However, Google indicates that it largely ignores this.
- The priority tag, intended to prioritize pages among themselves. In practice, search engines no longer take it into account for their crawl decisions.
The file is generally accessible at /sitemap.xml or /sitemap_index.xml. A single sitemap cannot exceed 50,000 URLs. Beyond that, an index of sitemaps must be created that points to several secondary files.
On WordPress, plugins like Yoast or Rank Math automatically generate these files. They segment URLs by content type (articles, pages, categories) and exclude pages marked noindex.
End of sitemap ping: what changes for monitoring
Until recently, it was possible to “ping” Google to notify it of an update to the sitemap. Google removed support for ping endpoints at the end of 2023, explaining that the mechanism was too exploited by spam.
This removal has a direct consequence for monitoring. It means that simply notifying a change is no longer sufficient. Google now relies on the actual content of the sitemap file and the consistency between the lastmod date, the date visible on the page, and the dateModified field in the structured data.
For a monitor, this development is good news. A sitemap with reliable dates becomes a more precise detection tool than a poorly maintained RSS feed. When the three dates (lastmod, displayed date, schema) match, one can consider that the update is authentic.
An outdated sitemap skews the perception of the web
A documented case describes an e-commerce platform whose server logs showed that Googlebot continued to crawl deleted URLs because the sitemap had not been regenerated after a migration. An unmaintained sitemap sends false signals, both to search engines and automated monitoring tools.
If you are using third-party sitemaps for your monitoring, check that the lastmod dates are actually changing. A file where all dates are identical across hundreds of pages is likely generated without reflecting real changes.
Sitemap and robots.txt: two complementary files for SEO
The sitemap lists the pages to explore. The robots.txt file indicates the pages not to explore. Confusing their roles is a common mistake.
The robots.txt blocks crawling, not indexing. A page blocked in robots.txt can still appear in search results if other sites link to it. The sitemap, on the other hand, facilitates the discovery of pages but does not guarantee their indexing or ranking.
For both files to work together, the best practice is to declare the location of the sitemap directly in the robots.txt, with a line like Sitemap: https://monsite.fr/sitemap.xml. This declaration allows search engines to locate the file without going through the Search Console.
- Only list pages in the sitemap that have a status of 200 (accessible) and are indexable.
- Exclude pages with a noindex tag or those redirected with a 301.
- Align lastmod, dateModified, and the visible date on the page to strengthen the credibility of the signal.
A well-structured sitemap does not transform a site’s SEO on its own. It reduces the time between a publication and its consideration by search engines. For monitoring, this shortcut between going live and visibility is precisely what matters: spotting new content before it gets lost in the mass.