100% CPU on WordPress: the sitemap nobody was caching

The learning hub of the site I work on runs WordPress in containers on AWS ECS: two tasks, each with 1 vCPU and 2 GB of memory, behind a CDN. For a while, both tasks sat at 100% CPU — and nothing about the content or the traffic to real pages had changed.

The cause turned out to be the most boring file on the site: sitemap.xml.

What the servers were actually doing

The first useful number was cost per request. A normal page render on these tasks is fast; a request for a sitemap page took about ten seconds of CPU. Every single time.

The second number was volume. The sitemaps were being fetched around 2,200 times a day, and most of those requests weren't search engines — they were scrapers, which tend to arrive in bursts and walk every page of the sitemap index at once.

Put those together and the arithmetic is unforgiving:

2,200 requests/day × ~10 s CPU  ≈ 22,000 CPU-seconds/day
                                ≈ 6 vCPU-hours of pure sitemap generation

On average that's a quarter of a core around the clock. But it doesn't arrive on average — it arrives as a scraper fetching twenty sitemap pages in parallel, and two 1-vCPU tasks simply don't have that to give.

Why was every sitemap built from scratch?

A sitemap is the textbook thing to cache: it changes only when content changes, and it's expensive to build. The SEO plugin we use caches sitemaps out of the box. So the real question wasn't why is it slow — it was why is the cache off?

Git history answered it. Almost two years earlier, a one-line hotfix had switched sitemap caching off:

add_filter('rank_math/sitemap/enable_caching', '__return_false');

The commit had no explanation, but the context was easy to reconstruct: sitemaps had been going stale, and turning the cache off made the staleness go away. It worked — and quietly turned every sitemap request into a full rebuild for the next 21 months.

Why the plugin's own cache couldn't just be turned back on

Re-enabling the cache would have brought the stale sitemaps straight back, so I read the plugin's source to find out why they went stale in the first place. Three things stood out.

1. Invalidation only runs in some contexts. The class that watches for content changes and clears the sitemap cache is only loaded inside wp-admin and cron:

if ( is_admin() || wp_doing_cron() ) {
    new Cache_Watcher();
}

Our posts aren't only published from wp-admin. Some arrive through the REST API and publishing automation, and scripts update content in bulk. None of those paths load the watcher, so none of them ever cleared the cache. That's the staleness.

2. The default cache lives on local disk. In file mode, each container keeps its own copy of the sitemaps. With two tasks, a cache cleared on one is still stale on the other — invalidation would have to reach every container.

3. Clearing the cache can crash. The plugin's reset goes through WordPress's filesystem layer, which can fall back to FTP in some environments and fail outright.

There was one more multiplier outside WordPress: every response, sitemaps included, was sent with Cache-Control: no-cache. So the CDN in front of the site couldn't absorb any of the scraper traffic either. Every request went to origin.

The fix: cache on inputs, not on events

The core idea is simple. Instead of trusting the plugin to notice a change, compute a fingerprint of everything a sitemap depends on, and throw the cache away whenever the fingerprint changes. That way it doesn't matter how content changed — admin, REST, CLI or a script — only whether it did.

The change has four parts.

Store the cache in the database. Switching the plugin to database mode means both tasks read and write the same copy, in the shared database.

add_filter('rank_math/sitemap/cache_mode', function () { return 'db'; });

Fingerprint the inputs. One hash covers everything that can change a sitemap: published posts and pages (count and latest modification time), categories, authors, the sitemap and title settings, both redirect lists — and the file containing this code, so deploying a change to the logic invalidates the cache too.

function site_sitemap_fingerprint() {
  global $wpdb;
  return md5(wp_json_encode([
    md5_file(__FILE__),
    $wpdb->get_row("SELECT COUNT(*) n, MAX(post_modified_gmt) m FROM {$wpdb->posts}
                    WHERE post_type IN ('post','page') AND post_status = 'publish'", ARRAY_A),
    $wpdb->get_results("SELECT t.slug, tt.count FROM {$wpdb->terms} t
                        JOIN {$wpdb->term_taxonomy} tt ON tt.term_id = t.term_id
                        WHERE tt.taxonomy = 'category' ORDER BY t.term_id", ARRAY_N),
    $wpdb->get_col("SELECT user_nicename FROM {$wpdb->users} ORDER BY ID"),
    get_option('rank-math-options-sitemap'),
    get_option('rank-math-options-titles'),
    array_keys((array) get_option('quickppr_redirects', [])),
    array_keys(site_url_redirects()),
  ]));
}

Check it before the plugin reads its cache. On sitemap requests only, early in the request, compare the fingerprint with the last one stored. If it changed, delete the cached sitemaps in one statement and store the new fingerprint. The plugin's own (crash-prone) invalidation is switched off.

add_action('parse_query', function ($query) {
  if (!$query->is_main_query() || !get_query_var('sitemap')) return;
  $fingerprint = site_sitemap_fingerprint();
  if (get_option('site_sitemap_fingerprint') !== $fingerprint) {
    site_sitemap_clear_cache();                 // DELETE the cached sitemap rows
    update_option('site_sitemap_fingerprint', $fingerprint, false);
  }
}, 0);

The check is a handful of cheap aggregate queries — trivial next to a ten-second rebuild.

Let the CDN help. Sitemap responses now carry Cache-Control: public, max-age=300, s-maxage=3600: browsers keep them for five minutes, the CDN for an hour. Most scraper traffic now never reaches WordPress at all.

The old disable-caching line is still there, commented out, as a one-line rollback switch.

A smaller sitemap, too

While I was in there, I found that about 1,132 of the 3,001 URLs in the learning-hub sitemap were published posts that 301-redirect somewhere else. That wastes crawl budget and shows up in Search Console as "page with redirect". The day before the caching change, I filtered them out: the sitemap query now reads both redirect stores live and excludes those post IDs — from both the page query and the count query, so pagination stays correct. New redirects drop out automatically.

That fix was about SEO, but it also means each rebuild — on the rare occasion one happens — does about 38% less work.

Trade-offs I accepted

• Up to an hour of edge staleness. After a change, the CDN can serve the previous sitemap for up to an hour. For a sitemap, that's fine.
• Not every input is fingerprinted. An image change that doesn't touch a post's modified time won't invalidate the cache. The fingerprint covers what actually changes sitemaps in practice; anything else is picked up the next time a post is saved.

What I took from it

A hotfix that removes a symptom can become a standing cost. Turning the cache off fixed staleness in an afternoon and cost a core's worth of CPU for almost two years. The bug was never "caching is broken" — it was "invalidation only runs in some places".

Invalidate on what the output depends on, not on who remembered to call clear(). Event-based invalidation is only as good as the list of events. A fingerprint of the inputs doesn't care how the change happened.

With more than one container, local caches are a trap. If a cache can't be shared or invalidated everywhere, it's a source of inconsistency, not speed.

Make the cheap thing cacheable at every layer. The most effective CPU optimisation here wasn't in PHP at all — it was letting the CDN answer.