How to audit and fix crawl budget waste on large websites

Key takeaways

• Google says crawl budget is mainly a concern for sites with 1 million+ pages that change weekly, sites with 10,000+ pages that change daily, or sites with many “Discovered – currently not indexed” URLs.
• Server log analysis is the only way to see exactly where Googlebot spends its requests.
• Faceted navigation, URL parameters and internal search pages are the most common sources of crawl waste.
• robots.txt stops crawling; noindex does not, because Google still has to request a page to see the tag.
• Clean sitemaps, fast servers and short redirect paths help Google reach new and updated content sooner.

Google allots each site a limited amount of crawling. For small sites the limit rarely makes a difference. For large catalogs, publishers and marketplaces, wasted crawling can delay indexing of the pages that earn revenue. This Emulent guide explains how to tell whether you have a crawl budget problem, find the waste and fix it.

What signals tell you crawl budget is a problem?

Slow indexing of new pages, crawling of URLs you never meant to expose and a growing pile of discovered-but-unindexed URLs. Google’s own crawl budget guide says it is written for large sites (1 million+ unique pages changing about weekly), medium sites (10,000+ pages changing daily) and sites with many URLs marked “Discovered – currently not indexed.”

• New pages take a week or more to be indexed even though they are linked and in your sitemap.
• The Crawl Stats report in Search Console shows flat or falling crawl requests, rising response times or many requests to non-200 URLs.
• Server logs show Googlebot requesting session IDs, sort parameters or internal search URLs.
• Important pages show old last-crawl dates in the URL Inspection tool.

Google retired cached page links in 2024, so use URL Inspection rather than the old cache view to check when a page was last crawled. If two or more of these signs appear, an audit is worth the time.

How do you use log file analysis to find crawl waste?

Pull your server or CDN access logs, isolate verified Googlebot requests and group them by URL type. Search Console shows what is indexed; logs show what was actually crawled.

• Export at least 30 days of logs and filter for Googlebot user agents, then confirm requests are genuine with a reverse DNS lookup or Google’s published IP ranges.
• Bucket URLs into products, categories, articles, parameter variations, pagination, internal search and so on.
• Compare the share of requests going to indexable, revenue-earning pages against non-indexable or duplicate URLs.
• Find the URL patterns that take a disproportionate share of requests.

In one Emulent client engagement, a retailer’s logs showed about 60% of Googlebot’s daily requests going to faceted filter combinations with no unique content. Your mix will differ, which is why the logs matter.

Which fixes have the biggest impact on crawl efficiency?

Fixes that stop low-value URLs from being crawled at the source.

• Block URL patterns that should never be crawled, such as internal search results and endless filter combinations, in robots.txt. Google warns not to use robots.txt to temporarily shift budget, so use it for permanent exclusions.
• Stop generating crawlable links to useless parameter variations; canonical tags consolidate signals but Google may still crawl the variants.
• Remove or return 404/410 for thin pages you do not need, such as empty tag archives. A noindex tag keeps a page out of the index, but Google still requests it.
• Fix soft 404s, which Google says keep getting crawled.
• Update internal links that point to redirects, and avoid long redirect chains.
• Improve server response times; Google says faster pages may let it read more of your site.

After changes, watch Crawl Stats and logs for two to four weeks to confirm Googlebot shifts toward priority pages.

How should you structure your XML sitemaps to guide crawling?

Treat sitemaps as a clean list of URLs you want indexed.

• Include only indexable, canonical URLs that return 200, are not blocked and carry no noindex tag.
• Split sitemaps by content type, such as products, articles and locations, to spot indexing gaps by section.
• Set lastmod only when content meaningfully changes; Google recommends the tag for updated content and learns to ignore it when it is inaccurate.
• Keep each file under 50,000 URLs or 50 MB uncompressed, and use a sitemap index for more.

Sitemaps support internal linking rather than replacing it. A page found only through a sitemap, with no internal links, sends a weak signal.

How do you maintain crawl efficiency as your site grows?

Make crawl checks part of normal operations.

• Repeat log audits quarterly and compare results with earlier baselines.
• Review the crawlable URL impact of every launch, migration or new filter before it ships.
• Compare indexed page counts in Search Console with sitemap counts, and investigate large gaps.

For WordPress sites, Emulent pairs these checks with fast Kinsta hosting and lean theme builds, so templates do not generate needless archive URLs.

How can the Emulent team help?

Emulent runs technical audits for enterprise website teams that include log file analysis, crawl diagnosis and implementation support, with ongoing monitoring as part of enterprise SEO engagements. A caveat: crawl improvements help Google find content but do not guarantee indexing or rankings, which still depend on quality. Contact the Emulent team to audit your crawl budget.