Crawl budget comes down to two things: crawl capacity (how much load your server can take without degrading) and crawl demand (how much a page's authority and freshness make Google want to fetch it). On large sites — tens or hundreds of thousands of URLs — misallocated crawl budget means delayed indexing for new pages and stale content lingering in the index.
Where the waste usually comes from
- Faceted navigation loops — filter and sort combinations on category pages generate near-infinite URL permutations. If a crawler follows uncanonicalised filter parameters, it can spend a large share of its total crawl budget on near-duplicate pages. Fix with nofollow on dynamic filter links, parameter blocking in robots.txt, or client-side filtering that doesn't mutate the URL for non-indexable facets.
- Orphan URLs — pages present in a sitemap or database but with no internal links pointing to them get crawled far less reliably than properly linked pages, regardless of how important they are.
- Internal redirect chains — an internal link pointing at a URL that itself redirects wastes a fetch cycle on the hop. Update internal links to point directly at the final 200 destination.
A rough health check
As a directional metric, divide the number of unique indexable pages crawled by the total number of Googlebot hits in your logs over the same period. If a large share of total hits are landing on non-indexable, duplicate, or redirect URLs rather than valuable pages, that's a concrete, fixable crawl-efficiency problem rather than a vague one.
Related articles
Extracting Googlebot Insights From Server Log Files
GSC gives you aggregates. Log files are the only record of what actually happened, request by request.
Google Indexing API vs. IndexNow: A Practical Comparison
Passive sitemaps aren't enough for high-velocity publishing. Here's how the two push-based indexing protocols actually differ.
Monitoring AI Crawler Logs: SearchGPT, ClaudeBot, and Perplexity
A growing share of discovery is shifting to AI answer engines. Your log files already show whether they can see your site at all.
Written by
Paul Lovell
International SEO Consultant & Founder of Always Evolving SEO. 15+ years running technical audits, log-file analysis, and root-cause fixes for multimillion-dollar businesses — speaker at SMX London, Search & Content Summit, and SEMrush events.