Crawl Budget

Crawl budget is the number of URLs a search engine like Google will crawl on your site within a given period. It is not a fixed quota you are handed; it emerges from two forces. The first is crawl capacity — how much crawling your servers can take without slowing down for real users. If the site responds fast and reliably, the crawler will fetch more; if it is slow, errors out, or makes expensive database calls per request, the crawler eases off to avoid overloading it. The second is crawl demand — how much the search engine actually wants to crawl your URLs, driven by their popularity, freshness, and value. Pages that change often or matter to users get revisited more; stale, unpopular, or low-value URLs get crawled rarely. Crawl budget is the practical outcome of both, the share of attention your site's URLs receive.

Crawl budget matters because a search engine cannot crawl every URL on the web as often as it would like, so it rations attention. For most sites this rationing is invisible and irrelevant — if your site is small and crawled the same day you publish, you do not need to think about crawl budget at all, and Google says as much. The concern is real only at scale or under strain. Very large sites with many thousands or millions of URLs, sites that generate endless low-value pages, and slow or error-prone sites can find that the crawler simply does not reach or revisit everything in time. When that happens, new pages take longer to index, updates lag, and important URLs compete for crawl attention with junk ones.

Be honest about when crawl budget is and is not your problem, because most SEO advice over-weights it. For a site of a few hundred or a few thousand well-organized pages on a healthy server, crawl budget is a non-issue: Google will keep your sitemap fresh and index coverage fine without any special effort. Worrying about it there wastes energy that belongs on content and relevance. Crawl budget becomes a genuine lever in specific conditions — very large sites (think e-commerce catalogs, classifieds, or databases with hundreds of thousands or millions of URLs), sites that spawn vast numbers of low-value or near-duplicate URLs through faceted navigation, parameters, or pagination, and sites slow enough that crawl capacity itself caps how much gets fetched. In those cases, crawl budget directly shapes what gets discovered, indexed, and refreshed.

Recent guidance has sharpened the emphasis on speed and efficiency, not just size. Search engineers have stressed that a site making expensive, slow server calls per request costs the crawler more and earns less crawling — so backend performance and efficient infrastructure increasingly govern crawl budget, alongside sheer URL count. The practical reading is that crawl budget is partly a technical-health issue: a fast, lean site invites more crawling, while a heavy, sluggish one throttles itself. So the question is not only how many URLs you have, but how cheaply the crawler can fetch them and how worth fetching they are. For a small, fast site, none of this bites; for a large or slow one, it is one of the most consequential technical-SEO levers there is.

Manage crawl budget by helping the crawler spend its attention on the URLs that matter and stop wasting it on the ones that do not. Keep the site fast and reliable, because crawl capacity rises with performance and falls with slow, expensive responses. Prune or block the low-value URLs that bloat the crawl space — endless faceted-navigation combinations, parameter variations, session URLs, and near-duplicates — using robots rules, canonical tags, parameter handling, and a clean architecture. Maintain accurate sitemaps, fix crawl errors and broken links, avoid long redirect chains, and keep internal linking pointing crawlers toward your important pages. The aim is to raise both halves of the equation: more capacity through speed, and better-directed demand by removing junk so the crawler reaches what counts.

The failures cut both ways. One is obsessing over crawl budget on a site too small for it to matter, which diverts effort from real ranking work. The other is, on a genuinely large site, letting low-value URLs proliferate unchecked, running slow servers that throttle crawling, and neglecting sitemaps and crawl errors until important pages go unindexed or stale. The discipline is to diagnose honestly first — confirm whether crawl budget is actually constraining you by checking index coverage and crawl stats — and then, only if it is, act on both capacity and demand: a fast site with a clean, high-value URL space that the crawler can move through efficiently. Crawl budget rewards technical hygiene and punishes sprawl, but only where there is enough scale or slowness for it to bite.

Crawl budget — the URLs a search engine crawls per period, set by crawl capacity and crawl demand — is a real lever for very large or slow sites and a non-issue for small, fast ones.

Etymology: source.

Search interest for this term over the last five years:

View interest-over-time on Google Trends →

Areas of marketing where crawl budget is a core concern: