Crawl budget

4 minute read · Updated 28 August 2026

Definition

Crawl budget is the finite attention a search engine gives a site — how many of its URLs get fetched in a given period. Google describes it as the product of two things: the crawl capacity your server can handle without slowing down, and the crawl demand your content justifies.

Below a few thousand pages it is rarely a real constraint. Where it does bind, internal links decide how the attention is spent: well-linked pages get revisited, orphans do not.

Most sites do not have a crawl budget problem

Worth saying plainly, because the term gets used to justify a great deal of unnecessary work. A site with a few hundred pages that publishes weekly is not competing for crawler attention in any meaningful sense. If such a site has pages that are not indexed, the cause is almost always something else: nothing links to them, they duplicate another page, or they are not worth indexing.

Crawl budget becomes a genuine constraint on large sites — shops with faceted navigation generating thousands of parameter URLs, publishers with deep archives, sites with slow servers that force the crawler to back off.

Where it actually gets wasted

Faceted navigation. Every combination of filters is a URL. Colour times size times price band is thousands of near-identical pages.
Session and tracking parameters. The same page fetched a dozen times under different query strings.
Redirect chains. Each hop is a request that returns no content.
Soft 404s and thin archives. Empty tag pages, date archives nobody reads.
Infinite calendars. A booking widget that generates a URL for every future month, forever.

The common thread: the crawler spends its visit on URLs nobody wants indexed, and the pages that matter are checked less often as a result.

What internal linking has to do with it

Crawl demand is partly a function of how important a page looks, and internal links are the main signal of importance a site controls. Pages with many inbound internal links and low profundidade de cliques get fetched more often; changes to them are noticed sooner.

So the practical advice is the same advice as everywhere else on this site: ligação the pages that matter, from pages that themselves get linked. What it is not: adding nofollow to internal links to steer the crawler. That does not conserve budget — use robots.txt for URLs that should never be fetched, and remove links to URLs that should not exist.

How to see your own

Search Console has a Crawl Stats report under Settings: requests per day, average response time, and a breakdown by file type and purpose. If total requests are flat while the site grows, or average response time is climbing, that is worth reading. Server log files give the same picture in more detail if you can get at them.

Crawl rate and crawl demand

The two halves behave differently and are worth separating.

Crawl capacity is a technical ceiling. Google raises it while the server responds quickly and lowers it when responses slow down or errors appear. A site returning 500s under load is teaching the crawler to visit less often, which is the one crawl budget problem that is genuinely a server problem.

Crawl demand is editorial. It rises with popularity, with how often content actually changes, and with how important a URL looks from its links. Republishing an unchanged page with a new date does not create demand; it teaches the crawler that dates on your site mean nothing.

Symptoms worth acting on

New pages take weeks to appear. On a small site this is a linking problem, not a budget problem — check whether anything links to them.
Changes to existing pages are noticed late. Look at where those pages sit in the link graph.
Search Console shows many URLs as Discovered – currently not indexed. Google knows the URLs exist and has chosen not to fetch them. On a large site this is the clearest crawl-budget signal there is.
Crawl Stats shows requests flat while the site grows. Worth reading alongside average response time.

Absent these, effort spent on crawl budget is effort not spent on something that would move the numbers.

The order of operations

Where it does bind, the sequence matters because each step reduces the work of the next.

First, stop generating junk URLs. Faceted navigation and tracking parameters produce more waste than everything else combined. Block the patterns in robots.txt or stop linking them.

Then flatten redirect chains. Each hop is a request that returns no content.

Then fix the internal links. Pages worth crawling should be well linked and shallow; pages not worth crawling should not be linked at all.

Then clean the sitemap. Only canonical, indexable URLs, with accurate modification dates.

Related terms