SEO

Crawl Budget: What It Is and Who Should Care About It

Crawl budget is the set of URLs Google can and wants to crawl on a hostname. Two levers set it: the capacity limit your server sustains, and Google's demand to crawl. Google writes its guidance for sites above 1 million pages, or above 10,000 pages that change daily.

Sercan Gökpınar

Two questions hide inside the phrase crawl budget, and they deserve separate answers. The first is mechanical: how does Google decide where to spend its crawling. The second matters more and gets asked less: does your site have this problem at all. For most sites the honest answer is no, and saying so early beats selling a fix nobody needs.

The term also gets blamed for problems it did not cause. Pages missing from the index, content that takes weeks to surface, traffic that slipped last quarter: all three get traced back to crawl budget by reflex. The sections below build the mechanism first, then separate the symptoms that genuinely belong to it.

What Crawl Budget Actually Is

Google's Definition, in Google's Words

Google's crawling infrastructure documentation defines crawl budget as the set of URLs Google can and wants to crawl. Both halves carry weight.

The first half is a capacity question: how much your server sustains. The second is a demand question: how much Google wants. You can sit far below your capacity ceiling and still be crawled lightly, because demand is low. The two levers move independently, and diagnosis only works once you separate them.

What the Common Definition Loses

Most articles describe crawl budget as the number of pages Googlebot crawls in a given period. As a measurement, fine. As a definition, incomplete.

Page count is the output. The budget is the pair of levers producing it. Consider a concrete case: crawling on your site fell 40% over one quarter. Under the output definition you hold a single fact, the drop. Under the mechanism definition you can ask two different questions. Did the server slow down, or did Google's interest fade? The fixes live in different departments.

Some definitions stretch further, calling crawl budget the number of pages Googlebot crawls and indexes. Worth recording plainly: crawling and indexing are separate decisions, and crawl budget touches only the first. The full separation is covered in crawling vs indexing.

Budget Is Assigned Per Hostname

Google's crawling infrastructure treats a unique hostname as a site. www.example.com and blog.example.com are two sites with two budgets.

Three consequences follow. A section moved to a subdomain starts building its own budget from the same conservative default every site begins with. If both the www and non-www versions answer, crawling splits across two buckets. Serving static assets from a separate host takes those requests off the main host's budget.

The Two Levers: Crawl Capacity Limit and Crawl Demand

What Raises and Lowers the Capacity Limit

Google calls this the crawl capacity limit; Search Console calls the same thing hostload. What it measures is not pages but time: how long your server spends holding connections open for Google, across however many run in parallel.

Every site starts at the same conservative default. The limit then moves in two directions:

  • Up. When a site responds consistently and its latency and Time to First Byte stay stable or improve, Google raises the number of connections it uses.
  • Down. When response times stretch, or the server returns 5xx errors or rate-limiting responses such as 429, Google crawls less.

Google's documentation frames the relationship in both directions, so the lever is real rather than folklore. Note which speed this is: server response time, not how fast the page feels in a browser. Core Web Vitals metrics such as LCP, INP, and CLS measure the user's experience. Crawl capacity responds to how quickly the server answers.

A second ceiling sits on Google's side. The documentation states that its resources are finite and must be allocated across the web, which means googlebot crawl budget is not a resource you can buy unilaterally by adding servers.

The Three Signals Behind Crawl Demand

If capacity asks whether Google can, crawl demand asks whether Google wants to. Three factors drive it for Googlebot:

  1. Perceived inventory. Without guidance, Google tries to crawl every URL it knows on your site. When that set is full of duplicates and unimportant variants, crawling time drains into them.
  2. Popularity. URLs that are more popular across the internet get crawled more often so they stay fresh.
  3. Staleness. Google recrawls often enough to catch changes.

Demand is also not one number. Each crawler carries its own: AdsBot shows higher demand on sites running dynamic ad targets, Google Shopping on products in merchant feeds. Site-wide events such as a migration push demand up temporarily while content is reprocessed under new URLs.

The Only Signal You Control

You cannot dial popularity or staleness directly. Google marks perceived inventory as the factor you can most positively control.

Crawl budget optimization is therefore URL inventory management. Google's recommended order runs like this: consolidate duplicate content first, block genuinely unimportant pages with robots.txt where consolidation is impossible, return 404 or 410 for permanently removed pages, clear soft 404s, keep sitemaps current, and avoid long redirect chains.

The order is deliberate. Consolidation comes before blocking, because blocking also prevents Google from seeing the consolidation signals.

Does Your Site Actually Have This Problem?

Google's Three Thresholds

Google states which sites its guide is written for:

  • More than 1 million unique pages with content changing moderately often, roughly weekly
  • More than 10,000 unique pages with content changing very rapidly, daily
  • Sites where a large share of URLs sit in Discovered - currently not indexed in Search Console

Below those thresholds, Google's advice shrinks to two items: keep your sitemap current and check the Page Indexing report regularly. If your pages get crawled the day you publish them, crawl budget is not your problem.

Four Symptoms That Beat the Thresholds

The third item is more useful than the first two, because it names a symptom rather than a page count. A site with 8,000 pages can have this problem; a site with 400,000 clean ones may not.

Four signals worth watching:

  1. Large groups of pages stuck in Discovered - currently not indexed
  2. Important pages taking weeks to appear after publication
  3. Thousands of low-value URLs getting crawled while priority pages are skipped
  4. A crawl stats report where activity concentrates on parameter URLs and filter variations

All four say the same thing from different angles. Google may be crawling plenty and crawling the wrong things. "Are we crawled enough" is the wrong diagnostic question. "Does the distribution match our priorities" is the right one.

Read together, the four also give crawl budget waste a working definition: any request Google spends on a URL you would not have asked it to fetch. Counting those requests is more useful than counting total ones.

What Drains Crawl Budget

Duplicate URLs and Crawl Traps

A crawl trap is any structure that is finite for a user and unbounded for a crawler.

On ecommerce sites the usual culprit is faceted navigation. Filters for price, color, brand, and size multiply rather than add. /shoes?color=black, /shoes?color=black&size=10, and /shoes?size=10&brand=nike are three useful views for a shopper and three near-identical URLs for a crawler.

Templates that generate endlessly come second. Calendar pages are the classic: if the system produces a page for any date, a crawler can walk forward without limit. Badly built pagination comes third.

The fix order matches Google's: parameter handling and canonical consolidation first, robots.txt only for what cannot be consolidated.

Choosing the Wrong Directive

Three directives look interchangeable and behave differently in the crawl queue.

DirectiveStops crawlingStops indexing
`robots.txt` disallowYesNo, a blocked URL can still be indexed
`noindex`NoYes
`404` / `410`Over time, strong signalYes

Two frequent mistakes fall out of that table. The first: noindex saves no crawl budget. A crawler has to fetch the page to read the directive, so the page keeps consuming requests.

The second is quieter. Writing a permanently deleted page into robots.txt is a worse choice than returning the right status code. Google treats 404 as a strong signal not to crawl a URL again. Blocked URLs, by contrast, stay in the crawl queue far longer and get recrawled the moment the block lifts.

Blocking carries one more hidden cost. Because Google cannot fetch a blocked page, it cannot read the canonical tag inside it either. Blocking duplicates freezes the duplicate problem rather than solving it.

The Server Side

One lever sits untouched on most sites: HTTP caching. When a page has not changed since the last crawl, returning 304 (Not Modified) lets Google reuse its cached copy. No body is transferred, and bandwidth and server capacity come back. Checking costs little; fixing it is infrastructure work.

Reading the Crawl Stats Report

The Order That Matters

The report lives under Settings → Crawl stats in Search Console. Read it in sequence rather than hunting for one number, because each step makes sense of the next:

  1. Host status. Can Google reach the server at all. Availability failures, timeouts, DNS errors, or a robots.txt that will not fetch mean you have a front-door problem rather than a budget problem. Close that first.
  2. Crawl request trend. On its own, neither a rise nor a fall is news. Compare the trend against what happened on the site: a migration, a new filter system, a publishing push that failed to move the line.
  3. Average response time. Look for sustained increases and whether fewer pages get crawled as the number climbs.
  4. Response codes. Waste becomes visible here: stacks of 404s, redirect chains, soft 404s.
  5. Crawl purpose. Refresh against discovery. A healthy site shows both. Steady publishing with flat discovery points at internal linking or demand.
  6. Page Indexing report. Google recommends reading the two together. Discovered points at crawling; Crawled points at content.
  7. URL Inspection. Per-URL confirmation: last crawl, index status, and whether Google picked the canonical you expected.

When to Move to Log Files

Crawl stats give totals and trends, never which URL was requested how many times. Server logs give exactly that.

Draw the line this way: crawl stats answer whether a problem exists, logs answer which URLs it lives in. Crawl stats come first because log analysis needs server log access and comfort with large datasets. Scale sets a practical threshold too. A few thousand wasted requests on a small site changes nothing; the same ratio across millions of URLs is money.

Do AI Crawlers Eat Your Crawl Budget?

No Directly, Yes Indirectly

Two separate things get conflated here.

Crawl budget is an allocation specific to Googlebot. Pages fetched by GPTBot, ClaudeBot, or PerplexityBot are not deducted from what Google set aside for your site. No direct effect exists.

The indirect effect does, and it travels through the capacity limit. Google lowers that limit when a site slows down or returns server errors. Heavy AI crawler traffic is one thing that slows a server. The chain runs: bot load rises, response time stretches, capacity limit falls, Googlebot crawls less.

Measure Before You Block

Start by measuring. Separate AI bot user agents in your server logs, quantify the load, and restrict at the robots.txt or CDN level only if resource consumption is genuinely causing harm.

Blocking without measuring repeats the same mistake as blocking duplicates blindly. Close more than you meant to and you never see what you lost.

More Crawling Is Not a Ranking Promise

The Chain Runs One Way

Crawling is discovery. Indexing is a separate decision made afterward about whether a page deserves storage. Ranking is a further judgment about whether an indexed page deserves a place for a specific query.

The direction is strict. A page that is never crawled cannot be indexed, and a page that is never indexed cannot rank. The reverse does not hold. Crawl budget acts on the first link alone, and a rise in crawling shows that Google allocated more resource to your site. It carries no promise about traffic or position.

The Two Ways to Get More Budget

Google lists exactly two. Adding server resources, which helps only when capacity is genuinely the constraint. And improving content quality for the Google product you are targeting: popularity, overall user value, content uniqueness, and serving capacity.

What the list leaves out informs as much as what it contains. Submitting a sitemap, pinging, and inflating internal link counts are absent. A sitemap does not enlarge the budget; it improves discovery and prioritization inside whatever budget exists.

That framing also inverts a common assumption. In Google's own documentation, quality drives crawling. Crawling does not drive quality.

Frequently Asked Questions

What is crawl budget in simple terms?

It is the set of URLs Google can and wants to crawl on your hostname, not a fixed page allowance. Two levers produce it: the capacity your server sustains and Google's demand to crawl your pages. Because it is a product of two variables, no single number answers "how big is my budget."

Does submitting a sitemap increase crawl budget?

No. A sitemap does not persuade Google to allocate more resource to your site. What it does is keep important URLs from getting buried inside complex architecture and help new pages get found sooner. Its value comes as much from what it excludes as from what it lists.

Does noindex save crawl budget?

No. noindex prevents a page from entering the index but does nothing to stop crawling, because Googlebot has to fetch the page to read the directive. Cutting crawl waste means shrinking the URL inventory, not adding directives to it.

Does crawl budget matter for small sites?

For most small and mid-sized sites, no. Google writes its guidance for sites above 1 million pages, or above 10,000 pages changing daily. If your pages get crawled the day you publish them, the work reduces to keeping your sitemap current and watching the Page Indexing report.

Sources and Further Reading