Crawl Rate Limit or Crawl Demand: Which One Limits Your Site?
SEO

Crawl Rate Limit or Crawl Demand: Which One Limits Your Site?

Crawl budget has two limits. The crawl rate limit is the load your server carries, and crawl demand is Google's interest in your pages. Only demand lifts the ceiling. On six sites we checked, crawling rose 3.6 times on one and stayed flat on another, so tell the two apart first.

Sercan Gökpınar
14 min read

Before a page can appear in search results, Google has to crawl it: Googlebot requests the page from your server and reads it. Google does not crawl any site without limit, and the URLs it can and wants to crawl are its crawl budget. What Google can crawl is bounded by your server's load, and that ceiling is the crawl rate limit, documented today as the crawl capacity limit. What it wants to crawl is crawl demand, and every site starts at the same conservative default, rising only where demand meets a healthy server.

The most widely circulated definition reduces crawl budget to one number: the pages Googlebot crawls in a given period. Measuring the output costs you the diagnosis: a falling page count cannot say whether your server or Google's interest caused the drop. An article on Google's July 22, 2026 documentation update goes further, turning the two limits into a formula where the budget is whichever is lower. Google's documentation carries no such formula, and the shortcut hides what decides where your effort goes: the capacity ceiling moves with demand.

To see what that changes, we read the Search Console Crawl Stats of six sites: three we manage, two we reported on, and our own. After we moved one site's infrastructure in July 2026, average response time fell from 2,339 ms to 416 ms and crawling rose about 3.6 times. On another site that got about 20% faster, daily crawling stayed flat, as we expect when demand is the limiting side. Before you put effort into the server or into your content, we suggest working out which limit binds your site.

Two Limits, Two Questions: Can Google Crawl, and Does It Want To

Google's crawl budget documentation builds a site's crawl budget from two parts. The core of the definition has held since 2017: the URLs Google can and wants to crawl on that site. The first part measures how much your server can take, the second how interested Google is. Google calculates it per hostname, so two subdomains of one domain count as two sites with two budgets.

Crawl Rate Limit Is Now Called Crawl Capacity Limit

The Google post from 2017 that introduced the term defined the crawl rate limit as the number of parallel connections plus the wait between fetches. The current documentation calls the same mechanism the crawl capacity limit, also known as hostload. It measures the total time your server spends holding connections open for Google. Reading the crawl rate limit as "the most pages crawled in a period" therefore misleads, since the ceiling is measured in time.

The limit moves both ways with crawl health. It goes up while the server answers consistently and Time to First Byte (TTFB) stays stable or improves. It goes down when responses slow or 5xx and 429 responses appear. The documentation names a second ceiling as well, since Google's own resources are finite.

Capacity Is Shared Across Google Crawlers, Demand Is Set per Crawler

Demand splits by crawler. In Google's example, AdsBot shows higher demand on a site running dynamic ad targets, and Google Shopping for the products in your merchant feed. The July 22, 2026 update adds the other half: capacity is shared across all crawlers. High demand from one crawler can therefore reduce what is left for the others.

For Googlebot, demand depends on a site's size, update frequency, page quality, and relevance compared with other sites. What you can influence is narrower: perceived inventory, popularity, and staleness. Google names perceived inventory, the set of URLs it knows about on your site, as the factor you have the most influence over.

Media

Demand Raises the Ceiling

Google's documentation ties the two limits together, and that link decides which side of your site is worth your effort.

Every Site Starts With the Same Conservative Default

According to the documentation, every site starts with the same default, conservative crawl capacity limit. The rise carries a condition: if there is "demand to crawl more and the site remains healthy," Google's systems adjust the limit over time. A healthy server allows the ceiling to rise, then, and demand is what sets the rise in motion.

The document's summary closes the other direction: even if the capacity limit isn't reached, Google crawls your site less when crawl demand is low. Crawl volume grows only when Google both can and wants to crawl more.

Media

Below the Ceiling, Freed-Up Crawl Time Goes Unused

The most practical consequence shows up in robots.txt. A caution box in the documentation tells you not to use robots.txt to temporarily reallocate crawl budget. Google shifts freed-up budget to other pages only if your site is already hitting its crawl capacity limit. Below that ceiling, blocking a group of URLs hands nothing to your important pages, and the time goes unused.

Our post on robots.txt crawl budget management covers what the file does for crawl budget. The same logic reaches further: when demand is the limiting side, nothing that eases capacity raises crawling on its own.

The Controls a Site Owner Has Only Turn Crawling Down

Every lever a site owner holds over the Googlebot crawl rate only slows it down. The Crawl Rate Limiter tool in Search Console was available for more than a decade and was removed on January 8, 2024. The 2017 post had already said that the setting could only reduce crawling and that a higher limit would not raise it.

Googlebot ignores the nonstandard crawl-delay rule in robots.txt, so writing crawl-delay: 0 does not speed it up either.

In an emergency, server responses are the fastest brake. Google's guide to reducing the crawl rate suggests returning 500, 503, or 429 instead of 200 for a few hours or 1–2 days. The slowdown covers the whole hostname.

The Crawl Stats help page warns that serving these codes past two or three days can signal Google to crawl less in the long term. A reporting form exists for infrastructure that cannot serve errors, but the guide states that you cannot request an increase in crawl rate.

Upward, Google lists two routes. Add server resources if the site genuinely cannot be crawled because of server capacity. In every case, raise your content's quality for the Google product you target. For Google Search, that quality covers popularity, overall user value, content uniqueness, and serving capacity.

Both Limits Show Up in Data From Six Sites

We checked the documentation against Search Console Crawl Stats exports from six sites we manage or have reported on. A and C are sites we prepared a report for; B, D, and F are sites we manage; E is our own site. Each period runs 88–89 days between late May and September 2026. By median daily crawl requests, the sites measure A 13,395, F 272, B 185, C 102, D 31, and E 6.

Crawling Rose 3.6 Times on a Site Whose Infrastructure We Moved

On July 21, 2026, we moved the infrastructure of Site F, a site we manage, to a new server and platform. In the 26 days before (June 24–July 19), average response time was 2,339 ms with 99 crawl requests a day. In the 60 days after (July 22–September 20), response time fell to 416 ms and requests rose to 356 a day. The site became roughly 5.6 times faster, and Google sent roughly 3.6 times as many requests.

Across the migration, requests went from 125 on July 20 to 232 and then 283, as response time dropped to 1,685, 717, and 488 ms. The busiest day was August 6, with 621 requests. Site F's same-day correlation between response time and daily requests is −0.75, the strongest of the six. That is the picture we expect where capacity binds: the server's ceiling rose and crawling followed.

We do not credit the whole increase to speed, because a migration changes more than speed. Download size per request rose from 68 KB to 133 KB, and 18% of responses were 302 and 12% 404. Part of the rise may therefore be changed URLs being fetched again.

With 96% of crawling Refresh and only 4% Discovery, we think the increase came mostly from recrawling rather than new URLs. Without server logs we cannot fully separate the two effects.

Media

A Site That Got About 20% Faster Kept the Same Crawl Volume

On Site C, a site we prepared a report for, the picture reverses. Across the first half of its 88-day period, the site averaged about 115 requests a day at a response time of 1,310 ms. In the second half, average response time fell to 1,049 ms, roughly 20% better, while daily requests averaged 113.

Google stated in 2017, and still confirms, that a faster site raises the crawl rate. Site C fits the condition attached to it: speed raises the ceiling, but when demand never reaches it, the higher ceiling goes unused. Site C is one site, so we read its flat crawling as consistent with a bottleneck on the demand side and nothing stronger.

Media

On the Other Sites, Capacity Leaves a Faint Trace and Demand Shows on Publishing Days

Where capacity binds, we expect response time and request count to move in opposite directions, because Google throttles requests as the server slows. On days with at least 20 requests, the same-day correlation was −0.75 on Site F. On the other five it stayed weak: A −0.22, B −0.16, C −0.12, D −0.01, and E 0.22.

The clearest trace of demand was on our own site, Site E, where crawling jumped on the days we published in bulk. The period's busiest day was September 17, with 243 requests, when we published nine posts together, while August 21, with four posts, reached 73. Our data does not show why August 29 came second with 151 requests. New content enlarges Google's perceived inventory, so demand rising on those days fits the documentation, though a correlation alone does not prove causation.

Media

Returning 304 for Unchanged Pages Frees Up Capacity

Where capacity goes matters even on sites it does not bind. On Site A, which we reported on, 91.2% of requests were refreshes of known pages, yet only 0.01% of responses were 304 Not Modified. According to Google, a bodiless 304 for an unchanged page saves bandwidth and server resources. Since capacity is shared, the saving can benefit every Google crawler.

On the same site, 59.1% of requests fell under Page resource load, the CSS, JavaScript, and similar resources Googlebot fetches to render pages. A share that large makes blocking them tempting. Our post on Google's render queue explains why rendering cannot be skipped. Our post on blocking /_next/ in a Next.js robots.txt shows what closing those resources costs.

The data has its limits too. Crawl Stats works at daily resolution, where an hourly capacity response may not show. We had no server access logs to compare against, and six sites are not a sample, so we read the finding as limited to them.

Reading Which Limit Binds Your Site

The two limits leave different traces in Search Console, and telling them apart first keeps your effort off the wrong side. If capacity binds, the work starts with the server team; if demand binds, it starts with your inventory and content.

Signs That Capacity Is the Limit

The most direct sign is the Hostload exceeded warning in the URL Inspection tool. Google gives it as its example of a site that cannot be crawled because of server capacity. The second sign is host status in Crawl Stats. After a significant availability issue in the last week, the help page asks you to check whether it recurs.

The third sign is a pattern, with requests falling while response time and the share of 5xx or 429 responses rise. Google throttles a server that responds slowly or returns a rising share of errors, and the drop is the trace of that reaction. Here the lever sits on the server side: more resources, faster responses, and 304 for unchanged pages.

Signs That Crawl Demand Is the Limit

If host status is clean, response time stable, and requests flat for weeks while new pages get crawled late, demand is most likely the limit. Among the sites its guide is written for, Google's documentation names those with many URLs sitting in Discovered - currently not indexed. That status means Google knows the URL but has not crawled it yet.

On the demand side, the levers are inventory and quality. Consolidating duplicates, returning 404 or 410 for pages removed for good, and keeping the sitemap current move perceived inventory toward the pages you want crawled. Over time, what carries demand is the quality and uniqueness of your content. To make this call on your site with data, we can start with a technical SEO audit that shows which limit binds your site.

Growing Demand Carries No Ranking Promise

Google wrote in 2017 that a higher crawl rate does not by itself bring better positions. Crawling is needed to appear in results, but it is not a ranking signal. The current documentation adds that not every crawled page gets indexed, a step we cover in our post on crawling vs indexing. Growing demand gets your new pages seen sooner, and other signals decide where they rank.

Frequently Asked Questions

Can You Change the Crawl Rate Limit in Search Console?

No, Google removed the Crawl Rate Limiter tool from Search Console on January 8, 2024, and the tool could only ever reduce crawling. Today you slow Googlebot down with server responses or Google's reporting form.

Does Googlebot Respect crawl-delay?

No, Googlebot does not process the nonstandard crawl-delay rule, so the line does not slow it down. For a short-term brake, Google recommends returning 500, 503, or 429 instead.

Does a Faster Site Get More Crawl Budget?

It does when server capacity limits crawling, and it does not when demand is the limiting side. After we moved one site's infrastructure, average response time fell from 2,339 ms to 416 ms and daily crawling rose about 3.6 times. On another site that looked demand-bound, a speed-up of about 20% left daily requests unchanged.

Do AI Bots Share Googlebot's Crawl Capacity?

The shared capacity in Google's documentation covers only Google's own crawlers and does not name third-party AI bots as part of that pool. An indirect effect is still possible: if heavy bot traffic slows your server, Google lowers the crawl capacity limit for the slower server.

What Does Hostload Exceeded Mean?

The warning in the URL Inspection tool means the site could not be crawled because of server capacity. Hostload is Google's other name for the crawl capacity limit, and Google recommends adding server resources if that makes sense for your business.

Sources and Further Reading