robots.txt Crawl Budget: What Blocking Actually Buys You
SEO

Where robots.txt Crawl Budget Management Ends

robots.txt crawl budget management is the second of six tools Google lists, and consolidating duplicates comes first. A blocked URL sits in the crawl queue longer than one returning 404 and its canonical tag goes unread. One external link can index it without content.

Sercan Gökpınar
11 min read

A robots.txt file does one job on the crawl budget side: it keeps URLs out of crawling while they carry on existing. Every other job carries its own cost: consolidating duplicates, removing a page for good, or taking something out of the index. In those jobs the file usually postpones the problem rather than solving it.

We keep seeing the file treated as a savings instrument. Someone lists a set of paths, writes them all into Disallow lines, and reads the drop in crawl requests as a win. The blocked URLs never leave the queue, and Google loses the signals inside the pages it can no longer fetch. On most sites the file's own mechanics are quietly doing something else entirely.

robots.txt Comes Second in Google's Own Tool Order

Google's crawl budget documentation treats only one of the signals behind crawl demand as something a site owner controls directly. That signal is the set of URLs Google knows about on your site. You cannot dial popularity or staleness. You can dial the inventory, and crawl budget work is inventory work in practice.

The Six Tools Google Lists

The same document gives its tools in a deliberate order:

  1. Consolidate duplicate content, and focus on unique content rather than unique URLs.
  2. Block crawling of unimportant pages that cannot be consolidated.
  3. Return 404 or 410 for pages removed for good.
  4. Clear out soft 404s, because Google keeps crawling them.
  5. Keep the sitemap current and use lastmod on content that changed.
  6. Avoid long redirect chains.

Consolidation comes before blocking because blocking also closes off Google's view of the consolidation signals. Blocking sits there as the second choice for pages that resist consolidation, never as the opening move.

Where Blocking Is the Right Tool

The place blocking genuinely wins is narrow and easy to describe. It covers structures that make sense to a person and produce a near-infinite set of URLs for a crawler. Faceted navigation is the standard case, because the combinations multiply as soon as price, color, and size filters can stack. Calendar templates and badly built pagination land in the same class.

What these URLs share is that nobody is deleting them. Visitors keep using them, so a 404 would be wrong, and there are far too many of them to consolidate one by one. That leaves robots.txt, and here it really is the right tool.

What a robots.txt Disallow Line Stops

A Disallow line stops crawling and leaves indexing alone. The three directives each do a separate job, and picking the wrong one usually starts with treating them as interchangeable. We covered crawling and indexing as two separate decisions in another post, and the choice of directive follows straight from that split.

DirectiveStops crawlingStops indexing
robots.txt disallowYesNo, a blocked URL can still be indexed
noindexNoYes
404 / 410Over time, a strong signalYes

The second row closes off a common mistake. A noindex directive lives in the page's HTML or in an HTTP header, so Googlebot has to fetch the page to see it at all. Every page carrying noindex keeps spending crawl requests and saves nothing on the crawl budget side.

Choosing a directive before you block
Choosing a directive before you block

A Blocked URL Stays in the Crawl Queue

Queue behavior runs against intuition here. Google's crawl budget documentation says blocked URLs stay part of the crawl queue much longer. The URL never leaves the line, and Google recrawls it as soon as the block is removed.

The same document describes 404 differently. Google does not forget a URL it knows, but a 404 is a strong signal to stop recrawling that URL. Writing a permanently deleted page into robots.txt is a weaker crawl budget decision than returning the right status code.

The "Indexed, though blocked by robots.txt" Warning

This line in Search Console's page indexing report sits under warnings rather than errors, and the distinction earns its place in a diagnosis. Google's definition holds three things at once. Google respects the block and never requests the URL, the URL can be indexed anyway, and the record that surfaces is thin. Google indexes it from whatever the linking page says, and the snippet, in the documentation's own words, will probably be very limited.

The way out splits in two, and the choice depends on intent. If the page genuinely should not be in search, remove the block and add noindex. If the page belongs in search, remove the block. Both paths open with the same step.

The Hidden Cost of Blocking

On a blocked URL Google loses more than the content. It loses every signal the page was carrying. When the blocking decision gets made, one gain usually gets counted: the requests that no longer happen. The three losses coming out of the same line never appear as a row in any report.

The four consequences of one Disallow line
The four consequences of one Disallow line

The Canonical Tag Goes Unread

Google's canonicalization documentation tells you, in its own "best practices" list, not to use robots.txt for canonicalization. The reason given there is the mechanism itself, since Google may still index a URL disallowed in robots.txt without its content. A blocked URL therefore does not drop out of the index; it stays there as an empty record.

Pointing parameter duplicates at the main version with a canonical tag consolidates them. Blocking those same URLs makes consolidation impossible, because Google cannot fetch the page and therefore cannot read the canonical tag inside it. Blocking a duplicate freezes the duplicate problem instead of clearing it.

Embedded Assets and Render Resources

A block reaches further than the path you wrote. Google's robots.txt introduction says images, video files, and PDFs embedded in a blocked page fall out of crawling as well. The one exception is an asset that some other crawlable page also references. On a site that earns traffic from image search, that is a second loss with no trace on the HTML side.

The same logic runs through rendering. Google's JavaScript documentation states that Google will not render JavaScript from blocked files, so a crawlable page whose script is blocked still renders empty. Blocking a framework's script directory is the concrete version of this, and we walked through Google's render queue step by step in a separate post.

The opening example in Google's own specification comes from the same place. It closes the directory holding .css and .js files to every crawler and opens it to Googlebot with an Allow line. The reason sits in a comment: "Google needs them for rendering."

Three Silent Mistakes in the File Itself

Everything above concerns the decision. Even with the decision right, the file may not read the way you expect, because Google's robots.txt specification contradicts three widespread assumptions at once. All three fail silently: the file is valid, the rule runs, and what you lost stays invisible.

Googlebot Reads One User-agent Group

The documentation leaves no room here: "Only one group is valid for a particular crawler… Other groups are ignored." The next sentence closes the gap, because user-agent-specific groups and the global * group are never combined. Write a separate group for Googlebot and Googlebot reads that group alone.

The trap this produces is everywhere. In the file below, /_next/ is not blocked for Googlebot, because the Googlebot group has no such line:

User-agent: *
Disallow: /_next/
Disallow: /api/

User-agent: Googlebot
Disallow: /admin/
Disallow: /api/

Bingbot, Applebot, and every AI crawler read the * group instead, so /_next/ stays blocked for them. The result inverts the author's intent. The file reads as "closed to everyone, open to Googlebot" but behaves the other way round: open to Googlebot and closed to everyone else. Nothing breaks on the Google side, so the mistake survives for months.

Rule Precedence Follows Path Length

The second assumption is that the file reads top to bottom and the last matching rule wins. Google ignores file order for conflicting rules: "crawlers use the most specific rule based on the length of the rule path. In case of conflicting rules, including those with wildcards, Google uses the least restrictive rule."

In practice that is a relief. Closing a directory and reopening a subdirectory inside it takes no particular ordering, because Allow: /_next/static/ next to Disallow: /_next/ is enough. The second path has more characters, so it wins. The mechanism runs the other way too: a broad Disallow with its exception forgotten is a mistake that raises no warning anywhere.

Path Matching Has No Regex

Google supports two wildcards in path values: * for zero or more characters and $ for the end of the URL. Character classes, ranges, and repetition operators are not part of the supported syntax. The parser does not flag them either. It reads them as part of the path and matches them literally.

The concrete case is a rule people write to close off deep pagination:

Disallow: /*?page=[2-9]*

The author believes pages two through nine are blocked. What the rule actually looks for is a URL carrying the literal string ?page=[2-9] in its path. No such URL exists, so the line blocks nothing and ?page=2 keeps getting crawled. Closing that path for real takes Disallow: /*?page=, and separating out the first page is something robots.txt cannot do.

Googlebot Ignores the crawl-delay Line

Google's parser recognizes four fields: user-agent, allow, disallow, and sitemap. The documentation states that other fields, crawl-delay among them, are not supported. A crawl-delay: 10 line does not slow Googlebot down, and crawl-delay: 0 does not speed it up.

The line is not meaningless everywhere, because crawlers such as AhrefsBot and SemrushBot do read it. It is meaningless for Google alone, and leaving that distinction out is how someone ends up with a wrong model of Googlebot's behavior. The ignored field carries a second, more dangerous effect. The parser does not treat an unrecognized line as a group separator. A line sitting between two user-agent blocks can therefore merge them into one group.

When robots.txt Crawl Budget Control Is the Right Tool

The case where robots.txt crawl budget control wins is narrow and describable. We suggest two questions before the blocking decision, because each one guards against a loss that is hard to reverse.

Two Questions To Ask Before You Block

The first question is about the URL's future: will this address carry on existing? If it was removed for good, the right tool is 404 or 410, because blocking holds the URL in the queue. If it duplicates content you already have, the right tool is a canonical tag. Blocking would make it impossible for Google to bind the duplicate to the main version.

The second question is about scope: what assets do we lose by closing this path? Images, PDFs, and script files living under the closed path fall out of crawling with it. The first question usually gets asked. The second one usually does not, and the difference reaches you two or three months later in image search or in a render output.

The File Itself Has To Stay Reachable

The last check concerns the file's own response, where two cases get mixed up constantly. According to Search Console's crawl stats documentation, a missing robots.txt, meaning one that returns 404, is a good response. The site counts as having no robots.txt and everything is treated as crawlable. An empty file is a valid response too.

Unreachable is the opposite. When Google cannot read the file, it cannot tell which rules apply, so it behaves conservatively and slows crawling down. If the file stays unreachable for a full day, Google stops crawling for a while until it gets an acceptable response. A bot protection layer serving Googlebot a different robots.txt, or a challenge page, falls under the same heading on your side.

Frequently Asked Questions

Does robots.txt help crawl budget?

In one narrow case, yes: URLs that carry on existing but do not need crawling stay out of crawling. Blocking sits second in Google's own tool order, and the first job on that list is consolidating duplicates.

Can a page blocked by robots.txt still be indexed?

Yes. Google respects the block and never requests the page, but it can index the URL from the information on the pages that link to it. Search Console reports this as "Indexed, though blocked by robots.txt", and the snippet that appears in search stays very limited.

Should I use robots.txt or noindex?

They do separate jobs. A noindex directive stops indexing and leaves crawling untouched, because Googlebot has to fetch the page to see the directive at all. Use noindex when the page should never appear in search, and robots.txt when the page does not need crawling.

Does Google support crawl-delay?

No. Google's parser recognizes only user-agent, allow, disallow, and sitemap, and ignores every other field, including crawl-delay. Crawlers such as AhrefsBot and SemrushBot do read the line, so there is no need to strip it from the file.

Sources and Further Reading