Crawling vs indexing reads like a technicality until the morning you open Search Console, find two hundred pages missing from the index, and have to decide where to look. Teams that skip the distinction lose months here. They hand a crawling problem to the content team and a content problem to the infrastructure team, then wonder why neither fix lands.
Shortened to crawl vs index in most conversations, the question is really about which of Google's decisions failed. The sections below split the chain into three links, show which Search Console screen exposes each one, and map every directive to the link it actually touches.
Crawling vs Indexing vs Ranking: Three Decisions
Crawling Is Discovery
Crawling means Googlebot follows a link, requests a URL, and sees what is there. On its own it promises nothing. What Google gains by fetching your page is the knowledge that the page exists.
Crawl budget operates only here. The capacity and demand levers described in what crawl budget is govern which URLs Google requests and how often. What they do not govern is what happens to those URLs afterward.
Indexing Is a Storage Decision
After crawling, Google makes a separate call: does this page deserve a place in the index?
That call looks at the content. Whether it duplicates something else, which version is canonical, whether the page offers a reader anything. Having been crawled is an input to the decision, never its outcome. A page can be crawled perfectly and stored nowhere.
Ranking Is Judged Per Query
The third link goes unmentioned in most articles. A page sitting in the index is separately assessed on whether it deserves a position for a particular query. Being indexed does not make a page visible.
A quick check demonstrates that the link is genuinely separate. Search the exact URL in quotes. If a result comes back, the page is indexed. If it is indexed and still takes no position for the query you targeted, the problem in front of you belongs to relevance and competition, and no technical fix will move it.
Direction matters here and runs one way. An uncrawled page cannot be indexed, and an unindexed page cannot rank. The reverse fails: crawling does not earn indexing, and indexing does not earn ranking.
Crawling vs indexing is the boundary between the first two links. Any fix applied without locating that boundary risks landing on the wrong link, however correct the fix is in itself.
Googlebot indexing decisions are also invisible from the crawling side. Nothing in your crawl stats tells you why a fetched page was not stored, which is why the two reports in Search Console have to be read together rather than in isolation.
Where Crawl Budget Stops
Google's Definition Ends at Crawling
Google's crawling infrastructure documentation defines crawl budget as the set of URLs Google can and wants to crawl. Indexing appears nowhere in the definition, because the budget's job finishes before it.
The practical translation: enlarging your crawl budget does not raise your indexing rate. The budget gets Google to look at the page. Whether the page is kept belongs to a different assessment entirely.
That distinction also decides where money goes. Adding server capacity to lift crawl requests, on a site whose pages are stalling on the content side, spends budget and moves no metric. The same spend on consolidating duplicates would improve crawl distribution and indexing rate together. Picking the correct link is what determines whether the spend returns anything.
The Definition That Stretches Into Indexing
A common definition describes crawl budget as the number of pages Googlebot crawls and indexes in a period. It folds two decisions into one sentence and erases the very boundary this article draws.
The cost shows up in diagnosis. Once the two are treated as one thing, unindexed pages send everyone to the crawling side by reflex: resubmit the sitemap, add internal links, edit robots.txt. When the problem sits on the content side, none of those touch it.
Take a concrete case. You run a catalog of 40,000 products and 12,000 product pages are missing from the index. The merged definition produces one diagnosis: not enough crawl budget. The separated definition opens the Page Indexing report first and asks which bucket those 12,000 pages sit in. In the Discovered bucket, there is real crawling work to do. In the Crawled bucket, Google has already seen them and declined, and touching the budget changes nothing. The two action lists share not a single item.
The Two "Not Indexed" States in Search Console
The clearest form of the distinction sits in the Page Indexing report. Two states that look alike point at two different problems.
Discovered Currently Not Indexed
Google knows the URL and has not crawled it yet.
The problem is on the crawling side. The URL is in Google's queue; its turn has not arrived, or Google did not judge it a priority. When this group swells, the places to look are known: crawl budget, internal linking, site architecture, and server response time.
On large sites a swelling Discovered bucket usually signals an inventory problem. When Google knows thousands of low-value URLs, priority pages wait behind them.
Crawled Currently Not Indexed
Google fetched the URL and chose not to store it.
Nothing on the crawling side will help. The page was seen. Resubmitting a sitemap, adding internal links, or growing the crawl budget leaves the outcome unchanged, because Google reaching the page was never the obstacle.
The place to look is the content: whether it duplicates or near-duplicates something else, whether canonical signals are consistent, and whether the page carries value on its own. Thin pages generated from a template accumulate in this bucket.
Which State Sends You Where
| Search Console state | What happened | Which side | Where to look first |
|---|---|---|---|
| Discovered currently not indexed | Known, not crawled | Crawling | Crawl stats, internal linking, server response time |
| Crawled currently not indexed | Crawled, not stored | Content | Duplicate check, canonical, page value |
| Indexed but not ranking | Stored, not selected | Relevance and competition | Search intent, depth, competitive level |
The third row is not a state Search Console displays. It still belongs in the sequence, because the possibility teams overlook most often is that the page is indexed and simply not good enough for the query.
Which Link Each Directive Touches
Three directives act on three different links. Substituting one for another is the most expensive consequence of not knowing this distinction.
robots.txt Stops Crawling, Not Indexing
A robots.txt disallow stops Googlebot from requesting the URL. What it does not stop is indexing. Google can index a URL it has never fetched, on the strength of external signals alone.
There is a further cost. Google cannot read the canonical tag on a page it cannot fetch. Closing duplicates with robots.txt freezes the duplicate problem instead of resolving it.
You can watch the result in the search results. A blocked URL that external links point at may get listed with no description at all, because Google could read neither the title nor the content. If you want a page out of the index and out of crawling, the order runs: lift the block, let Google see the noindex, wait for the page to drop out, then reapply the block.
noindex Stops Indexing, Not Crawling
A noindex directive lives in the page's HTML or its HTTP header. Googlebot has to fetch the page to read it. A page carrying noindex therefore keeps consuming crawl requests.
The conclusion follows directly: noindex cannot serve as a crawl-saving tool. It controls indexing, and that is all it controls.
404 and 410 Close the Chain
Returning the correct status code on a permanently removed page is a strong signal to Google not to crawl the URL again. The contrast with blocking matters: per Google's documentation, blocked URLs stay in the crawl queue considerably longer and return the moment the block is lifted.
Writing a deleted page into robots.txt is a worse decision than returning the right code.
Three Mistakes in Diagnosing the Chain
The distinction is clean in theory and gets lost in three places. All three share a root: reading a symptom and acting before identifying the link.
Reading a Single Number
"How many of my pages are indexed" tells you nothing on its own. What carries meaning is which bucket the missing ones sit in.
The Page Indexing report supplies that breakdown already. Reading the total without the breakdown is like taking a temperature and stopping there. The first screen to open is the state list rather than the headline figure.
Ignoring Time
A newly published page missing from the index is not evidence of a problem. The chain runs in sequence and every link takes time.
The right question is how long the page has been live against how long indexing normally takes on your site. Without that baseline, a single page's status cannot be interpreted. If new content on your site normally indexes in three days, raising an alarm on day two invites a wrong diagnosis.
Changing Several Things at Once
The costliest mistake. Pages are missing, so within one week the sitemap gets resubmitted, internal links get added, robots.txt gets edited, content gets expanded, and the server gets upgraded.
If things improve next month, nobody knows which change did it. If they do not, nobody knows which change failed. The practical value of the chain model lands exactly here: identify the link, touch only that link, and learning becomes possible.
Sequence it this way. Identify the bucket in the Page Indexing report. For the Discovered bucket, make one change on the crawling side and wait. For the Crawled bucket, make one change on the content side and wait. Keep the waiting period no shorter than your site's normal indexing time.
What More Crawling Does and Does Not Promise
Correlation and Causation
Organic traffic often rises in a month when the crawling graph rises. Two lines moving together says nothing about one causing the other.
Other things usually happen in the same window: content ships, internal links get added, technical fixes land, the season turns. Without a controlled test, no one can say which variable contributed what. The honest frame for crawl data is that an increase signals Google allocating more resource to the site.
The same discipline applies in reporting. Putting the crawl graph beside the organic traffic graph and drawing an arrow produces narrative, not measurement. Present the relationship between the two curves as a hypothesis and state plainly what was never tested.
The Target Is Distribution, Not Volume
Framing the goal as more crawling puts the wrong metric under the microscope. Even with the total budget unchanged, Google reaching the right pages first is what gets new and updated content seen sooner.
The question to ask: does the distribution of crawl requests match the site's order of priority? A rise in crawling that flows into parameter URLs is not good news for being a rise.
Frequently Asked Questions
Crawling vs indexing: which one is my problem?
Open the Page Indexing report and read the state, not the total. Discovered currently not indexed means Google has not crawled the page yet, so the problem sits on the crawling side. Crawled currently not indexed means Google fetched it and declined to store it, so the problem sits in the content. The two share no fixes.
What does crawled but not indexed mean?
Google fetched the page and chose not to store it in the index. Nothing on the crawling side will change that, because Google already saw the page. The places to look are duplication, canonical consistency, and whether the page carries value on its own.
How do you fix crawled currently not indexed?
Start by asking whether the page carries genuinely distinct value. Pages generated from a template as variations of one another collect in this bucket. Work in order: merge near-duplicates into one stronger page, confirm canonical tags on what remains are consistent, then either expand or remove thin pages. Resubmitting a sitemap does not move this bucket.
How do you fix discovered currently not indexed?
Google knows the URL and has not crawled it. The fix belongs to the crawling side: add internal links from strong pages, confirm the URL appears in your sitemap, check server response time, and shrink the low-value URL inventory. On large sites a swelling bucket here usually signals an inventory problem.
Does crawl budget affect indexing?
Crawl budget acts on the first link of the chain only. Enlarging it does not raise the indexing rate, because indexing is a separate decision. The budget gets Google to see the page; whether the page is kept is judged elsewhere.
My page is indexed but not ranking, why?
Being in the index does not earn selection for a query. Ranking assesses the page against its competitors for that specific query. The places to look are whether search intent is met, how deep the content goes, and how competitive the query is. Crawling and indexing fixes have no effect at this stage.
Sources and Further Reading
- Google Search Central, "Crawl Budget Management | Google Crawling Infrastructure", https://developers.google.com/crawling/docs/crawl-budget (accessed August 11, 2026)
- Google Search Console Help, "Page Indexing report", https://support.google.com/webmasters/answer/7440203 (accessed August 11, 2026)