Summary: Blocking /_next/ in a Next.js robots.txt file saves no crawl budget. The JavaScript under /_next/static/ is what Googlebot runs to render the page, and Google does not render JavaScript from blocked files. The page is crawled, and Google indexes an empty version of it.
Server logs on a Next.js site fill up with lines like /_next/static/immutable/chunks/turbopack-02sl120el9-1r.js. They return no HTML, there are a lot of them, and the first instinct is a Next.js robots.txt rule that shuts them out. The worry behind that instinct is real. The target is wrong.
The sections below cover what actually sits under /_next/, which mechanism the block breaks, and where the loss surfaces. The last two are for anyone who has already written that line.
What Actually Sits Under /_next/
Disallow: /_next/ is one line, and it closes three separate behaviors at once. Their correct answers differ, so a single line is wrong for at least two of them.
/_next/static/: Build Output With a Fingerprint
Compiled JavaScript, CSS, and font files live here. According to the Next.js documentation, the framework writes Cache-Control: public, max-age=31536000, immutable on these assets, and the header cannot be overridden. The filenames carry a SHA hash, so a change in content changes the name.
That hash is what makes the file permanently cacheable. The one-year max-age is only the declaration of it. The distinction matters two sections down, because only the first of the two works on Googlebot's side.
/_next/image: Images Generated at Request Time
Optimization through next/image runs at request time rather than during the build, which the same documentation states directly. The image tag on the page points at a parameterized endpoint, not at a file in the build output.
A rule that closes everything under /_next/ therefore closes the image path too. Google's robots.txt introduction covers the case. Images, videos, and PDFs embedded in a blocked page fall out of crawling too. On a site earning traffic from Google Images, that loss never appears in the HTML reports.
/_next/data/: The Navigation Data Request
In Pages Router projects, a next/link transition sends a separate request to the server, where getServerSideProps runs. On a first load the data already sits inside the HTML. The extra request exists only for client-side transitions.
Googlebot does not click its way around a site; it requests every URL the way a first load does. Blocking that path therefore carries the least risk of the three. Its name is not in the Next.js documentation, so confirm it in your own build.
Read Your Own Build Before You Block It
The three paths above belong to the framework. Which of them your pages actually lean on is a fact about your project. Two Next.js sites lose different amounts to the same Disallow line.
What decides it is where the text on the page comes from. Server Components, static generation, request-time rendering and incremental regeneration all put the content into the server's first response. Client-side rendering does not: the first response is a shell, and the JavaScript under /_next/static/ is what fills it. Only that last case makes a page's visibility depend on robots.txt.
The check costs nothing and needs no Search Console. Open view-source: on the page in a browser and look for the text. Text in the source means the chunk carries no content Google needs. Text missing means the chunk is the page, and blocking it empties the page.
A 'use client' directive written where it was not needed costs more visibility than any robots.txt line gives back. Write the disallow rule once you know which of your pages sit in the second case. Written from the shape of the path alone, it is written blind.
Google Does Not Render Blocked Files
Google's JavaScript guide splits the work into three phases: crawling, rendering, and indexing. The common assumption is that robots.txt governs only the first one. The documentation says otherwise: "Google Search won't render JavaScript from blocked files or on blocked pages."
Two scenarios hide in that sentence. A blocked page is not rendered, which surprises nobody. A blocked file is not rendered either, and that half gets missed. The page can be perfectly crawlable and still come out empty if the JavaScript it needs is disallowed.
Crawled but Empty
The same guide gives the indexing condition: content that is not visible in the rendered HTML cannot be indexed. Googlebot rendering runs on whatever files it was allowed to fetch, so cutting the input shrinks the output.
What you get is a technical cause of the gap between crawling and indexing. The server log looks healthy. Googlebot arrived, took a 200, and left. On Google's side the page has a body nobody could fill.
The Second Loss: Links
Googlebot parses the response for URLs in href attributes and adds what it finds to the crawl queue. Where navigation or listing links enter the DOM only after rendering, no rendering means no discovery.
Internal links are both a discovery path and a priority signal. If the product links on a category page depend on rendering, the block turns those products into orphans in practice.
The Bot You Blocked Probably Was Not Googlebot
The robots.ts template that circulates in published Next.js SEO guides tends to have one shape. A * group carrying a long disallow list with /_next/ in it, and a separate Googlebot group underneath.
User-agent: *
Disallow: /_next/
Disallow: /api/
User-agent: Googlebot
Disallow: /admin/
Disallow: /api/Whoever writes that file believes /_next/ is now closed to everyone. Google's specification says otherwise: "Only one group is valid for a particular crawler. … Other groups are ignored." User agent specific groups and global groups are not combined.
Googlebot therefore reads only its own group, and /_next/ is not in it. The block lands instead on Bingbot, on Applebot, and on every AI crawler that respects robots.txt.
At first glance that reads as good news, since Google is untouched. The real problem is that nothing tells you. Search Console stays clean, URL Inspection shows no blocked resource, and Page indexing does not move.
The loss accumulates where you are not looking. Pages hollow out in Bing's index, and the crawlers feeding products like ChatGPT and Perplexity cannot render them. Group order offers no protection either, because the documentation states that order is irrelevant.
Reading the file for the string /_next/ is not the check. The check is which user agent group the line sits in. Once a Googlebot group exists, nothing written under * applies to Googlebot.
Where the Traffic Loss Shows Up
The cost does not land in one place at one time, and knowing where to look is what separates a diagnosis from an unexplained decline.
Page Indexing: Two Rows That Mean Different Things
Two rows in the Page indexing report move under this scenario, and they get confused with each other.
"Indexed, though blocked by robots.txt" applies to the blocked URL itself. Google's documentation treats it as a warning: the page was indexed without being crawled, using information from the pages linking to it. The snippet shown in search results will probably be very limited.
"Crawled - currently not indexed" is the row to watch for pages that depend on a blocked resource. Google's definition is short: the page was crawled but not indexed, and it may or may not be indexed later. No cause list accompanies that row.
A page whose render input was cut belongs in that bucket by mechanism. Google makes the connection nowhere, so read it as Beaked's analysis.
URL Inspection Is Where You See It
Inspect a URL in Search Console and open View crawled page > More info. You get the list of loaded resources, the JavaScript console output, and the rendered DOM. On a live test the same panel sits under View tested page > More info.
The indexed version says what Google knows now; the live test says what Google would see today. Check the live test after a fix.
A resource marked as blocked there puts the problem in robots.txt. A resource missing from the list entirely may have another explanation. Google's guidance notes that the Web Rendering Service may skip resources that do not contribute to essential page content.
Core Web Vitals Does Not Move
A widely repeated claim, and a wrong one. The Core Web Vitals report rests on real world usage data, and the documentation describes all three metrics as measured by actual user data.
A visitor's browser never reads robots.txt. Blocked JavaScript is served to that visitor normally, so Largest Contentful Paint (LCP) and Interaction to Next Paint (INP) stay where they were.
One narrow link survives. Only indexed URLs appear in the report, so a page dropping out of the index drops out of the report. What breaks is its representation, not its performance.
The Real Crawl Budget Loss Is Not JavaScript
Most Next.js crawl budget advice aims at the wrong target. The files under /_next/static/ are cheap, and the cache header is not the reason.
Google's own JavaScript troubleshooting guide states the behavior plainly. Googlebot caches aggressively to reduce network requests, and the Web Rendering Service may ignore caching headers. Any argument resting on the one-year immutable header is therefore incomplete.
The filename does the work. Google recommends content fingerprinting in the same document, and Next.js already ships it. While the name holds, the file is the same file.
The expensive URLs are somewhere else. Filter combinations, sort parameters, calendar templates, and pagination chains turn finite content into an unbounded URL space. Every one of those URLs returns crawlable HTML. A Disallow line earns its place there, not in the build output.
Two traps sit in that line. Path matching supports exactly two wildcards, * and $, so a widely copied rule like Disallow: /*?page=[2-9]* is read as literal text and blocks nothing.
The second trap is canonicalization. Canonicalizing a parameter URL while also blocking it sets the two tools against each other. Google's guidance is explicit: robots.txt is not a canonicalization mechanism, because blocked URLs can still be indexed without their content.
The distinction runs back to the question at the center of crawl budget management. The goal is not more crawling but the same crawling pointed at the right pages. Most Next.js SEO blocking mistakes skip that question.
How to Write a Next.js robots.txt
The correct pattern for a Next.js robots.txt file is block plus exception, and the source for it is not an opinion. The opening example in Google's robots.txt specification does exactly this. It closes a .css and .js directory to all crawlers and opens it to Googlebot. The reason sits in a comment line: Google needs those files for rendering.
Rule Precedence: The Longer Path Wins
A common assumption holds that robots.txt is read top to bottom and the last matching line wins. Google works differently. The specification states that crawlers apply the most specific rule, measured by the length of the rule path. Conflicting rules of equal length resolve to the least restrictive one.
In practice the exception does not depend on line order:
User-agent: *
Disallow: /api/
Disallow: /_next/data/
Allow: /_next/static/
Allow: /_next/image
Sitemap: https://example.com/sitemap.xml/_next/static/ is a longer path than /_next/, so the Allow wins. Adding Disallow: /_next/ to the same file changes nothing.
Writing It in app/robots.ts
Next.js serves robots.txt from the root of the app directory, either as a static app/robots.txt or as an app/robots.ts function returning a MetadataRoute.Robots object. The generated form carries a behavior note that catches people out: the documentation describes robots.js as a Route Handler cached by default.
Google's own cache sits on top of that. The specification says Google caches robots.txt content for up to 24 hours. With two caches stacked, a corrected rule needs a deploy plus a refresh on Google's side. That gap is where "I fixed it and nothing changed" comes from.
If the Block Is Already Live
The order of the fix matters, because anyone checking in the wrong order concludes that the fix did not work.
Add the Allow exception and deploy. Then open the live robots.txt in a browser to confirm the line actually shipped, because the generated file may be cached. Then run the live test in URL Inspection rather than the indexed version. If the blocked entry is gone from the resource list, the fix holds on Google's side.
The second check is in the page source. Open view-source in a browser and see whether the content sits in the HTML or arrives only through JavaScript. Server-side rendered (SSR) and statically generated (SSG) content is already in the HTML and suffers less from the block. How much you lose depends on how much of the project runs on the client.
After that it is a waiting game. Blocked URLs return when the block lifts, but the schedule is not yours. A reindex request works for single URLs; a whole site means waiting.
Frequently Asked Questions
Should I block /_next/ in robots.txt?
No. /_next/static/ and /_next/image are needed for rendering and for image search. Make the decision path by path: /api/ and /_next/data/ are reasonable to close, the build output and the image endpoint are not.
What happens if I disallow /_next/static/?
Googlebot stops requesting those files, and Google does not render JavaScript from blocked files. Content rendered on the client, and internal links that appear only after rendering, become invisible to Google. How much you lose depends on how much of the project runs client-side.
Should I block /api/ in robots.txt?
Usually yes. API endpoints return no HTML and have no reason to be indexed. The exception is an endpoint that feeds page rendering, where a block does the same damage as blocking /_next/static/.
Should I give Googlebot its own group in robots.txt?
Only if that group is complete. Googlebot applies the single most specific group matching it and ignores * entirely; the two are never combined. Opening a Googlebot group while assuming the * rules still apply is the most common silent mistake here.
How long until Google picks up a Next.js robots.txt change?
Google caches robots.txt content for up to 24 hours, and a file generated by app/robots.ts is cached on the build side too. The change needs a deploy first and a refresh on Google's side second. Opening the live file in a browser tells you which cache is holding you up.
Sources and Further Reading
- Google Search Central, "Understand the JavaScript SEO basics" (accessed September 9, 2026)
- Google Search Central, "How Google interprets the robots.txt specification" (accessed September 9, 2026)
- Google Search Central, "Introduction to robots.txt" (accessed September 9, 2026)
- Google Search Central, "Fix Search-related JavaScript problems" (accessed September 9, 2026)
- Google Search Central, "How to specify a canonical URL" (accessed September 9, 2026)
- Google Search Console Help, "URL Inspection tool" (accessed September 9, 2026)
- Google Search Console Help, "Page indexing report" (accessed September 9, 2026)
- Google Search Console Help, "Core Web Vitals report" (accessed September 9, 2026)
- Next.js documentation, "How to self-host your Next.js application" (accessed September 9, 2026)
- Next.js documentation, "Metadata Files: robots.txt" (accessed September 9, 2026)
- Next.js documentation, "getServerSideProps" (accessed September 9, 2026)