What is Google-Extended and should you block it?

Google-Extended is a robots.txt token from Google. It controls what Google may do with pages that Googlebot fetched; it does not fetch anything itself.

What Google-Extended does

Google says Google-Extended controls whether content "may be used for training future generations of Gemini models" and for grounding in Gemini Apps and Vertex AI. It is a robots.txt token only: "Google-Extended doesn't have a separate HTTP request user agent string. Crawling is done with existing Google user agent strings."

On robots.txt: Google-Extended is honoured as a robots.txt control token; the fetching itself is done by Googlebot and Google's other documented agents.

Google's wording on the part people worry about: "Google-Extended does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search." AI Overviews and AI Mode are Search features served from the same index, so blocking this token does not take you out of them. The only way out of those is out of Search itself.

What blocking it costs

It costs nothing in search. Disallowing Google-Extended tells Google not to use your pages for the purposes the token covers. The pages are still fetched by Googlebot and still appear wherever Googlebot normally shows them.

How many of the top 5,000 sites block it

In my September 2026 census of the Tranco top 5,000, 2,778 sites returned a robots.txt. 480 of them block Google-Extended (17.3%), which makes it the 6th most-blocked of the 20 crawlers I checked. 516 files mention it by name; the rest of the mentions either allow it explicitly or restrict only part of the site.

For comparison, 726 sites (26.1%) block at least one training crawler, against 393 (14.1%) that block at least one AI search crawler. Most sites that opt out of training are doing it deliberately; most sites that block AI search are not. The full census is here.

The robots.txt lines

To opt out of the uses the token covers, with no effect on search:

User-agent: Google-Extended
Disallow: /

How to verify a request really came from Google-Extended

Anyone can put a name in a user-agent string. Since Google-Extended is a token and Googlebot does the fetching, verify the fetcher: Google publishes the address ranges at developers.google.com/static/crawling/ipranges/common-crawlers.json. A request from outside them is not Googlebot.

Source: Google's crawler documentation. Census: the September 2026 robots.txt census, raw data at /data/census.json.

Other Google agents: Googlebot. All 20 crawlers. Check your own robots.txt against all of them with the checker.