Google AI Overviews: what actually controls whether you appear

Google-Extended manages Gemini model training, and the token that decides whether your pages can turn up in an AI Overview is Googlebot.

The two Google tokens people confuse

Googlebot is the crawler that builds the Google Search index. Google-Extended is a separate product token whose only job is to manage whether your content helps improve Gemini Apps and the Vertex AI generative APIs. In Google's own documentation, Google-Extended has no crawler behind it and no user agent string of its own. Google states plainly that it has no bearing on a site's inclusion in Google Search and no bearing on ranking.

AI Overviews are produced inside Google Search and are assembled from pages that are already in the search index. Googlebot is what puts pages in that index. The robots.txt line that governs your eligibility for AI Overviews is therefore the Googlebot line, which is the same line that governs whether you can appear as an ordinary blue link.

User-agent: Google-Extended
Disallow: /
# Gemini training opt-out. No effect on Search or AI Overviews.

User-agent: Googlebot
Disallow: /
# Removes you from Google Search entirely, AI Overviews included.

Almost nobody blocks the token that matters

I ran a robots.txt census across the Tranco top 5,000 on 2026-09-07. I checked 4,999 sites and 2,771 of them served a robots.txt file. Every percentage here uses those 2,771 as the base.

• GPTBot, OpenAI training: 535 sites, 19.31%
• ClaudeBot, Anthropic training: 516 sites, 18.62%
• Google-Extended, Gemini training: 459 sites, 16.56%
• bingbot, Bing search: 95 sites, 3.43%
• Googlebot, Google Search and AI Overviews: 77 sites, 2.78%

More than five times as many sites block Google-Extended as block Googlebot. Blocking Google-Extended is a sensible thing to do if the aim is to stay out of Gemini's training data. Blocking it leaves AI Overviews completely untouched. The per-crawler breakdown is in the 5,000-site census, and the crawler reference has a page for each token setting out what it actually feeds.

OpenAI splits the same way

GPTBot is OpenAI's training crawler. OAI-SearchBot is the one behind ChatGPT's citations, so blocking GPTBot does not take you out of ChatGPT Search. 535 sites block GPTBot and 241 block OAI-SearchBot. 238 block both together, which drops them from ChatGPT citations whether or not that was the plan. 297 block GPTBot alone, which is the training opt-out done correctly. I go through that decision in should I block GPTBot, and does ChatGPT see my website covers how to test what ChatGPT can actually retrieve from your pages.

Across the AI search crawlers, 359 sites block PerplexityBot and 250 block Claude-SearchBot. 296 sites block at least one AI search crawler while still allowing Googlebot, so they stay in Google Search and in AI Overviews while dropping out of the engine they blocked. Only 73 block an AI search crawler and Googlebot. Named sites are listed on who blocks AI search, and every site in the census that blocks one is on the full list.

Two ways to actually opt out, and how they differ

There are two exceptions to the Googlebot rule above, and they need stating separately since they work differently. The first has been around for a while and is blunt. The second is new, and precise.

The snippet controls

Google's preview controls limit how much text Google may show from a page. The robots meta value nosnippet suppresses the text snippet for the whole page. max-snippet:[number] caps the snippet at a character count, and max-snippet:0 behaves the same way as nosnippet. The data-nosnippet attribute applies the restriction to a marked section instead of the whole document.

<meta name="robots" content="nosnippet">
<meta name="robots" content="max-snippet:0">
<p>Shown normally. <span data-nosnippet>Withheld from previews.</span></p>

Google's guidance for its AI experiences says these preview controls also apply to AI Overviews and AI Mode, so text you have made ineligible for a snippet is not used in them. Your page still stays in the index and can be listed as a result, because these controls restrict previews and leave indexing alone. The restriction is also blunt, since the setting that withholds your text from an AI Overview withholds it from your ordinary search snippet as well. I would expect that to cost most publishers more traffic than it saves. I would not describe it as a clean AI Overviews opt-out, and I would not deploy it on a site that depends on click-through.

The Search Console toggle, worldwide since August 31, 2026

Google built a dedicated control for this: a toggle in Search Console that lets a site opt out of its generative AI Search features by name. It did not appear everywhere at once. Google first tested it with a limited set of sites starting June 2026, then, in Google's own words, "as of August 31, 2026, we've rolled out this control to all websites worldwide" - so every property now has it, but it has only been universal for a few days as of this writing. In Google's own words on what it does, "website owners can decide if they want their site to appear in and help ground responses in our generative AI Search features (like AI Overviews, AI Mode or AI Overviews in Discover)." Google is explicit that this does not reach into ordinary Search: "sites that opt out will not receive traffic or impressions from our generative AI features. This control will not be used as a ranking signal for search results outside of these generative AI Search features."

That is the thing the snippet controls above cannot do: leave AI Overviews, AI Mode, and AI Overviews in Discover, without touching your ordinary snippet or your ranking anywhere else in Search. Google describes it as an addition to its existing tools rather than a replacement for either of them, calling it a continuation of "our long history of designing tools, like snippet controls and Google-Extended." Google-Extended still only governs Gemini's training data. The snippet controls still exist and still work the same blunt way described above. This toggle is the one built specifically for the opt-out decision itself, and it lives in Search Console under the site's own settings, not in robots.txt or a meta tag.

What to do instead

• Decide crawler by crawler. Training and search are separate tokens at every vendor, and the company name on the token tells you very little about what it does.
• Check what your site currently serves. A robots.txt that returns 404, or a 200 with nothing in it, means crawling is allowed. 2,235 of the 4,999 sites I checked did not return one.
• Make the page readable without JavaScript. In a separate crawl of homepages around Tranco rank 10,000, I fetched 607 pages once with JavaScript switched off. 16.0% returned under 250 characters of text and 58.0% carried no JSON-LD structured data.
• Put the answer near the top in plain sentences that survive being quoted out of context. How to rank in AI search covers what I have seen hold up.
• Treat llms.txt as optional. 385 of the 4,999 sites had a valid one, 7.7%. Google says outright that it does not use llms.txt for Search or its AI features; the other engines have not confirmed anything either way. It is cheap to add regardless, and it will not fix anything on its own. The llms.txt generator will build and validate one.

Checking your own site

The free checker reads your robots.txt and reports what each rule actually affects, separating the training tokens from the ones that decide citations. If it shows Google-Extended blocked and Googlebot allowed, your AI Overviews exposure is unchanged and your Gemini training opt-out is in place. That is a defensible setup, and it is worth knowing that it is the setup you have. It cannot show you the Search Console toggle above, since that setting lives inside your Search Console account and is not visible in a public robots.txt fetch; check that one directly in Search Console if you want to know whether it is on. The rest of the guides sit on the guide index.