Google AI Overviews: what actually controls whether you appear

Google-Extended manages Gemini model training, and the token that decides whether your pages can turn up in an AI Overview is Googlebot.

The two Google tokens people confuse

Googlebot is the crawler that builds the Google Search index. Google-Extended is a separate product token whose only job is to manage whether your content helps improve Gemini Apps and the Vertex AI generative APIs. In Google's own documentation, Google-Extended has no crawler behind it and no user agent string of its own. Google states plainly that it has no bearing on a site's inclusion in Google Search and no bearing on ranking.

AI Overviews are produced inside Google Search and are assembled from pages that are already in the search index. Googlebot is what puts pages in that index. The robots.txt line that governs your eligibility for AI Overviews is therefore the Googlebot line, which is the same line that governs whether you can appear as an ordinary blue link.

User-agent: Google-Extended
Disallow: /
# Gemini training opt-out. No effect on Search or AI Overviews.

User-agent: Googlebot
Disallow: /
# Removes you from Google Search entirely, AI Overviews included.

Almost nobody blocks the token that matters

I ran a robots.txt census across the Tranco top 5,000 on 2026-09-07. I checked 4,999 sites and 2,771 of them served a robots.txt file. Every percentage here uses those 2,771 as the base.

• GPTBot, OpenAI training: 535 sites, 19.31%
• ClaudeBot, Anthropic training: 516 sites, 18.62%
• Google-Extended, Gemini training: 459 sites, 16.56%
• bingbot, Bing search: 95 sites, 3.43%
• Googlebot, Google Search and AI Overviews: 77 sites, 2.78%

More than five times as many sites block Google-Extended as block Googlebot. Blocking Google-Extended is a sensible thing to do if the aim is to stay out of Gemini's training data. Blocking it leaves AI Overviews completely untouched. The per-crawler breakdown is in the 5,000-site census, and the crawler reference has a page for each token setting out what it actually feeds.

OpenAI splits the same way

GPTBot is OpenAI's training crawler. OAI-SearchBot is the one behind ChatGPT's citations, so blocking GPTBot does not take you out of ChatGPT Search. 536 sites block GPTBot and 241 block OAI-SearchBot. 238 block both together, which drops them from ChatGPT citations whether or not that was the plan. 297 block GPTBot alone, which is the training opt-out done correctly. I go through that decision in should I block GPTBot, and does ChatGPT see my website covers how to test what ChatGPT can actually retrieve from your pages.

Across the AI search crawlers, 359 sites block PerplexityBot and 250 block Claude-SearchBot. 296 sites block at least one AI search crawler while still allowing Googlebot, so they stay in Google Search and in AI Overviews while dropping out of the engine they blocked. Only 73 block an AI search crawler and Googlebot. Named sites are listed on who blocks AI search, and every site in the census that blocks one is on the full list.

The snippet controls, described precisely

There is one partial exception, and it needs stating carefully. Google's preview controls limit how much text Google may show from a page. The robots meta value nosnippet suppresses the text snippet for the whole page. max-snippet:[number] caps the snippet at a character count, and max-snippet:0 behaves the same way as nosnippet. The data-nosnippet attribute applies the restriction to a marked section instead of the whole document.

<meta name="robots" content="nosnippet">
<meta name="robots" content="max-snippet:0">
<p>Shown normally. <span data-nosnippet>Withheld from previews.</span></p>

Google's guidance for its AI experiences says these preview controls also apply to AI Overviews and AI Mode, so text you have made ineligible for a snippet is not used in them. Your page still stays in the index and can be listed as a result, because these controls restrict previews and leave indexing alone. The restriction is also blunt, since the setting that withholds your text from an AI Overview withholds it from your ordinary search snippet as well. I would expect that to cost most publishers more traffic than it saves. I would not describe it as a clean AI Overviews opt-out, and I would not deploy it on a site that depends on click-through.

What to do instead

• Decide crawler by crawler. Training and search are separate tokens at every vendor, and the company name on the token tells you very little about what it does.
• Check what your site currently serves. A robots.txt that returns 404, or a 200 with nothing in it, means crawling is allowed. 2,235 of the 4,999 sites I checked did not return one.
• Make the page readable without JavaScript. In a separate crawl of homepages around Tranco rank 10,000, I fetched 607 pages once with JavaScript switched off. 16.0% returned under 250 characters of text and 58.0% carried no JSON-LD structured data.
• Put the answer near the top in plain sentences that survive being quoted out of context. How to rank in AI search covers what I have seen hold up.
• Treat llms.txt as optional. 373 of the 4,999 sites had a valid one, 7.46%. No major AI search engine has confirmed it as a ranking or citation input, so it is cheap to add. It will not fix anything on its own. The llms.txt generator will build and validate one.

Checking your own site

The free checker reads your robots.txt and reports what each rule actually affects, separating the training tokens from the ones that decide citations. If it shows Google-Extended blocked and Googlebot allowed, your AI Overviews exposure is unchanged and your Gemini training opt-out is in place. That is a defensible setup, and it is worth knowing that it is the setup you have. The rest of the guides sit on the guide index.