What is Googlebot and should you block it?
Googlebot is Google's search crawler. Blocking it takes you out of Google Search.
What Googlebot does
Googlebot is Google's search crawler. Everything Google Search shows comes from what it fetches, and that now includes AI Overviews and AI Mode, which Google serves from the same index and the same ranking systems.
On robots.txt: Googlebot respects robots.txt.
What blocking it costs
Blocking Googlebot removes you from Google Search, and with it from AI Overviews and AI Mode, which are Search features. If you want to stay in Search but out of Gemini training, the token for that is Google-Extended, which Google says has no effect on Search inclusion or ranking.
How many of the top 5,000 sites block it
In my September 2026 census of the Tranco top 5,000, 2,778 sites returned a robots.txt. 95 of them block Googlebot (3.4%), which makes it the 20th most-blocked of the 20 crawlers I checked. 390 files mention it by name; the rest of the mentions either allow it explicitly or restrict only part of the site.
Googlebot is blocked by 95 sites (3.4%) and bingbot by 118 (4.2%). Those are the floor: a site that blocks a classic search crawler at the root has usually decided not to be indexed at all. The full census is here.
The robots.txt lines
You almost certainly do not want to block it. If you must keep it out of one directory:
User-agent: Googlebot Disallow: /private/
How to verify a request really came from Googlebot
Anyone can put the string Googlebot in a request. Google publishes the addresses it uses at developers.google.com/static/crawling/ipranges/common-crawlers.json. A request from outside those ranges is an impostor, and impostors using well-known crawler names are common enough that blocking by name alone catches the honest crawler and misses the rest.
Source: Google's crawler documentation. Census: the September 2026 robots.txt census, raw data at /data/census.json.
Other Google agents: Google-Extended. All 20 crawlers. Check your own robots.txt against all of them with the checker.