What is GPTBot and should you block it?
GPTBot is OpenAI's training crawler. Blocking it opts your pages out of model training and costs you nothing in search, as long as the file is written correctly.
Want your site counted in the October census re-run?
Leave your domain and address to be counted for the October census re-run. Results are posted on /data; nothing is emailed.
What GPTBot does
OpenAI says GPTBot "is used to crawl content that may be used in training our generative AI foundation models" and that disallowing it "indicates a site's content should not be used in training generative AI foundation models". It has no role in ChatGPT search; that job belongs to OAI-SearchBot.
On robots.txt: GPTBot respects robots.txt, per OpenAI's documentation.
What blocking it costs
It costs nothing in search, if the file is written correctly. Blocking GPTBot tells OpenAI not to use your pages for training. It does not remove you from any search product.
The way sites get this wrong is the catch-all. A robots.txt that names GPTBot and then has a catch-all group that disallows everything also blocks OAI-SearchBot, because a crawler that is not named falls into the catch-all. OpenAI's search crawler needs its own group to stay allowed.
How many of the top 5,000 sites block it
In my September 2026 census of the Tranco top 5,000, 2,771 sites returned a robots.txt. 535 of them block GPTBot (19.3%), which makes it the 3rd most-blocked of the 20 crawlers I checked. 630 files mention it by name; the rest of the mentions either allow it explicitly or restrict only part of the site.
Of the 535 sites that block GPTBot, 238 (44.5%) also block OAI-SearchBot, and so are out of ChatGPT search. 297 block GPTBot alone, which is the version that costs nothing. The 46% is the number this whole site was built around, because it is the most common way a site loses AI search visibility without anyone deciding to.
The robots.txt lines
To block training and keep search, name both. The explicit group for OAI-SearchBot is the part people leave out:
User-agent: GPTBot Disallow: / User-agent: OAI-SearchBot Allow: / User-agent: * Allow: /
The GPTBot user-agent string
OpenAI's documentation gives this string (read 2026-09-26). OpenAI gives it as an example, says the version number may change, and says that when it fetches robots.txt it may add a robots.txt marker to the string. robots.txt matches on the token GPTBot, not on the whole string, and anyone can send any string, so the address check below is what shows a request is real.
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot
How to verify a request really came from GPTBot
Anyone can put the string GPTBot in a request. OpenAI publishes the addresses it uses at openai.com/gptbot.json. A request from outside those ranges is an impostor, and impostors using well-known crawler names are common enough that blocking by name alone catches the honest crawler and misses the rest.
Source: OpenAI's crawler documentation. Census: the September 2026 robots.txt census, raw data at /data/census.json.
Other OpenAI agents: OAI-SearchBot, ChatGPT-User. All 20 crawlers. Check your own robots.txt against all of them with the checker.