What is GPTBot and should you block it?
GPTBot is OpenAI's training crawler. Blocking it opts your pages out of model training and costs you nothing in search, as long as the file is written correctly.
What GPTBot does
OpenAI says GPTBot "is used to crawl content that may be used in training our generative AI foundation models" and that disallowing it "indicates a site's content should not be used in training generative AI foundation models". It has no role in ChatGPT search; that job belongs to OAI-SearchBot.
On robots.txt: GPTBot respects robots.txt, per OpenAI's documentation.
What blocking it costs
It costs nothing in search, if the file is written correctly. Blocking GPTBot tells OpenAI not to use your pages for training. It does not remove you from any search product.
The way sites get this wrong is the catch-all. A robots.txt that names GPTBot and then has a catch-all group that disallows everything also blocks OAI-SearchBot, because a crawler that is not named falls into the catch-all. OpenAI's search crawler needs its own group to stay allowed.
How many of the top 5,000 sites block it
In my September 2026 census of the Tranco top 5,000, 2,778 sites returned a robots.txt. 560 of them block GPTBot (20.2%), which makes it the 3rd most-blocked of the 20 crawlers I checked. 633 files mention it by name; the rest of the mentions either allow it explicitly or restrict only part of the site.
Of the 560 sites that block GPTBot, 258 (46.1%) also block OAI-SearchBot, and so are out of ChatGPT search. 302 block GPTBot alone, which is the version that costs nothing. The 46% is the number this whole site was built around, because it is the most common way a site loses AI search visibility without anyone deciding to.
The robots.txt lines
To block training and keep search, name both. The explicit group for OAI-SearchBot is the part people leave out:
User-agent: GPTBot Disallow: / User-agent: OAI-SearchBot Allow: / User-agent: * Allow: /
How to verify a request really came from GPTBot
Anyone can put the string GPTBot in a request. OpenAI publishes the addresses it uses at openai.com/gptbot.json. A request from outside those ranges is an impostor, and impostors using well-known crawler names are common enough that blocking by name alone catches the honest crawler and misses the rest.
Source: OpenAI's crawler documentation. Census: the September 2026 robots.txt census, raw data at /data/census.json.
Still deciding? Should you block GPTBot? walks through what it costs and what it does not, including the mistake above.
Other OpenAI agents: OAI-SearchBot, ChatGPT-User. All 20 crawlers. Check your own robots.txt against all of them with the checker.