Should you block GPTBot?

Blocking GPTBot opts your site out of OpenAI's model training, and that is a reasonable thing to want. It costs nothing in search visibility as long as the robots.txt is written correctly. In my census, nearly half the sites that block it got the file wrong.

What GPTBot is

GPTBot is the crawler OpenAI uses to collect web content that may be used to train its models. OpenAI documents it under that name and says it respects robots.txt. Disallowing it tells OpenAI not to fetch your pages for training from now on. It does not reach back into models already trained. For a site whose text is the product, that is a sensible choice. In my September census of the top 5,000 sites, 2,778 served a robots.txt and 560 of those block GPTBot (20.16%), which makes it one of the most blocked crawlers on the list.

What blocking it does not control

OpenAI runs three crawlers, each with its own name in robots.txt. GPTBot collects training data. OAI-SearchBot builds the index behind ChatGPT search and decides whether ChatGPT can cite you. ChatGPT-User fetches a page live when someone asks about it. A rule for GPTBot says nothing to the other two.

Google is unaffected because Search and AI Overviews are served from Googlebot's index, and Google's training opt-out is a separate token, Google-Extended. Bing is unaffected because bingbot is a different crawler from a different company. So the short answer to "does blocking GPTBot hurt SEO" is that by itself it does not. It only reaches OpenAI training, provided the rest of the file catches nothing else.

Where the real cost comes from

A crawler follows the most specific robots.txt group that names it. If nothing names it, the catch-all group applies. The mistake is a file that names GPTBot and then closes the catch-all, like this:

User-agent: GPTBot
Disallow: /

User-agent: *
Disallow: /

Nothing in that file names OAI-SearchBot, so it follows the catch-all and is blocked from ChatGPT search. Files like this usually carry Allow groups for Googlebot and bingbot higher up, which is how a site ranks normally in Google while being absent from ChatGPT. My census found 305 sites in that position.

Among the 560 sites that block GPTBot, 258 also block OAI-SearchBot. That is 46.1%. Most of those files read like a training opt-out that never mentions search, with the search block coming from the catch-all rather than from a decision. The other 302 block GPTBot and leave OAI-SearchBot allowed, which is what their authors meant. The per-crawler numbers are on the data page.

The correct file for opting out of training while staying in ChatGPT search is three groups:

User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: *
Allow: /

The explicit OAI-SearchBot group is what matters. Without it, the crawler inherits whatever the catch-all says.

Blocking at the firewall instead

robots.txt is a request. OpenAI documents that GPTBot honors it, but a site that wants enforcement blocks at the firewall or in a bot-management product. Those rules are usually written against the string "OpenAI" or against anything that does not look like a browser, and they catch OAI-SearchBot and ChatGPT-User along with GPTBot. If you go this route, match on the GPTBot user agent alone, or use the IP list OpenAI publishes for GPTBot, which is separate from the lists for its search and user-action crawlers. Then check your logs afterward for OAI-SearchBot.

Who should block it

Sites whose value is the text itself. If you publish fiction, paid research, a reference work, or a forum whose archive is the product, a model trained on that text can substitute for a visit, and opting out is reasonable.

Most business sites should not bother. If your pages describe a service, a store, a product or a practice, the text is there to be found, and you want models to know what you sell and where. If the file is written wrong, blocking removes you from a channel that sends traffic while protecting nothing. The opt-out also covers only future crawling, so a site that has been in models for years gains little.

Check it in one minute

Paste your address into the checker on the home page. It reads your robots.txt as each crawler would, including the catch-all logic, and tells you which ones are blocked. It also fetches your homepage without JavaScript and reports how much readable text a crawler gets. To do it by hand, open yourdomain.com/robots.txt and find the group that names OAI-SearchBot. If there is none, read the User-agent: * group. Whatever it says is what ChatGPT search gets.