What is ClaudeBot and should you block it?
ClaudeBot is Anthropic's training crawler. Blocking it opts your pages out of model training and costs you nothing in search, as long as the file is written correctly.
What ClaudeBot does
Anthropic says ClaudeBot collects "web content that could potentially contribute to" the training of its models. Search and user fetches use different agents (Claude-SearchBot and Claude-User).
On robots.txt: ClaudeBot respects robots.txt "do not crawl" signals, per Anthropic's documentation.
What blocking it costs
It costs nothing in search, if the file is written correctly. Blocking ClaudeBot tells Anthropic not to use your pages for training. It does not remove you from any search product.
The way sites get this wrong is the catch-all. A robots.txt that names ClaudeBot and then has a catch-all group that disallows everything also blocks Claude-SearchBot, because a crawler that is not named falls into the catch-all. Anthropic's search crawler needs its own group to stay allowed.
How many of the top 5,000 sites block it
In my September 2026 census of the Tranco top 5,000, 2,778 sites returned a robots.txt. 542 of them block ClaudeBot (19.5%), which makes it the 4th most-blocked of the 20 crawlers I checked. 556 files mention it by name; the rest of the mentions either allow it explicitly or restrict only part of the site.
For comparison, 726 sites (26.1%) block at least one training crawler, against 393 (14.1%) that block at least one AI search crawler. Most sites that opt out of training are doing it deliberately; most sites that block AI search are not. The full census is here.
The robots.txt lines
To block training and keep search, name both. The explicit group for Claude-SearchBot is the part people leave out:
User-agent: ClaudeBot Disallow: / User-agent: Claude-SearchBot Allow: / User-agent: * Allow: /
How to verify a request really came from ClaudeBot
Anyone can put the string ClaudeBot in a request. Anthropic publishes the addresses it uses at claude.com/crawling/bots.json. A request from outside those ranges is an impostor, and impostors using well-known crawler names are common enough that blocking by name alone catches the honest crawler and misses the rest.
Source: Anthropic's crawler documentation. Census: the September 2026 robots.txt census, raw data at /data/census.json.
Other Anthropic agents: Claude-SearchBot, Claude-User. All 20 crawlers. Check your own robots.txt against all of them with the checker.