What is Claude-SearchBot and should you block it?

Claude-SearchBot is the crawler that builds the index behind Claude's web search. Block it and Claude cannot cite your pages.

What Claude-SearchBot does

Anthropic says Claude-SearchBot "navigates the web to improve search result quality for users" and "analyzes online content specifically to enhance the relevance and accuracy of search responses". It is separate from ClaudeBot, the training crawler.

On robots.txt: Claude-SearchBot respects robots.txt "do not crawl" signals, per Anthropic's documentation.

What blocking it costs

Block Claude-SearchBot and Claude's web search cannot index your pages, so it will not quote them, will not link to them, and will not send you the visit. The cost never shows up in analytics, because you never see the visits that did not happen. If your site sells anything, or exists to be found, this is the crawler to leave alone.

If what you actually want is to keep your text out of model training, block ClaudeBot instead. That is a separate crawler with a separate robots.txt group, and blocking it has no effect on Claude's web search.

How many of the top 5,000 sites block it

In my September 2026 census of the Tranco top 5,000, 2,778 sites returned a robots.txt. 267 of them block Claude-SearchBot (9.6%), which makes it the 17th most-blocked of the 20 crawlers I checked. 219 files mention it by name; the rest of the mentions either allow it explicitly or restrict only part of the site.

For comparison, 393 sites block at least one of the three AI search crawlers, and 305 of those still allow Googlebot: they are visible in Google and invisible in AI answers, usually by accident. The full census, with every crawler, is here.

The robots.txt lines

To make sure it is allowed even when your catch-all disallows everything, give it its own group. A crawler follows the most specific group that names it, so this wins over the catch-all:

User-agent: Claude-SearchBot
Allow: /

User-agent: *
Disallow: /

To block it (you probably do not want this), the two lines are:

User-agent: Claude-SearchBot
Disallow: /

How to verify a request really came from Claude-SearchBot

Anyone can put the string Claude-SearchBot in a request. Anthropic publishes the addresses it uses at claude.com/crawling/bots.json. A request from outside those ranges is an impostor, and impostors using well-known crawler names are common enough that blocking by name alone catches the honest crawler and misses the rest.

Source: Anthropic's crawler documentation. Census: the September 2026 robots.txt census, raw data at /data/census.json.

Other Anthropic agents: Claude-User, ClaudeBot. All 20 crawlers. Check your own robots.txt against all of them with the checker.