What is Claude-User and should you block it?

Claude-User is not a crawler in the usual sense. It fetches one page because a person using Claude asked about it.

What Claude-User does

Anthropic says that "when individuals ask questions to Claude, it may access websites using a Claude-User agent". It fetches one page for one person who asked.

On robots.txt: Claude-User respects robots.txt, per Anthropic's documentation.

What blocking it costs

The person who asked gets nothing from your page, or gets the answer from a competitor's page instead. There is no index to fall out of, and no training to opt out of; the only thing you block is a reader who was already interested enough to ask.

How many of the top 5,000 sites block it

In my September 2026 census of the Tranco top 5,000, 2,778 sites returned a robots.txt. 269 of them block Claude-User (9.7%), which makes it the 16th most-blocked of the 20 crawlers I checked. 218 files mention it by name; the rest of the mentions either allow it explicitly or restrict only part of the site.

For comparison, 384 sites block at least one of the three on-demand fetchers. Since two of the three vendors say robots.txt may not apply to these anyway, much of that blocking is on paper only. The full census is here.

The robots.txt lines

The line, for what it is worth given the vendor's note above:

User-agent: Claude-User
Disallow: /

How to verify a request really came from Claude-User

Anyone can put the string Claude-User in a request. Anthropic publishes the addresses it uses at claude.com/crawling/bots.json. A request from outside those ranges is an impostor, and impostors using well-known crawler names are common enough that blocking by name alone catches the honest crawler and misses the rest.

Source: Anthropic's crawler documentation. Census: the September 2026 robots.txt census, raw data at /data/census.json.

Other Anthropic agents: Claude-SearchBot, ClaudeBot. All 20 crawlers. Check your own robots.txt against all of them with the checker.