What is Claude-User and should you block it?
Claude-User is not a crawler in the usual sense. It fetches one page because a person using Claude asked about it.
Want your site counted in the October census re-run?
Leave your domain and address to be counted for the October census re-run. Results are posted on /data; nothing is emailed.
What Claude-User does
Anthropic says that "when individuals ask questions to Claude, it may access websites using a Claude-User agent". It fetches one page for one person who asked.
On robots.txt: Claude-User respects robots.txt, per Anthropic's documentation.
What blocking it costs
The person who asked gets nothing from your page, or gets the answer from a competitor's page instead. There is no index to fall out of, and no training to opt out of; the only thing you block is a reader who was already interested enough to ask.
How many of the top 5,000 sites block it
In my September 2026 census of the Tranco top 5,000, 2,771 sites returned a robots.txt. 253 of them block Claude-User (9.1%), which makes it the 16th most-blocked of the 20 crawlers I checked. 215 files mention it by name; the rest of the mentions either allow it explicitly or restrict only part of the site.
For comparison, 363 sites block at least one of the three on-demand fetchers. Claude-User is the one of the three whose vendor documents full robots.txt compliance, so a block on Claude-User actually does what the file says, unlike the other two. The full census is here.
The robots.txt lines
The line, which Anthropic's own documentation says this fetcher honours:
User-agent: Claude-User Disallow: /
How to verify a request really came from Claude-User
Anyone can put the string Claude-User in a request. Anthropic publishes the addresses it uses at claude.com/crawling/bots.json. A request from outside those ranges is an impostor, and impostors using well-known crawler names are common enough that blocking by name alone catches the honest crawler and misses the rest.
Source: Anthropic's crawler documentation. Census: the September 2026 robots.txt census, raw data at /data/census.json.
Other Anthropic agents: Claude-SearchBot, ClaudeBot. All 20 crawlers. Check your own robots.txt against all of them with the checker.