The 20 AI crawlers, one page each
What each crawler is for, what blocking it costs, how many of the top 5,000 sites block it, the exact robots.txt lines, and how to verify it. Numbers from the September 2026 census of 2,771 sites with a robots.txt.
The short version: the three AI search crawlers decide whether you get cited, the on-demand fetchers act for one person at a time, and the training crawlers can be blocked at no cost to search. Most sites missing from AI answers blocked a search crawler while trying to block a training one, deliberately at large, recognizable sites more often than the rank alone would suggest, and by accident of the catch-all rule everywhere else. The breakdown is here.
Want your site counted in the October census re-run?
Leave your domain and address to be counted for the October census re-run. Results are posted on /data; nothing is emailed.
AI search crawlers (these decide whether you get cited)
| Crawler | Vendor | Sites blocking it | Share |
|---|---|---|---|
| PerplexityBot | Perplexity | 359 | 13.0% |
| Claude-SearchBot | Anthropic | 250 | 9.0% |
| OAI-SearchBot | OpenAI | 241 | 8.7% |
On-demand fetchers (one page, because a person asked)
| Crawler | Vendor | Sites blocking it | Share |
|---|---|---|---|
| ChatGPT-User | OpenAI | 303 | 10.9% |
| Perplexity-User | Perplexity | 259 | 9.3% |
| Claude-User | Anthropic | 253 | 9.1% |
Training crawlers (blocking these does not affect search)
| Crawler | Vendor | Sites blocking it | Share |
|---|---|---|---|
| CCBot | Common Crawl | 573 | 20.7% |
| Bytespider | ByteDance | 542 | 19.6% |
| GPTBot | OpenAI | 535 | 19.3% |
| ClaudeBot | Anthropic | 516 | 18.6% |
| meta-externalagent | Meta | 476 | 17.2% |
| Google-Extended | 459 | 16.6% | |
| Amazonbot | Amazon | 428 | 15.4% |
| Applebot-Extended | Apple | 413 | 14.9% |
| Diffbot | Diffbot | 377 | 13.6% |
| cohere-ai | Cohere | 376 | 13.6% |
| omgili | Webz.io | 376 | 13.6% |
| TimpiBot | Timpi | 312 | 11.3% |
Classic search crawlers
| Crawler | Vendor | Sites blocking it | Share |
|---|---|---|---|
| bingbot | Microsoft | 95 | 3.4% |
| Googlebot | 77 | 2.8% |
Where the numbers come from
Each page quotes the same census: 4999 of the Tranco top 5,000 fetched, 2,771 with a robots.txt, a crawler counted as blocked when the most specific group that applies to it disallows the site root. Method, tables and the raw JSON are on the data page. Vendor claims on each page link to the vendor's own documentation.