Which of the top 5,000 sites block AI crawlers?

535 sites block GPTBot, the crawler OpenAI uses to gather training data. 238 of those, 44.5%, also block OAI-SearchBot, the separate crawler that indexes pages for ChatGPT search. The other 297 block GPTBot alone, which keeps them out of training and still lets ChatGPT cite them. OpenAI documents the two crawlers separately, and most of these sites otherwise welcome search engines, so for many of the 238 this looks like an accident rather than a decision. See the full named list and the citable/opted-out split.

296 sites block at least one AI search crawler and still allow Googlebot. Among the largest: facebook.com, instagram.com, twitter.com, amazon.com, x.com, tiktok.com, pinterest.com, yahoo.com, ebay.com, imdb.com, unsplash.com, quora.com. News publishers on the list include nytimes.com, cnn.com, theguardian.com, forbes.com, bbc.com, reuters.com, washingtonpost.com, wsj.com, bloomberg.com, cnbc.com. Big publishers often do this on purpose, for licensing reasons. For a smaller business it is nearly always unintended. Check your own site if you are not sure which crawlers you block.

A separate check on 31 August 2026 fetched the homepage of 607 sites ranked around 10,000 in the same list, the way a crawler that does not run JavaScript sees them. 79.2% had substantial readable text. 16.0% were effectively empty without JavaScript, under 250 characters of text. 4.8% were thin, under 800 characters. 58.0% had no JSON-LD structured data and 26.4% had no meta description. So roughly one site in five that lets AI crawlers in still gives them nothing to read. See the named invisible and thin sites.

On 19 September 2026 I requested /llms.txt from the same 5,000 domains. 385 of them, 7.7%, served a file that follows the proposed format (a markdown H1 first, at least one link). 2194 answered 404. 179 returned an HTML page instead, which is what a site does when it serves its app for any path, and 350 redirected somewhere else. 115 returned text that was not an llms.txt. 421 refused the request outright and 1246 could not be reached at all; both are counted, not treated as absent.

61 of the 385 also publish an llms-full.txt, and 335 include the blockquote summary the format asks for. The highest ranked sites with one: cloudflare.com, github.com, fastly.net, digicert.com, wordpress.org, adobe.com, opera.com, samsung.com, sentry.io, wordpress.com, unity3d.com, kaspersky.com. Every domain with a valid file is listed in the JSON below. For the question people search, whether you need an llms.txt to show up in ChatGPT: adoption is 7.7% across the 5,000 and 10.0% in the top 100, and none of the AI search engines has documented reading it for search, so treat it as cheap and optional. The generator and validator will make one if you want it.

Crawlmind, a similar AI-visibility tool, published its own AI-Readiness Report on 25 August 2026: the Tranco top 1,000 hostnames, 561 with a reachable homepage (the other 439 were CDN, DNS or cloud endpoints with nothing to test). Their finding: 24.6% of the 561 block at least one major AI crawler. This census, built the same month from a different, larger sample, finds 25.2% of its 2,771 sites with a robots.txt do the same. The two studies do not share an identical crawler list or an identical denominator (this count keeps every domain that served a robots.txt, including infrastructure hosts a reachable-homepage filter would drop), so treat the closeness as a rough cross-check, not an exact match - but two independent measurements landing within a point of each other is a real signal that neither number is a sampling fluke. Their full report has the methodology.

The two studies disagree more on llms.txt. Crawlmind found 14.3% of its 561 reachable top-1,000 sites publish a spec-valid llms.txt (16.8% publish something under that name, spec-valid or not). This census's own top-1,000 slice, all 1000 domains rather than only the reachable ones, finds 9.1%. Both sit well above this census's 5,000-site average of 7.7%, the same pattern the by-rank table above already shows: adoption is heaviest near the top of the ranking and falls off fast past it. Crawlmind's report reaches a similar conclusion from its own sector breakdown - SaaS and developer tools mostly leave crawlers alone, while publishers and social platforms block them aggressively.

On October 1, 2026 I repeated the census and the Cloudflare edge, agents and robots.txt-path checks described above. The headline numbers moved by single digits: 2,729 of the same 5,000 domains answered with a robots.txt this time, against 2,771 in September. OAI-SearchBot's count moved to 239 from 241, and PerplexityBot's to 354 from 359.

The Cloudflare edge panel, the sites currently served through Cloudflare that answered a normal Chrome request, still shows GPTBot and ClaudeBot refused while PerplexityBot is served on 322 of 411 of them, against 336 of 422 at the last check. The non-Cloudflare control group is unchanged at 17 of 269. Of four more crawlers probed on the same 468-site panel this round: CCBot was refused by 334 (342 last time), Amazonbot by 345 (354), Bytespider by 354 (362), meta-externalagent by 42 (43). A request for /robots.txt as the bot still reads a clean file on most sites that turn the bot away from the homepage: GPTBot was refused at the homepage by 347 panel sites and still served a normal robots.txt to 324 of them (330 of 351 last time).

Across the top 5,000, the same first-pass edge-refusal test found 50 sites this round, down from 56: 41 for PerplexityBot (48 before), 19 for OAI-SearchBot (20), 10 for Claude-SearchBot (9). The sites behind this count are not published by name, only the aggregate figures here and in the linked JSON. 25 of the 489 readable sites in the retired-block panel still carry a Content-Signal line, up from 20 of 491, and 9 still serve the old managed-block marker directly. Of the full 754-site panel, 25 changed fetch status (served, 404, unreachable or similar) between the two checks; with one fetch per host, that is as likely to be transient churn as a real change.

Full dated figures: /data/census_update_20261001.json. Method and limits are the same as the original write-up: one IP, one fetch per host per crawler, so a network-range rule and a transient blip both look the same as a real decision.

Domains come from the Tranco top list, a research ranking that averages several traffic lists. Each robots.txt was fetched once with my own identified user agent. I never impersonate another crawler. A crawler counts as blocked when the most specific robots.txt group that applies to it disallows the site root. If no group names the crawler, the catch-all group applies, so a site that disallows everyone counts as blocking every crawler. The "named in robots.txt" column counts files that mention the crawler at all, whether to allow or block it.

Percentages use the 2,771 sites that served a robots.txt as the base. Sites that returned an error, a redirect to HTML, or an empty file are excluded. The rendering check was a single homepage fetch per site with scripts and styles stripped, so a site that serves crawlers a different page than browsers can be misjudged. Everything here is a snapshot. robots.txt files change, and I rerun the census and update this page.

The per-crawler counts, the misconfiguration counts, the named examples, the rendering results and the llms.txt adoption list are at /data/census.json. The data is free to reuse under CC BY 4.0. A link back to this page is the only ask. Building something that reads this programmatically? See /for-agents and /openapi.json. Prefer one fixed page to cite instead of individual stats scattered across this site? See the Q3 2026 report. Curious how this site itself gets found? See the full, disclosed distribution funnel.

DOI for this version: 10.5281/zenodo.22729641. Concept DOI, always resolves to the latest version: 10.5281/zenodo.22729640.

@misc{calder_ai_crawler_census_2026,
  author = {Calder, Reese},
  title = {AI Crawler Census: robots.txt access for 5,000 sites},
  year = {2026},
  publisher = {Zenodo},
  doi = {10.5281/zenodo.22729641},
  url = {https://doi.org/10.5281/zenodo.22729641}
}

A separate follow-up, measuring what Cloudflare's September 15 crawler change did to robots.txt files (a Common Crawl sample, a 754-site panel and an edge test), is on Zenodo as 10.5281/zenodo.22919706 and written up in this guide. IT-Administrator cited it on September 23.