Which of the top 5,000 sites block AI crawlers?

560 sites block GPTBot, the crawler OpenAI uses to gather training data. 258 of those, 46.1%, also block OAI-SearchBot, the separate crawler that indexes pages for ChatGPT search. The other 302 block GPTBot alone, which keeps them out of training and still lets ChatGPT cite them. OpenAI documents the two crawlers separately, and most of these sites otherwise welcome search engines, so for many of the 258 this looks like an accident rather than a decision.

305 sites block at least one AI search crawler and still allow Googlebot. Among the largest: facebook.com, instagram.com, twitter.com, amazon.com, x.com, tiktok.com, pinterest.com, yahoo.com, soundcloud.com, ebay.com, imdb.com, unsplash.com, statista.com. News publishers on the list include nytimes.com, cnn.com, theguardian.com, forbes.com, bbc.com, reuters.com, washingtonpost.com, wsj.com, bloomberg.com, cnbc.com. Big publishers often do this on purpose, for licensing reasons. For a smaller business it is nearly always unintended. Check your own site if you are not sure which crawlers you block.

A separate check on 31 August 2026 fetched the homepage of 607 sites ranked around 10,000 in the same list, the way a crawler that does not run JavaScript sees them. 79.2% had substantial readable text. 16.0% were effectively empty without JavaScript, under 250 characters of text. 4.8% were thin, under 800 characters. 58.0% had no JSON-LD structured data and 26.4% had no meta description. So roughly one site in five that lets AI crawlers in still gives them nothing to read.

On 6 September 2026 I requested /llms.txt from the same 5,000 domains. 373 of them, 7.5%, served a file that follows the proposed format (a markdown H1 first, at least one link). 2201 answered 404. 180 returned an HTML page instead, which is what a site does when it serves its app for any path, and 347 redirected somewhere else. 122 returned text that was not an llms.txt. 416 refused the request outright and 1241 could not be reached at all; both are counted, not treated as absent.

55 of the 373 also publish an llms-full.txt, and 320 include the blockquote summary the format asks for. The highest ranked sites with one: cloudflare.com, azure.com, github.com, fastly.net, digicert.com, wordpress.org, adobe.com, opera.com, samsung.com, sentry.io, wordpress.com, unity3d.com. Every domain with a valid file is listed in the JSON below. For the question people search, whether you need an llms.txt to show up in ChatGPT: adoption is 7.5% across the 5,000 and 11.0% in the top 100, and none of the AI search engines has documented reading it for search, so treat it as cheap and optional. The generator and validator will make one if you want it.

Domains come from the Tranco top list, a research ranking that averages several traffic lists. Each robots.txt was fetched once with my own identified user agent. I never impersonate another crawler. A crawler counts as blocked when the most specific robots.txt group that applies to it disallows the site root. If no group names the crawler, the catch-all group applies, so a site that disallows everyone counts as blocking every crawler. The "named in robots.txt" column counts files that mention the crawler at all, whether to allow or block it.

Percentages use the 2,778 sites that served a robots.txt as the base. Sites that returned an error, a redirect to HTML, or an empty file are excluded. The rendering check was a single homepage fetch per site with scripts and styles stripped, so a site that serves crawlers a different page than browsers can be misjudged. Everything here is a snapshot. robots.txt files change, and I rerun the census and update this page.

The per-crawler counts, the misconfiguration counts, the named examples, the rendering results and the llms.txt adoption list are at /data/census.json. The data is free to reuse under CC BY 4.0. A link back to this page is the only ask.