AI Crawler Census: Q3 2026 report
The first in a quarterly series. Every number below comes from this site's own census of the Tranco top 5,000 sites, fetched 2026-09-07 through 2026-09-19, and is already published somewhere else on this site -- this page gives the whole set one fixed, citable home instead of leaving each stat to be quoted from wherever it first appeared.
Want to know if YOUR site's AI visibility changes?
Enter your domain and email. One email if a crawler you rely on gets newly blocked, or an existing block goes away. No spam.
The headline numbers
Crawler access
2,771 of the Tranco top 5,000 sites served a robots.txt. Among AI-training crawlers, GPTBot is blocked by 535 (19.3%), CCBot by 573 (20.7%), ClaudeBot by 516 (18.6%). Among AI-search crawlers, the ones that actually decide whether an AI answer can cite you, PerplexityBot is blocked by 359 (13.0%), Claude-SearchBot by 250 (9.0%), OAI-SearchBot by 241 (8.7%). 369 sites (13.3%) block at least one of the three search crawlers, and 296 of those (80.2% of the 369, 10.7% of all 2,771) still allow Googlebot in -- visible in Google, invisible in AI answers. Full per-crawler table: the crawler census.
The 296: deliberate or accidental, by rank
A reader's critique (Mike of viewfy.ai, on the original Dev.to post) pushed a rank-stratified breakdown of the 296 rather than treating them as one population. The gradient runs the expected direction but is modest, not sharp:
| Rank band | Blocks an AI search crawler | ...and still allows Googlebot | Share |
|---|---|---|---|
| 1-100 | 15 | 13 | 86.7% |
| 101-1,000 | 82 | 68 | 82.9% |
| 1,001-5,000 | 272 | 215 | 79.0% |
| All 369 | 369 | 296 | 80.2% |
All 13 of the top-100 sites in the 296 are recognizable platforms and publishers (facebook.com, amazon.com, x.com, tiktok.com among them) with the resources to write a robots.txt rule on purpose. Rank correlates with intent here, but not enough to sort the 296 cleanly. Full breakdown, credited: the rank split.
Content-Signal adoption
165 of the 5,000 sites (3.3%) carry Cloudflare's Content-Signal line, a stated preference rather than an enforced rule. 125 of those state ai-train=no; 99 of the 125 (79.2%) back that preference with an actual robots.txt Disallow naming at least one AI-training crawler, and 26 (20.8%) state the preference with nothing behind it in robots.txt at all. Detail: what Content-Signal does and does not do.
llms.txt adoption and the Honor Roll
385 of the 4,999 domains checked (7.7%) serve a spec-valid llms.txt: a markdown H1 first, at least one link. Adoption is heaviest near the top of the ranking: 10.0% in ranks 1-100, 9.0% in ranks 101-1,000, 7.3% past rank 1,000. 366 sites (7.3% of the 4,999) clear a higher bar at once -- a valid llms.txt AND zero blocks on any of the three AI-search crawlers -- the elite intersection published as the AI-Visibility Honor Roll.
Can the crawlers read the page once they get in
A separate rendering check of 607 homepages (ranked near 10,000 in the same list, fetched once with no JavaScript executed) found 79.2% readable, 16.0% effectively empty (under 250 characters), and another 4.8% thin (under 800 characters). Roughly one site in five that lets AI crawlers in still gives them nothing to read. Detail: the named invisible and thin sites.
Method
Domains come from the Tranco top list. Each robots.txt was fetched once with this site's own identified user agent; a crawler counts as blocked when the most specific applicable robots.txt group disallows the site root. Percentages for the crawler-access tables use the 2,771 sites that served a robots.txt as the base; the llms.txt and Content-Signal checks use the fuller 4,999/5,000-domain samples named above. The rendering check is a separate 607-site sample fetched without JavaScript. Full method, per section: the crawler census, the llms.txt census, the rendering check.
Get the data
The raw per-crawler and per-site data behind every number on this page is at /data/census.json, CC BY 4.0. This report is the fixed citation target for it: future reports (starting with the October re-run) link back here for the Q3 baseline rather than restate individual stats.
Cite this report
DOI: 10.5281/zenodo.22729641. Concept DOI, always resolves to the latest version: 10.5281/zenodo.22729640.
@misc{calder_ai_crawler_census_q3_2026,
author = {Calder, Reese},
title = {AI Crawler Census: Q3 2026 report},
year = {2026},
publisher = {Zenodo},
doi = {10.5281/zenodo.22729641},
url = {https://ai-visibility.lastminutedealshq.com/report/2026-q3}
}