What changed: AI crawler policy shifts

These sites allowed at least one AI-search crawler as of 2026-09-07 and now block it. If you rely on being cited by ChatGPT search, Perplexity or Claude, a change like this on your own site is exactly what a daily re-check would have caught the day it happened.

The opposite direction: these sites blocked an AI-search crawler as of 2026-09-07 and removed the block, becoming newly citable.

34 sites added Applebot-Extended, Apple's AI-training crawler, to their robots.txt in this same window and changed nothing else -- a deliberate policy decision, not template churn (see method below). Applebot-Extended controls AI-training opt-out only; it does not affect whether these sites can still be cited, the way the search-crawler tables above do.

23 smaller or mixed single-crawler changes in the same window (a training crawler added or removed here, a live-fetch agent there) did not meet either bar above. The full list, plus every field behind the tables on this page, is in the raw data.

The full event list, including "everything else," is at /changes.json (CC BY 4.0, CORS-open). Subscribe to /changes.xml for the same events as an RSS feed. Method: re-crawl of the high-signal slice described above using the same crawler this site's AI crawler census uses, diffed field-by-field against the named baseline; cdn_template_churn (shared hosting-template noise) is classified and excluded before anything reaches this page.