Cloudflare's AI crawler settings since September 15, and the two that also block Googlebot
Since September 15, 2026, Cloudflare's Training setting has four options. Disallow AI Training refuses AI training and keeps Googlebot, Bingbot and Applebot crawling for search. Block and Block on pages with ads stop those three crawlers as well. New ad-supported domains start on Disallow AI Training. Existing domains kept their settings.
Want your site counted in the October census re-run?
Leave your domain and address to be counted for the October census re-run. Results are posted on /data; nothing is emailed.
Correction, September 23, 2026. I wrote the first version of this guide on September 7 from Cloudflare's July plan, which said that from September 15 any site blocking AI training would also block Googlebot, Applebot and BingBot. What Cloudflare released on September 15 works differently. It added a Disallow AI Training setting that leaves those three crawlers allowed for search, and it moved sites that had chosen Block for Training onto that setting. The old headline and summary said the new default for ad-supported domains blocks Googlebot, and it does not. This page now follows Cloudflare's September 15 blog post and press release.
Update, September 22, 2026. On September 15 Cloudflare replaced Managed robots.txt with a new setting called Bot Preference Sync (Cloudflare's announcement). Its blog post says the new block is marked # BEGIN Cloudflare Bot Preference Sync and that sites using the old feature would be asked to review their settings. I re-checked the census sites on September 22. On September 7, 79 of them carried exactly the eight-crawler block list the managed block writes. Today 6 of the 2,771 census sites with a readable robots.txt still serve the old Cloudflare Managed content block, and none serve the new marker. Of the 54 of those 79 sites I could read again, 45 no longer block any of the eight crawlers in robots.txt. Patreon and blender.org are two examples: archived copies show the managed block on patreon.com on September 15 and on blender.org on September 10, and neither file has it today. A robots.txt file cannot show what Cloudflare now enforces at the network edge. A request that names a crawler in its user agent can show part of it: on September 23, 336 of 422 Cloudflare-served sites that used to carry the old block and no longer block any of the eight crawlers in robots.txt answered GPTBot and ClaudeBot with a 403 while serving PerplexityBot the page. A rule that checks whether a request really comes from the crawler's own network would not show up in that test. Corrected September 25: this note first said a public request cannot see the edge at all. If you turned the old setting on to publish a training opt-out, open yoursite.com/robots.txt and check whether it still says so. The new settings described below went live the same day, per the same announcement. The full counts and the per-site list are on the retirement data page.
The three categories
Cloudflare sorts AI crawlers into three categories. Search fetches a page to index it and answer questions about it later. Agent acts in real time on a user's behalf, fetching a page because someone asked an assistant to do something with it right now. Training scrapes a page to put it in a model's training data. Each category has its own setting in the dashboard.
What the Training setting does now
Training has four options: Allow, Disallow AI Training, Block on pages with ads, and Block. Cloudflare describes Disallow AI Training this way: accountable mixed-use crawlers remain allowed for search, and every other training crawler is blocked, including the training-only crawlers run by Amazon, Anthropic, Meta and OpenAI. The setting also publishes a no-training preference in the site's robots.txt through Bot Preference Sync.
The mixed-use crawlers are Googlebot, Bingbot and Applebot. Each one fetches pages for a search engine, and the same fetches can also feed AI training. Cloudflare calls Google, Microsoft and Apple "Accountable" because each has a training opt-out today or a dated commitment to one. For Google the opt-out is a robots.txt Disallow for Google-Extended, and for Apple it is a Disallow for Applebot-Extended. Microsoft is still building a way for Bingbot to honor a no-training preference in robots.txt and is aiming for early 2027.
Block and Block on pages with ads behave differently. Cloudflare's post says both of them now apply to mixed-use crawlers, including Applebot, Bingbot and Googlebot, so either one affects search as well as training. A site that picks one of them to keep its pages out of AI training also shuts out Google and Bing search, on every page or on the pages with ads.
Defaults and what happened to existing sites
Existing domains kept their settings. A domain that had the old Block AI bots toggle on was moved to Search on Allow, Training on Disallow AI Training and Agent on Block on pages with ads, and any earlier Training choice of Block or Block on pages with ads was moved to Disallow AI Training. So the migration did not start blocking Googlebot on anyone's site.
New domains that run ads start with Training on Disallow AI Training and Agent on Block on pages with ads. New domains without ads start with Training and Agent on Allow. Search is allowed by default for both.
The edge setting and robots.txt are separate
Cloudflare's settings work at the network edge, in front of the request, for every visitor that reaches the site through Cloudflare. Robots.txt is a text file a crawler can choose to read and honor. Disallow AI Training uses both: the edge blocks training-only crawlers, and robots.txt carries the no-training preference to the mixed-use crawlers that are still let through. For Google and Apple that preference only counts if the Google-Extended and Applebot-Extended lines are actually in the served file. On September 22 none of the 2,771 census sites with a readable robots.txt carried the Bot Preference Sync marker (details), so check the file rather than assume the setting wrote it.
What to do about it
- If you want your pages kept out of AI training and still want to be in Google and Bing search, Disallow AI Training is the Cloudflare setting for that. Afterwards, open yoursite.com/robots.txt and confirm the Google-Extended and Applebot-Extended Disallow lines are there.
- Treat Block and Block on pages with ads under Training as search settings as well. They stop Googlebot, Bingbot and Applebot, so choosing one of them removes those pages from Google and Bing search.
- Bingbot has no training opt-out that leaves search in place until Microsoft ships one, which it is aiming for in early 2027. Until then the only way to keep Bingbot from collecting pages for training is to keep it off them completely, which also takes those pages out of Bing.
- None of this replaces checking robots.txt. A site can be right on Cloudflare's dashboard and still be wrong in its own robots.txt file, or the other way around. The checker on this site reads what a site's robots.txt actually allows for the AI search crawlers. It also sends one test request for each of them, which shows whether the server turns that name away. It cannot see Cloudflare's dashboard setting, and a rule that checks whether a request really comes from the crawler's own network would not show up in a test request.
Related: the AI Crawl Control guide covers the per-crawler blocks and what they look like on real sites, the Managed robots.txt retirement data shows what happened to sites that used Cloudflare's old robots.txt block, should you block GPTBot covers the robots.txt version of training versus search, and what actually controls AI Overviews goes through the Google-Extended and Googlebot distinction.