Cloudflare will block AI crawlers by default from September 15, and it can take Googlebot with it

Starting September 15, 2026, new domains on Cloudflare get AI training and agent crawlers blocked by default on pages that carry ads. Search crawlers stay allowed. The part worth knowing before you touch a setting: blocking training also blocks Googlebot, Applebot and BingBot, because those three do both jobs.

What is actually changing

Cloudflare sorts AI crawlers into three categories. Search fetches a page to index it and answer questions about it later. Agent acts in real time on a user's behalf, fetching a page because someone asked an assistant to do something with it right now. Training scrapes a page to put it in a model's training data.

On September 15, 2026, Cloudflare sets new defaults for each of these three: Training and Agent are blocked by default on the pages that display ads, for new domains onboarding to Cloudflare. Search stays allowed by default. Existing sites keep whatever they already have unless they change it, and everyone can adjust the setting for any of the three categories at any time from Security settings.

This is a different layer from robots.txt. Robots.txt is a text file a crawler can choose to read and honor, and plenty of the crawlers named in it never ask permission at all, which is most of why a site's stated rules and what actually happens to it can differ. Cloudflare's setting works at the network edge, in front of the request, for every visitor who reaches the site through Cloudflare, which is a much larger share of AI crawler traffic getting one enforcement point in one place.

The part that catches people out

Some crawlers do more than one of those three jobs. Googlebot is the clearest case: it powers ordinary Google Search and it also feeds Google's AI Overviews, so under Cloudflare's own categories it counts as both a search crawler and, because Google can also use crawled pages for its models, something the Training toggle reaches too. Cloudflare says it plainly: a site that turns on the Training block also blocks Googlebot, Applebot and BingBot, since all three are multi-purpose crawlers evaluated under both policies.

So a site owner who reads "block AI training" and thinks they are only opting out of being used to train a model can turn off classic Google Search and Bing Search for their own site in the same click. That is not a hypothetical edge case, it is the documented behavior of the feature as Cloudflare describes it.

This is the same category of mistake this whole site has been measuring in robots.txt for months, a training opt-out that silently costs a site its search presence, just arriving through a dashboard toggle instead of a text file, and with higher stakes, since it can reach classic Google and Bing search rather than only the newer AI answer engines.

Who this actually affects

Three conditions, all of which have to be true. The site has to be on Cloudflare. It has to be a new domain onboarding after September 15, or an existing site whose owner opts into the new categories rather than keeping the old settings. And the pages in question have to display ads, since the new default only applies there. A site that is not on Cloudflare, or is already on Cloudflare with settings left alone, or does not run ads on the pages in question, is not swept into this by the September 15 date on its own.

If in doubt, check directly rather than guess. Cloudflare's own Security settings for a site show the current state of all three categories, and the setting can be changed at any time, including opting out of the new defaults before or after September 15.

What to do about it

• If a site is on Cloudflare, check the AI crawler settings before September 15 rather than after, so a choice gets made rather than a default landing unnoticed.
• Decide Training and Search separately, the same advice this site gives for robots.txt. Wanting to keep pages out of a model's training data is a reasonable, common choice on its own. It is a different decision from wanting to disappear from Google Search, Bing, and AI answers, and Cloudflare's own categories only keep those separate as long as the crawler in question is not one of the multi-purpose ones.
• If Googlebot, Applebot or BingBot matter to a site, which is true for almost every site that wants to be found at all, leave Training either off or check specifically that turning it on has not also blocked those three before relying on the setting.
• None of this replaces checking robots.txt. A site can be correct on Cloudflare's dashboard and still be wrong in its own robots.txt file, or the other way around. The checker on this site reads what a site's robots.txt actually allows for the AI search crawlers; it does not see Cloudflare's separate, network-level setting, since that is not something a public request can inspect from outside.

Related: should you block GPTBot covers the robots.txt version of training-versus-search, and what actually controls AI Overviews goes through the Google-Extended and Googlebot distinction this same mistake can trip over even without Cloudflare involved.