The "BEGIN Cloudflare Managed Content" block in your robots.txt, and what it does to rules you already had

If your robots.txt now has lines you never wrote, bracketed by a comment that says BEGIN Cloudflare Managed Content, here is exactly where that comes from, how it actually combines with what you already had, and the one case that catches people out: your own rule for a crawler Cloudflare's block already names.

Where the block comes from

This is Cloudflare's managed robots.txt feature, part of AI Crawl Control. When it is turned on for a domain on Cloudflare, either by a site owner in Security settings or by the new default Cloudflare is rolling out to new domains from September 15, 2026 (covered separately in what that default actually changes), Cloudflare writes rules for a fixed list of named AI crawlers into the file served at yoursite.com/robots.txt, plus a Content-Signal line on the wildcard group (covered in is Content-Signal a syntax error).

It is prepended, not merged into your file's own groups

Cloudflare's own documentation states the mechanism plainly: if a website already has a robots.txt file, verified by an HTTP 200 response, Cloudflare will prepend its managed robots.txt before the existing robots.txt, combining both into a single response. Prepend means first. Whatever you wrote stays in the file, moved after Cloudflare's block, not replaced.

Cloudflare's own documentation gives a worked example. Before the feature is enabled, a site's robots.txt reads:

User-agent: *
Disallow: /lp
Disallow: /feedback
Disallow: /langtest
Sitemap: https://www.crawlstop.com/sitemap.xml

With the feature enabled, the same request instead returns this, exactly as shown in Cloudflare's own docs:

# BEGIN Cloudflare Managed content
User-Agent: *
Content-signal: search=yes, ai-train=no, use=reference
Allow: /
User-agent: Amazonbot
Disallow: /
User-agent: Applebot-Extended
Disallow: /
User-agent: Bytespider
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: GPTBot
Disallow: /
User-agent: meta-externalagent
Disallow: /
# END Cloudflare Managed Content
User-agent: *
Disallow: /lp
Disallow: /feedback
Disallow: /langtest
Sitemap: https://www.crawlstop.com/sitemap.xml

What happens if you already had a rule for one of those crawler names

Named crawlers get their own group either way. If your own robots.txt, before Cloudflare's block was added, already named one of the same crawlers, GPTBot for instance, the file now has two separate User-agent: GPTBot groups: Cloudflare's, which disallows it, sitting first, and yours, sitting second, saying whatever you originally wrote, allow or disallow.

RFC 9309 says a crawler should treat every group written for its own name as one combined ruleset, gathering the rules together regardless of how many separate groups declare that name. Google's own robots.txt documentation confirms this is exactly how its crawlers behave, checked directly against its current specification page: if there is more than one specific group declared for a user agent, all the rules from the groups applicable to that user agent are combined internally into a single group, and the order of the groups within the file is irrelevant.

That is Google's documented behavior for its own crawlers. None of the eight names in Cloudflare's block above are Google's. OpenAI, Anthropic, Common Crawl, Amazon, ByteDance and Meta have not published how their own crawlers handle two separate groups for the same name. The RFC's advice is a should, not a must, and this site's own checker already flags, on any file, that some parsers only read the first group they find for an agent and ignore the rest. If that turns out to be true for any of the eight, and nothing public says it is not, then whichever policy sits first in the file is the one that survives for that crawler, no matter what a site owner separately wrote lower down. Because Cloudflare's own mechanism always prepends, its policy is always the one sitting first.

A second collision Cloudflare's own example already shows

Look again at the example above. Cloudflare's block opens with its own User-agent: * group, allowing everything, right before the site's original User-agent: * group, which disallows three specific paths. That is two catch-all groups in one file, not just two named-crawler groups. Google's documentation draws a clear line between a crawler-specific group and the wildcard group, stating the two are not combined with each other, but it does not say what happens with two separate wildcard groups declared back to back, which is exactly what Cloudflare's own example produces. Whether a site's own /lp, /feedback and /langtest exclusions still apply to crawlers that fall through to the wildcard group is not something either company's documentation actually settles.

Sometimes it does not prepend at all

Cloudflare's own docs describe prepending as the guaranteed behavior. A site owner reported the opposite in March 2026, on Cloudflare's own documentation repository on GitHub: turning on managed robots.txt served only the managed rules, with the site's own existing rules nowhere in the response. Cloudflare's own team replied that this sounded like a real functionality problem rather than a documentation error and forwarded it to their engineering team. The issue was closed there, and there is no public follow-up confirming a fix. One reported case is not proof of a widespread bug, but it is a real, sourced reason not to assume your own served file matches the documented behavior without checking it.

What to actually do

Related: Cloudflare's September 15 AI-crawler defaults covers when this feature turns on by itself, the Content-Signal line this same block adds covers the other part of what gets prepended, and the robots.txt checker reads your actual served file, including this exact duplicate-group situation.