Is Content-Signal in robots.txt a syntax error?

No. RFC 9309 explicitly allows lines like this. But a real Google tool had a confirmed bug that flagged it as invalid anyway, and a second, separate Google tool's current status on this is not confirmed. Here is what each one actually said, checked against the source.

Where this line comes from

Cloudflare's Content Signals policy adds a line inside a User-agent group:

User-Agent: *
Content-Signal: search=yes, ai-train=no

It expresses three separate preferences a crawler can read: search (build a search index and show results for the page), ai-input (use the page's content in a live AI answer, such as retrieval-augmented generation), and ai-train (use the page to train or fine-tune a model). Each takes yes or no, and leaving one out states no preference. Cloudflare says plainly that this is advisory: some crawlers may simply ignore it. By Cloudflare's own count, over 3.8 million domains already have their managed robots.txt feature turned on, and Cloudflare adds this line to those automatically. That scale is exactly why so many site owners are seeing a line they never wrote themselves and asking what it means.

How common it actually is, in the sites that matter for search

Cloudflare's 3.8 million figure spans every domain on their managed robots.txt feature, most with negligible search traffic. I checked the same Tranco top-5,000 sample already published on this site's crawler census, fetched fresh on September 8, 2026: 165 of the 5,000 sites (3.3%) already carry a Content-Signal line. Of those 165, all but one explicitly allow search indexing (search=yes, none say no), 125 explicitly refuse AI training (ai-train=no) against 40 that allow it, and 74 state a position on ai-input (63 yes, 11 no) while the remaining 91 leave it unstated, which under the spec means no preference either way, not a no.

The more interesting number: 85 of those 165 sites, just over half, already carry a fourth field, use=reference, that Cloudflare's own announcement of it describes as still being tested. The value is meant to set how permissively content already collected may be handled: use=immediate permits interacting with the content but storing or reusing nothing, use=reference permits indexing, excerpting and linking back, and use=full permits summarizing and reproducing it. Every occurrence found in this sample used the same value, reference. The exact same three-field string, byte for byte, turned up across sites with nothing else in common, a membership platform, a cookie-consent vendor, a domain marketplace, and several unrelated content sites, which reads as one default already rolling out from Cloudflare's own side rather than site owners each choosing it. Raw data: content_signal_census_20260908.json, CC BY 4.0, same terms as the rest of this site's data.

The preference does not always have a robots.txt rule behind it

Cloudflare says plainly that Content-Signal is advisory, not a block, and I wanted to see what that means in practice rather than leave it as a footnote. I checked three of the sites already named on this page's own data, trendmicro.com, visma.com and usercentrics.eu, live against their raw robots.txt files rather than trust the census alone. All three carry Content-Signal: ai-train=no. None of the three disallows a single named training crawler from the site root. Trend Micro's file lists ClaudeBot, GPTBot and several others by name, but only to exclude specific utility paths, PDFs, a config folder, a few admin sections, the same exclusions applied to Googlebot and Bingbot in the same group; the site itself is otherwise open to all of them. Visma's entire file is six lines, one Disallow for /checkout, nothing else. Usercentrics excludes a list of tracking query parameters and ends with Allow: /.

None of this means these three sites are being dishonest. Cloudflare's managed robots.txt feature adds the Content-Signal line on a site owner's behalf, and someone who turned on the training opt-out through Cloudflare's dashboard reasonably expects that to be the whole story. But the line by itself only states a preference. If a crawler operator does not choose to honor it, and Cloudflare's own words leave that possibility open, nothing else in these three files stops it from crawling and training on the page. These three turn out not to be unusual; the full-sample check below shows they sit in the real minority, and it is exactly the kind of gap the checker on this site exists to catch: check what your own file actually disallows, not just what it states a preference about.

I checked this across the whole sample rather than stop at three examples. Of the 125 sites in the same September 8 census stating Content-Signal ai-train=no, 99 of 125 sites stating ai-train=no (79.2%) back it with a robots.txt disallow for a named training crawler, naming and blocking at least one of the 12 training-purpose crawlers this site's own census tracks. 26 (20.8%) state ai-train=no with nothing in robots.txt behind it, the same shape as trendmicro.com, visma.com and usercentrics.eu above. The 79.2% is not spread evenly either: 82 of those 99 backed sites name and block exactly the same eight crawlers, gptbot, claudebot, google-extended, ccbot, meta-externalagent, bytespider, applebot-extended and amazonbot, the identical list turning up across sites with nothing else in common. That reads as one bundled Cloudflare feature writing the Content-Signal line and the actual block rules together for most adopters, not each site choosing its own list by hand, though Cloudflare has not stated this and it is read from the pattern, not confirmed by them. Raw data: content_signal_backing_census_20260908.json, CC BY 4.0, same terms as the rest of this site's data.

The setting that does enforce it, if you want that

Cloudflare's managed robots.txt is one setting. A separate one, AI Crawl Control, is the feature Cloudflare frames as actual enforcement: it blocks a crawler at the network edge before the request ever reaches the site, regardless of whether that crawler bothers reading robots.txt at all. Cloudflare's own documentation states plainly that the two are meant to work together: robots.txt to express a preference, AI Crawl Control to enforce it. Turning on managed robots.txt by itself only writes a line. If a site wants the training opt-out to actually hold against a crawler that ignores files like this one, AI Crawl Control, not Content-Signal, is the setting that does that, and it lives in the same Security > Bots section of the dashboard as the toggle that generates the robots.txt line in the first place.

What the spec actually says

RFC 9309, the robots.txt standard, does not define a closed list of allowed lines. It defines allow, disallow and user-agent precisely, then adds: crawlers may interpret other records that are not part of the core protocol, and parsing those other records must not interfere with parsing the records the RFC does define. An unrecognized field name is not a syntax error under the spec that governs robots.txt. Google's own robots.txt documentation says the same thing in its own words: rules other than allow, disallow and user-agent are ignored by the parser, naming crawl-delay as an example of a field Google does not support but also does not reject the file over. Content-Signal is exactly this kind of line: unread by the crawling parser, present for something else to read, and not a break in the file.

The real, unanswered question this traces back to

A Stack Overflow question from November 2025, viewed over 4,000 times, asks exactly this: Google Search Console reported a format error over this line, and the person wanted to keep it while satisfying Search Console. It has three answers, none accepted and none scoring above zero. One suggests commenting the line out, which defeats the point of having it. Another cites a GitHub issue as evidence the problem is already fixed. I checked that issue directly rather than take the citation at face value, and the real answer is more specific than either.

What was actually confirmed, checked at the source

Lighthouse, the auditing tool behind Chrome DevTools and PageSpeed Insights, had this exact bug. Issue #16776 on the Lighthouse GitHub repo, filed October 2025, reports Content-Signal flagged as an unknown directive on a real site using Cloudflare's feature. A contributor opened a fix within two weeks, pull request #16767, which was merged. A Lighthouse maintainer confirmed directly on the issue in June 2026 that the fix has been in Node since the merge, and that it reaches Chrome and PageSpeed Insights on their own separate rollout schedule, which took a few months longer. If a current, up to date PageSpeed Insights or Lighthouse run still shows this as an unknown directive today, that is worth re-running rather than trusting as final, since the underlying bug is fixed.

Google Search Console is a separate product from Lighthouse, built and maintained separately, and one commenter on that same GitHub issue in January 2026 pointed out that Search Console's own message for this is worded differently: syntax not understood, not unknown directive. That is a real, useful distinction the Stack Overflow answers missed by treating one Google tool's fix as if it covered both. I could not find a Google-published confirmation that Search Console's specific check has also been corrected, and I am not going to claim it has without a source. If Search Console flags this line for you today, the honest position is: the line itself is valid per the spec Google itself documents, a sibling Google tool had the identical bug and fixed it months ago, and whether Search Console's checker has caught up is currently unconfirmed.

What to actually do about the warning

Related: does the noindex meta tag stop AI training covers where Content-Signal fits among the real opt-out mechanisms, and Cloudflare's September 15 AI-crawler defaults covers a separate, network-level Cloudflare setting that this line does not control.