Why your robots.txt User-agent line may not be doing anything

If you have a robots.txt group that names a crawler and it does not seem to be taking effect, there is a specific reason this happens that almost nobody checks for. The name you wrote may not be a legal crawler name, and the parser that matters throws away the illegal part before it compares anything.

I measured how common this is across the top 5,000 sites. Of the 2,725 that serve a robots.txt containing User-agent lines, 582 have at least one value that is not a legal crawler name. In 484 of them the illegal character sits inside the name itself, so the group applies to a different name than the one written. On 120 sites there is a line that resolves to nothing at all and can never match any crawler.

The rule

RFC 9309, the robots.txt standard, is strict about what a crawler name can contain. Section 2.2.1 says the product token must contain only uppercase and lowercase letters, underscores and hyphens. No digits. No dots. No slashes. No spaces.

Google's robots.txt parser is open source, and it enforces that rule by truncating. Before comparing a User-agent value to a crawler's own name, it walks the value from the start and stops at the first character that is not a letter, hyphen or underscore. Everything from that point on is discarded. Then it compares what is left, case-insensitively, for an exact match.

/*static*/ absl::string_view RobotsMatcher::ExtractUserAgent(
    absl::string_view user_agent) {
  // Allowed characters in user-agent are [a-zA-Z_-].
  const char* end = user_agent.data();
  while (absl::ascii_isalpha(*end) || *end == '-' || *end == '_') ++end;
  return user_agent.substr(0, end - user_agent.data());
}

There is one exception written into the same file. A value that starts with an asterisk, followed by either the end of the value or a space, is still the catch-all group and is never truncated. So User-agent: * is fine, and so is User-agent: * v2.

What that does to real lines

These are the values that appear most often in the top 5,000, with what Google's parser actually compares. The count is how many of those sites carry the line.

A value that begins with a digit is the worst case. 008 is the real name of a crawler, and 360Spider is the real name of another, and both start with a character that is not a letter, so the truncated value is empty and the group cannot match anything. Those lines are doing nothing on 120 sites.

The case that is not a problem

Truncation is not always a bug, and it is worth being clear about that because the raw count overstates the damage. If the illegal character comes after a complete, legal name, the truncation just strips a suffix and the comparison still succeeds. CCBot/2.0 becomes CCBot, which is correct. meta-externalagent/1.1 becomes meta-externalagent, which is correct. Brightbot 1.0 becomes Brightbot, which is correct.

That accounts for 98 of the 582 sites. The other 484 are cut inside the name, and those are the ones where the group is not doing what it looks like it is doing.

Which crawlers this actually changes

It depends on the parser the crawler was built with, and this is the part that gets reported badly. Google's parser truncates. The two most common Python libraries, urllib.robotparser and Protego, do not. Run the same file through them and they report AI2Bot as disallowed, because they compare the whole string. So a group headed User-agent: AI2Bot blocks AI2Bot under those libraries and does not block anything under Google's.

Which behaviour you get is a property of the crawler, not of your file, and you cannot tell from the outside. Worth noting too that a crawler calling itself AI2Bot is already using a name the standard forbids, so there is no version of this where every parser agrees. That is the honest answer, and anyone telling you a single number for how many sites are affected is skipping over it.

How to check your own file

Read the User-agent lines and look for anything other than letters, hyphens and underscores. Digits and dots are the two that catch people. To have it done for you, paste your file or your address into the robots.txt checker on this site and it will list every line that does not survive.

If you would rather run it yourself, the linter that produced the numbers above is a single Python file with no dependencies, and it takes a path or a URL:

python robots_agent_lint.py https://example.com/robots.txt

It prints every value that does not survive, what the parser reads instead, and whether the truncated name collides with another group in the same file. It exits non-zero if anything truncates, so it can run in CI. The source is in the aicheck repository.

What to do about it

There is no fix that makes every parser agree, because the crawler's own name breaks the rule. What you can do is stop relying on a line that half the parsers ignore. If you want a crawler kept out and its name contains a digit or a dot, add a catch-all group that covers it, or block it at the server rather than by request. A User-agent: * group with the rules you want is the one thing every parser reads the same way.

If you want to know whether the crawlers that feed AI answers can currently reach your site, the checker reads your robots.txt the way Google's parser does, including this truncation rule.