robots.txt checker for AI crawlers

A robots.txt rule can be spelled correctly, sit in the right place, and still govern nothing at all. This reads your file the way the reference parser does and tells you which lines are doing no work.

Or paste the file

Useful if the site is not live yet, sits behind a login, or serves a bot wall instead of the real file. Nothing you paste is stored.

What it looks for

User-agent values that get cut short. The one almost nobody knows about, explained below.

Rules above the first User-agent line. Allow and Disallow only mean anything inside a group. A Disallow written before any User-agent line belongs to no group, and parsers throw it away. It is an easy mistake to make when you edit the top of the file.

Agent names that are not real crawler tokens. User-agent: ChatGPT and User-agent: Claude look right and match nothing, because those are product names. The crawlers are called OAI-SearchBot, ChatGPT-User, GPTBot, ClaudeBot, Claude-User and Claude-SearchBot. A group headed by a name no crawler sends is never selected, so its rules never run.

No catch-all group. If a crawler finds no group for its own name and there is no User-agent: *, RFC 9309 says it has no rules to obey, so it may fetch everything. A file made entirely of named groups is more permissive than it looks.

The same agent declared twice. Matching is case-insensitive, so GPTBot and gptbot are one agent in two groups. The RFC says merge them. Not every crawler does, and some obey only the first group they find, which makes the outcome depend on whose parser is reading.

Directives the parser never reads. The reference implementation handles user-agent, allow, disallow and sitemap. Nothing else, including crawl-delay.

The rule most people have never heard of

RFC 9309, the standard that finally wrote robots.txt down in 2022, says a crawler's name may contain only letters, underscores and hyphens. No digits, no dots, no slashes, no spaces. Google's open-source parser, the one most other implementations were built from, enforces that by cutting every User-agent value at the first character outside that set and then matching what is left as an exact, case-insensitive equal.

So the line does not mean what it reads as:

User-agent: Brightbot 1.0     ->  matched as  Brightbot
User-agent: bigsur.ai         ->  matched as  bigsur
User-agent: GPTBot/1.2        ->  matched as  GPTBot
User-agent: img2dataset       ->  matched as  img
User-agent: Crawl4AI          ->  matched as  Crawl

The first and third are harmless: the version number falls off and the right crawler is still matched. The others are not. img2dataset becomes img, and no crawler on earth is called img, so that group is dead weight. If nothing else in the file matches that crawler and there is no catch-all group, the rules you wrote for it never apply and it is free to fetch the lot.

This is not theoretical. Running the check over one widely copied public block list turned up 21 values out of 175 that do not survive, 16 of them landing on a name declared nowhere else in the file. I reported it to that project rather than writing it up quietly: the finding, with the source and the full table.

The honest caveat. Not every parser truncates. Python's built-in urllib.robotparser and Scrapy's Protego both compare the whole value, so under those a line like img2dataset works fine. Which behaviour you get depends on the library the crawler was built with, and you do not get to choose. That is the reason to write names that are correct under both: letters, underscores and hyphens only, nothing else on the line.

Other free tools here