Six free AI crawler checkers, tested on a site whose server refuses AI crawlers
I built a small test site. Its robots.txt allows every crawler, but its server answers 403 to eight AI crawler user agents. On September 26, 2026 I ran six free AI crawler checkers on it, once each. All six printed an all-clear for crawler access. Here is what each one said, what it says it reads, and what the server did.
Want your site counted in the October census re-run?
Leave your domain and address to be counted for the October census re-run. I will send one plain-text email with the result for that domain once the re-run is done, and nothing else. The figures go on /data too.
The test site
The site is crawler-refusal-demo.lastminutedeals.workers.dev. Its robots.txt is two lines, User-agent: * and Allow: /. Its server answers 403 to any request whose user agent names GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot or Perplexity-User, from any IP address. A request that names Googlebot, Bingbot or nothing special gets the page. I read the answers with the crawler names in the user agent right before and right after the runs, and they did not change:
| User agent in the request | Homepage answer |
|---|---|
| Chrome (a browser) | 200 |
| ControlBot (a made-up name) | 200 |
| Googlebot | 200 |
| Bingbot | 200 |
| GPTBot | 403 |
| OAI-SearchBot | 403 |
| ClaudeBot | 403 |
| PerplexityBot | 403 |
ChatGPT-User, Claude-SearchBot, Claude-User and Perplexity-User also get 403. Any checker can be pointed at the URL above, so you can repeat this.
What six checkers printed
One run each on that URL, with no account and no email address. The right-hand column is the tool's own description of what it reads.
| Tool | What it printed for the test site | What the tool says it reads |
|---|---|---|
| Hyperleap LLM Bot Checker | "All 15 AI bots can crawl your site", with GPTBot, ClaudeBot, OAI-SearchBot and PerplexityBot each marked ALLOWED | "based on robots.txt" |
| ClayHog Free GEO Audit | "32/32 AI crawlers can access your site", crawler score 100, "All AI crawlers are allowed" | "through robots.txt analysis" |
| InfuseOS AI Search Readiness Checker | "Major AI/search crawlers appear allowed by robots.txt." with PASS beside it | "robots.txt crawler access" |
| Topify AI Robots Checker | "10 Allowed 0 Blocked 0 Not mentioned" and "AI can access your site", with GPTBot, ClaudeBot and PerplexityBot each Allowed | Prints "robots.txt found" above the results |
| Centium AI Crawler Access Checker | "All Allowed", with all 21 crawlers Allowed, GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot among them | "Centium reads your robots.txt file" |
| Provimedia GEO-Check (German) | "robots.txt & KI-Crawler OK", 20 of 20 points, with GPTBot, ClaudeBot and PerplexityBot each "erlaubt" (allowed) | Says it checks seven AI crawlers against your robots.txt |
Every one of them describes its check as a robots.txt check, and for the file on this site they are right: the file allows everything. The headline is what a site owner reads, though. "AI can access your site" and "All 15 AI bots can crawl your site" describe something the server contradicts for four of the crawlers named in those tables.
What my checker printed on the same site
This is the summary line from the result page for the test site: crawler-refusal-demo.lastminutedeals.workers.dev allows the AI search crawlers in its robots.txt. But its server refused a test request that named OAI-SearchBot and PerplexityBot in the user agent, so the real crawlers may be turned away before robots.txt matters.
I built the checker to look for exactly this, so it catches a test site made of nothing else. That shows what one extra request per crawler name changes. It does not show that my checker is better at the other things these tools do (llms.txt, schema, scores), which I did not test.
Why a robots.txt check misses it
Robots.txt is a request that a crawler reads and chooses to obey. Whether the server answers the crawler at all is decided separately, by the host, a CDN, a firewall or a security plugin, and many of those rules match on the crawler's name in the user agent. When the two disagree, the file says welcome and the server says forbidden. A tool that reads only the file reports the welcome. Sending one request per crawler name and printing the status code separates the two cases, and for the eight names above that is eight requests.
How common the mismatch is
On September 23 I ran the same one-request test across my 5,000-site census: 2,429 sites served a browser and allowed at least one AI search crawler in robots.txt, and 56 of them (2.3%) refused a request naming one while serving both controls. Among 422 Cloudflare-served sites that used Cloudflare's old Managed robots.txt block and no longer block any crawler in robots.txt, 336 answered GPTBot and ClaudeBot with a 403 while serving PerplexityBot the page (how that test worked).
Limits of this test
- One run per tool, on one day. Tools change. If one of these starts sending a request per crawler name, tell me the date (reese@lastminutedealshq.com) and I will re-run it and update the table.
- The test site refuses by name from every IP address. Some real sites let the real crawler in from its own network and refuse the same name from everywhere else. A request from one address cannot see that, and neither can any of the tools above. Your server log settles it.
- I only tested the crawler access check. These tools do other things, and I have no view on how well they do them.
- I run a checker of my own, so I am not a neutral judge. The page text and a screenshot of every run are saved, and I will send them to anyone who asks.
Check your own site in two minutes
Run these two and compare the numbers. The first names GPTBot, the second is an ordinary browser:
curl -s -o /dev/null -w "%{http_code}" -A "GPTBot" https://yoursite.com/; echo
curl -s -o /dev/null -w "%{http_code}" -A "Mozilla/5.0 Chrome/124.0" https://yoursite.com/; echo403 for the first and 200 for the second means something in front of your site refuses that crawler by name, whatever your robots.txt says. If both print 200 it did not reproduce from your network. To see the same test for OAI-SearchBot and PerplexityBot, the two crawlers that decide whether ChatGPT search and Perplexity can read a page, run the checker on your domain.
Related: does ChatGPT see my website covers the whole chain from robots.txt to a readable page, and the OAI-SearchBot page has the exact user agent string and how many of the top 5,000 sites block it.