Can AI read crawler-refusal-demo.lastminutedeals.workers.dev?

crawler-refusal-demo.lastminutedeals.workers.dev allows the AI search crawlers in its robots.txt. But its server refused a test request that named OAI-SearchBot and PerplexityBot in the user agent, so the real crawlers may be turned away before robots.txt matters. Read live from the site's own robots.txt with Google's reference parser, plus a user-agent test at the server.

✗AI search crawlers can reach it
robots.txt allows them, but the server refused a test request naming OAI-SearchBot, PerplexityBot.
✓Homepage is readable by crawlers
The page ships some text in its HTML (508 characters), so an AI crawler can read it, but that is thin. In my census of 607 homepages, 79.2% sent 800 characters or more. More real text in the HTML gives a crawler more to quote.
✗Has an llms.txt
No /llms.txt. It is an emerging, optional convention, not required to be cited today. You can generate one at https://ai-visibility.lastminutedealshq.com/llms-txt
About the crawlers named here: OAI-SearchBot, PerplexityBot. Each page has what the vendor says it does, the robots.txt lines, how many of the top 5,000 sites block it and how to verify a request.

Your robots.txt allows OAI-SearchBot and PerplexityBot, but a test request that named them in its user agent was refused by the server (HTTP 403) while a normal request got the page. If that rule matches on the user agent, the real crawler is turned away too. Check your CDN or firewall bot settings, for example Cloudflare's AI Crawl Control.

If you want this changed, I can write the exact fix for crawler-refusal-demo.lastminutedeals.workers.dev and confirm it works once it is live. It is paid work, quoted up front before anything starts. Email me at reese@lastminutedealshq.com.

What the refusal means. The checker requested this homepage naming OAI-SearchBot and PerplexityBot and was turned away, while the same request naming Googlebot and one naming no crawler were served, so the refusal follows the crawler's name. On September 23 I ran the same test across my 5,000-site census: 2,429 sites served a browser and allowed at least one AI search crawler in robots.txt, and 56 of them (2.3%) refused a request naming one while serving both controls. The test comes from one IP address, so a site that lets the real crawler in from its own network and refuses the name from anywhere else looks the same; your server log settles it. How the census test worked.

To see it from your own machine, run these two and compare the numbers. The first sends the user agent string OAI-SearchBot publishes, the second a Chrome one:

curl -s -o /dev/null -w "%{http_code}" -A "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot" https://crawler-refusal-demo.lastminutedeals.workers.dev/; echo
curl -s -o /dev/null -w "%{http_code}" -A "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36" https://crawler-refusal-demo.lastminutedeals.workers.dev/; echo

403 for the first and 200 for the second means something at the CDN, firewall, host or a security plugin refuses that crawler by name. If both print 200 it did not reproduce from your network, and if your logs show the real crawler getting 200, ignore this page.

Want a daily check on this site?

Daily monitoring for crawler-refusal-demo.lastminutedeals.workers.dev: a re-check every day of its AI-crawler access, readable text and llms.txt. Each result and each change shows on a private status page that also has an Atom feed. You also get an email when a watched status changes. $19/mo, cancel anytime. See a live example of the status page.

Monitor this site - $19/mo
Add this badge to your site:
Want to be counted for the October re-run?
Share this result: Post on X ·

Related guides

All of them are on the guides page.

Check your own site or see the crawler census.