Can AI read fox17.com?

Your robots.txt allows OAI-SearchBot, but a test request that named it in its user agent was refused by the server (HTTP 403) while a normal request got the page. If that rule matches on the user agent, the real crawler is turned away too. Check your CDN or firewall bot settings, for example Cloudflare's AI Crawl Control.

If you want this changed, I can write the exact fix for fox17.com and confirm it works once it is live. It is paid work, quoted up front before anything starts. Email me at reese@lastminutedealshq.com.

What the refusal means. The checker requested this homepage naming OAI-SearchBot and was turned away, while the same request naming Googlebot and one naming no crawler were served, so the refusal follows the crawler's name. If that holds from the crawler's own network too, it cannot read this page to build its answers, so this page gets no citations from whichever engine that crawler serves, no matter what robots.txt allows. On September 23 I ran the same test across my 5,000-site census: 2,429 sites served a browser and allowed at least one AI search crawler in robots.txt, and 56 of them (2.3%) refused a request naming one while serving both controls. The test comes from one IP address, so a site that lets the real crawler in from its own network and refuses the name from anywhere else looks the same; your server log settles it. The plain request that reached this homepage was served by Cloudflare, so the fix is usually in the CDN or firewall bot settings there, for example Cloudflare's AI Crawl Control: allow this crawler by name instead of blocking it, or better, allow it by the crawler's published IP ranges. How the census test worked.

Checked 2026-10-10 18:29 UTC (this page is cached up to 10 minutes, so run it yourself for the current moment). To see it from your own machine, run these two and compare the numbers. The first sends the user agent string OAI-SearchBot publishes, the second a Chrome one:

curl -s -o /dev/null -w "%{http_code}" -A "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot" https://fox17.com/; echo

curl -s -o /dev/null -w "%{http_code}" -A "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36" https://fox17.com/; echo

403 for the first and 200 for the second means something at the CDN, firewall, host or a security plugin refuses that crawler by name. If both print 200 it did not reproduce from your network, and if your logs show the real crawler getting 200, ignore this page.

Want a daily check on this site?

Daily monitoring for fox17.com: a re-check every day of its AI-crawler access, readable text and llms.txt. Each result and each change shows on a private status page that also has an Atom feed. You also get an email when a watched status changes. $19/mo after a 7-day free trial, cancel anytime. See a live example of the status page.