Can AI read dune.com?

dune.com allows the AI search crawlers in its robots.txt, but the server's answer to a test request that named a crawler was inconclusive, so this check cannot say whether ChatGPT and Perplexity are able to read it. Read live from the site's own robots.txt with Google's reference parser, plus a user-agent test at the server.

?AI search crawlers can reach it
OAI-SearchBot and PerplexityBot are allowed in robots.txt. The server turned away a test request that named an AI search crawler, and also one that named Googlebot. It probably refuses any request that claims to be a crawler but does not come from that crawler's own network. The real crawlers do come from their own networks, so this test cannot tell whether they get in.
✓Homepage is readable by crawlers
The page ships real text in its HTML, so an AI crawler can read it.
✓Has an llms.txt
Found /llms.txt. You can check it against the spec at https://ai-visibility.lastminutedealshq.com/llms-txt
About the crawlers named here: OAI-SearchBot, PerplexityBot. Each page has what the vendor says it does, the robots.txt lines, how many of the top 5,000 sites block it and how to verify a request.

What an undecided result means. The server turned away the test request that named an AI search crawler and the one that named Googlebot. That usually means a rule that refuses any crawler claim that does not come from the crawler's own network, and the real crawlers do come from their own networks, so this check cannot say whether they get in. Of the 119 sites in my 5,000-site census that refused a request naming an AI search crawler on September 23, 53 also refused Googlebot's name. Your server log settles it: look for OAI-SearchBot and PerplexityBot and the status they got. How the census test worked.

Want a daily check on this site?

Daily monitoring for dune.com: a re-check every day of its AI-crawler access, readable text and llms.txt. Each result and each change shows on a private status page that also has an Atom feed. You also get an email when a watched status changes. $19/mo after a 7-day free trial, cancel anytime. See a live example of the status page.

Start free 7-day trial
Add this badge to your site:
Want to be counted for the October re-run?
Share this result: Post on X ·

Related guides

All of them are on the guides page.

Check your own site or see the crawler census.

Do you fix AI search visibility for clients? A list of sites whose server refuses the ChatGPT or Perplexity search crawler comes with a free 25-row sample.