# AI visibility checker > Free tools that show whether ChatGPT, Perplexity and Google's AI answers can read and cite a website, and that generate or validate an llms.txt. Built by Reese Calder, backed by a 5,000-site crawler census. ## Tools - [AI visibility checker](https://ai-visibility.lastminutedealshq.com/): checks a site's robots.txt for the AI search crawlers (OAI-SearchBot, PerplexityBot), whether the homepage ships readable text, and whether an llms.txt exists. - [aicheck, the command-line version](https://ai-visibility.lastminutedealshq.com/tools/aicheck.py): a single-file MIT Python CLI running the same checks locally, with no dependencies, exiting non-zero so it can run in CI. - [Which well-known sites block AI search crawlers](https://ai-visibility.lastminutedealshq.com/who-blocks-ai-search): a check of 32 recognizable sites showing which block ChatGPT's and Perplexity's search crawlers. - [Common Crawl checked 584,107 llms.txt files. Most of them are not doing what site owners think.](https://ai-visibility.lastminutedealshq.com/guides/common-crawl-llms-txt-study): A new Common Crawl analysis of 584,107 llms.txt files found many are template-generated, link-free, or trying to write unenforceable robots.txt-style rules into a file that has no enforcement mechanism. - [Which AI crawlers actually ignore robots.txt: Bytespider, DeepSeekBot and LAIONDownloader checked](https://ai-visibility.lastminutedealshq.com/guides/ai-crawlers-that-ignore-robots-txt): Bytespider, DeepSeekBot and LAIONDownloader checked: two are companies with no published crawler policy at all, and one may have no single operator to comply in the first place. - [The BEGIN Cloudflare Managed Content block in your robots.txt, and what it does to rules you already had](https://ai-visibility.lastminutedealshq.com/guides/cloudflare-managed-content-robots-txt): Cloudflare's block always goes first in the file, not merged with your own rules. What that means if you already had a rule for the same crawler, and what to check on your own served file. - [Shopify and llms.txt: what you already have, and how to change it](https://ai-visibility.lastminutedealshq.com/guides/shopify-llms-txt): Every Shopify store already serves a default agents.md, llms.txt and llms-full.txt as of May 28, 2026. No app needed to customize it: the three Liquid template files and the fallback order. - [The AI search visibility checklist](https://ai-visibility.lastminutedealshq.com/guides/ai-search-optimization-checklist): A working, in-order checklist for AI search visibility: crawler access, AI Overviews, readable content, firewall blocks, answer placement, and where llms.txt actually fits. - [llms.txt vs robots.txt: what each one actually controls](https://ai-visibility.lastminutedealshq.com/guides/llms-txt-vs-robots-txt): Two small text files people keep confusing: one is a permission gate, the other is a summary card handed over after permission is already granted. - [Is Content-Signal in robots.txt a syntax error? What Google's own tools disagree about](https://ai-visibility.lastminutedealshq.com/guides/content-signal-robots-txt-error): Cloudflare added this line to millions of sites automatically. It is valid under RFC 9309. Lighthouse and PageSpeed Insights had a confirmed bug flagging it anyway, fixed in 2025; whether Search Console's separate checker is fixed is still unconfirmed. - [Cloudflare will block AI crawlers by default from September 15, and it can take Googlebot with it](https://ai-visibility.lastminutedealshq.com/guides/cloudflare-ai-crawler-defaults): Starting September 15, new domains on Cloudflare block AI training crawlers by default on ad pages, and that default also blocks Googlebot, Applebot and BingBot. - [Does llms.txt support multiple languages? What to actually do on a multi-language site](https://ai-visibility.lastminutedealshq.com/guides/llms-txt-multilingual-sites): The spec has no language field. Three real structures for a multi-language site, and why a per-locale path like /fr/llms.txt does not work the way people expect. - [Does the noindex meta tag stop AI training?](https://ai-visibility.lastminutedealshq.com/guides/does-noindex-block-ai-training): What actually controls AI training opt-out when a whole-domain robots.txt rule is not granular enough, and why the noai meta tag is not a working answer. - [Why your robots.txt User-agent line may not be doing anything](https://ai-visibility.lastminutedealshq.com/guides/robots-txt-user-agent-not-working): RFC 9309 forbids digits and dots in a crawler name and Google's parser truncates at the first one, so AI2Bot reads as AI. 582 of the top 5,000 sites carry a name that does not survive. - [What is llms.txt, and do you actually need one?](https://ai-visibility.lastminutedealshq.com/guides/what-is-llms-txt): What the file is, what the format actually requires, and honest numbers on who has one: 373 of the 4,999 sites I checked, or 7.46 percent. - [Why your site is not showing up in ChatGPT](https://ai-visibility.lastminutedealshq.com/guides/not-showing-up-in-chatgpt): The causes in the order they actually bite, from a blocked crawler to a homepage that only renders in JavaScript. - [How to show up in Perplexity](https://ai-visibility.lastminutedealshq.com/guides/show-up-in-perplexity): PerplexityBot is the most blocked AI search crawler I measured, at 358 sites, so Perplexity is the engine you are likeliest to be invisible to. - [Google AI Overviews: what actually controls whether you appear](https://ai-visibility.lastminutedealshq.com/guides/google-ai-overviews): Google-Extended only controls Gemini training. Googlebot is what decides AI Overviews, and 457 sites block the one that does not matter. - [llms.txt generator and validator](https://ai-visibility.lastminutedealshq.com/llms-txt): builds an llms.txt from a site's real pages and sitemap, or validates an existing one against the spec, including link reachability. - [robots.txt checker for AI crawlers](https://ai-visibility.lastminutedealshq.com/robots-txt-check): finds User-agent lines the reference parser cuts short, rules above the first User-agent line that get dropped, agent names no crawler actually uses, and duplicate groups. - [AI crawler census](https://ai-visibility.lastminutedealshq.com/data): which of the top 5,000 sites block each AI crawler (OAI-SearchBot, PerplexityBot, GPTBot, ClaudeBot and 16 more), measured from their robots.txt in September 2026, with the raw data at https://ai-visibility.lastminutedealshq.com/data/census.json. ## Guides - [How to get your website to show up in ChatGPT](https://ai-visibility.lastminutedealshq.com/guides/show-up-in-chatgpt): which OpenAI crawler builds the ChatGPT search index, the robots.txt lines that allow it while still blocking training, why pages that render in JavaScript go missing, and what the cited pages have in common. - [Does ChatGPT see my website?](https://ai-visibility.lastminutedealshq.com/guides/does-chatgpt-see-my-website): three checks, from reading robots.txt as OAI-SearchBot would to what server logs prove. - [How to rank in AI search](https://ai-visibility.lastminutedealshq.com/guides/rank-in-ai-search): how ChatGPT, Perplexity and Google AI Overviews find and cite pages, in five steps. - [Should you block GPTBot?](https://ai-visibility.lastminutedealshq.com/guides/should-i-block-gptbot): what blocking OpenAI's training crawler does and does not control, and the accidental way sites drop out of ChatGPT search. ## AI crawlers - [The 20 AI crawlers, one page each](https://ai-visibility.lastminutedealshq.com/crawlers): what each crawler is for, what blocking it costs, how many of the top 5,000 sites block it, and the robots.txt lines. - [What is OAI-SearchBot and should you block it?](https://ai-visibility.lastminutedealshq.com/crawlers/oai-searchbot) - [What is PerplexityBot and should you block it?](https://ai-visibility.lastminutedealshq.com/crawlers/perplexitybot) - [What is Claude-SearchBot and should you block it?](https://ai-visibility.lastminutedealshq.com/crawlers/claude-searchbot) - [What is ChatGPT-User and should you block it?](https://ai-visibility.lastminutedealshq.com/crawlers/chatgpt-user) - [What is Perplexity-User and should you block it?](https://ai-visibility.lastminutedealshq.com/crawlers/perplexity-user) - [What is Claude-User and should you block it?](https://ai-visibility.lastminutedealshq.com/crawlers/claude-user) - [What is GPTBot and should you block it?](https://ai-visibility.lastminutedealshq.com/crawlers/gptbot) - [What is ClaudeBot and should you block it?](https://ai-visibility.lastminutedealshq.com/crawlers/claudebot) - [What is Google-Extended and should you block it?](https://ai-visibility.lastminutedealshq.com/crawlers/google-extended) - [What is Applebot-Extended and should you block it?](https://ai-visibility.lastminutedealshq.com/crawlers/applebot-extended) - [What is CCBot and should you block it?](https://ai-visibility.lastminutedealshq.com/crawlers/ccbot) - [What is Bytespider and should you block it?](https://ai-visibility.lastminutedealshq.com/crawlers/bytespider) - [What is meta-externalagent and should you block it?](https://ai-visibility.lastminutedealshq.com/crawlers/meta-externalagent) - [What is Amazonbot and should you block it?](https://ai-visibility.lastminutedealshq.com/crawlers/amazonbot) - [What is cohere-ai and should you block it?](https://ai-visibility.lastminutedealshq.com/crawlers/cohere-ai) - [What is Diffbot and should you block it?](https://ai-visibility.lastminutedealshq.com/crawlers/diffbot) - [What is omgili and should you block it?](https://ai-visibility.lastminutedealshq.com/crawlers/omgili) - [What is TimpiBot and should you block it?](https://ai-visibility.lastminutedealshq.com/crawlers/timpibot) - [What is Googlebot and should you block it?](https://ai-visibility.lastminutedealshq.com/crawlers/googlebot) - [What is bingbot and should you block it?](https://ai-visibility.lastminutedealshq.com/crawlers/bingbot) - [GEO vs SEO vs AEO: what generative engine optimization actually means](' + SELF + '/guides/geo-vs-seo): Where the term GEO came from, the 2023 paper that coined it, and what actually controls it on a real site. ## About - [Reese Calder](https://lastminutedealshq.com/about): who builds these and the public record of shipped work.