Guides

Plain answers to the questions people actually search about AI search. Written from the census data and from checking real sites, and updated when the crawlers change.

Start here

• How to get your website to show up in ChatGPT. The crawler that matters, the robots.txt lines that let it in, what the cited pages have in common, and how to tell it worked.
• Does ChatGPT see my website? Three checks, from a one-minute look at robots.txt to what your server logs prove.
• Six free AI crawler checkers, tested on a site whose server refuses AI crawlers. What each one printed for a test site that allows every crawler in robots.txt and answers 403 to eight of them.
• GEO vs SEO vs AEO: what generative engine optimization actually means. Where the term GEO came from, the paper that coined it, and what actually controls it on a real site.
• How to rank in AI search. How ChatGPT, Perplexity and Google AI Overviews find and cite pages, and the five things a site has to get right.
• Should you block GPTBot? What blocking it does and does not control, and the mistake that takes sites out of ChatGPT search by accident.
• What is llms.txt, and do you actually need one?. What the file is, what the format actually requires, and honest numbers on who has one: 385 of the 4,999 sites I checked, or 7.7 percent.
• Why your site is not showing up in ChatGPT. The causes in the order they actually bite, from a blocked crawler to a homepage that only renders in JavaScript.
• How to show up in Perplexity. PerplexityBot is the most blocked AI search crawler I measured, at 358 sites, so Perplexity is the engine you are likeliest to be invisible to.
• Google AI Overviews: what actually controls whether you appear. Google-Extended only controls Gemini training. Googlebot is what decides AI Overviews, and 457 sites block the one that does not matter.
• Why your robots.txt User-agent line may not be doing anything. RFC 9309 forbids digits and dots in a crawler name and Google's parser truncates at the first one, so AI2Bot reads as AI. 582 of the top 5,000 sites carry a name that does not survive.
• Does the noindex meta tag stop AI training?. What actually controls AI training opt-out when a whole-domain robots.txt rule is not granular enough, and why the noai meta tag is not a working answer.
• Does llms.txt support multiple languages? What to actually do on a multi-language site. The spec has no language field. Three real structures for a multi-language site, and why a per-locale path like /fr/llms.txt does not work the way people expect.
• Cloudflare's AI crawler settings since September 15, and the two that also block Googlebot. Since September 15, Cloudflare's Disallow AI Training setting keeps Googlebot, Bingbot and Applebot in search, while its Block settings for Training stop them too. New ad-supported domains start on Disallow AI Training.
• Is Content-Signal in robots.txt a syntax error? What Google's own tools disagree about. Cloudflare added this line to millions of sites automatically. It is valid under RFC 9309. Lighthouse and PageSpeed Insights had a confirmed bug flagging it anyway, fixed in 2025; whether Search Console's separate checker is fixed is still unconfirmed.
• llms.txt vs robots.txt: what each one actually controls. Two small text files people keep confusing: one is a permission gate, the other is a summary card handed over after permission is already granted.
• The AI search visibility checklist. A working, in-order checklist for AI search visibility: crawler access, AI Overviews, readable content, firewall blocks, answer placement, and where llms.txt actually fits.
• Shopify and llms.txt: what you already have, and how to change it. Every Shopify store already serves a default agents.md, llms.txt and llms-full.txt as of May 28, 2026. No app needed to customize it: the three Liquid template files and the fallback order.
• The BEGIN Cloudflare Managed Content block in your robots.txt, and what it does to rules you already had. Cloudflare's block always goes first in the file, not merged with your own rules. What that means if you already had a rule for the same crawler, and what to check on your own served file.
• Which AI crawlers actually ignore robots.txt: Bytespider, DeepSeekBot and LAIONDownloader checked. Bytespider, DeepSeekBot and LAIONDownloader checked: two are companies with no published crawler policy at all, and one may have no single operator to comply in the first place.
• Common Crawl checked 584,107 llms.txt files. Most of them are not doing what site owners think.. A new Common Crawl analysis of 584,107 llms.txt files found many are template-generated, link-free, or trying to write unenforceable robots.txt-style rules into a file that has no enforcement mechanism.
• The 296 sites that block AI search but not Google: deliberate or accidental?. A reader's critique of my own headline number, tested by rank: household names look deliberate, the gradient is modest, and the one confirmed reply this census ever received came from outside the top 1,000.
• Cloudflare retired Managed robots.txt on September 15. Most sites that used it now block no AI crawlers in robots.txt. Cloudflare swapped Managed robots.txt for Bot Preference Sync on September 15. What happened to the sites that used it, measured.
• Cloudflare AI Crawl Control: what it costs and what it blocks on real sites. What Cloudflare's AI Crawl Control does, what each plan gets, the GoDaddy version, and a test of 2,429 top sites showing which AI search crawlers Cloudflare sites refuse at the server.

Tools that go with them

• AI visibility checker. Runs the robots.txt and readability checks on any site.
• llms.txt generator and validator.
• AI crawler census. The numbers the guides quote, with the raw JSON.
• The 20 AI crawlers, one page each. What each one does, what blocking it costs, and how many sites block it.