The AI search visibility checklist

Everything below is covered in more depth elsewhere on this site. This page exists for people who want the order of operations in one place, checked off, rather than another explainer. Work top to bottom. The first three items matter far more than everything after them.

1. Allow the crawlers that feed citations

Check your robots.txt for these three by name. Training crawlers (GPTBot, ClaudeBot, Google-Extended) are a separate decision that does not affect citations either way.

User-agent: OAI-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

A wildcard User-agent: * group with a broad Disallow catches these unless a named group overrides it. In my census of the Tranco top 5,000, 241 sites block OAI-SearchBot, 359 block PerplexityBot, and 369 block at least one of the three AI search crawlers. 296 of the 369 still allow Googlebot, usually by accident. Should you block GPTBot and how to show up in Perplexity cover the training-versus-search distinction in full. The free checker reads this for you in a few seconds.

2. Check Googlebot separately from Google-Extended

Google-Extended controls Gemini training only and has no bearing on Google Search or AI Overviews. Googlebot is what actually decides both. Blocking Google-Extended is a reasonable training opt-out that changes nothing about AI Overviews eligibility, and 459 sites in the census block it while presumably not intending to touch AI Overviews at all. What actually controls AI Overviews covers this and the Search Console toggle Google added for opting out of its AI features specifically, without leaving Google Search.

3. Make sure a crawler sees text, not a blank page

Load your homepage with JavaScript switched off, or curl it, and read what comes back. In a crawl of 607 real homepages this way, 16.0% returned under 250 characters of text and 58.0% carried no JSON-LD structured data. Neither OpenAI nor Perplexity has said its crawlers run JavaScript, so the safe assumption is that they do not. If your content only exists after a framework hydrates, a crawler that can reach your page still gets nothing from it. Does ChatGPT see my website walks through testing this on your own domain.

4. Check for a firewall block robots.txt cannot show you

A WAF rule or a bot-management setting can reject an AI crawler at the edge before robots.txt is ever read, and that block never shows up in the file. If your robots.txt looks correct and the checker says you are allowed but nothing is happening, check your CDN or firewall's bot-management rules next, and grep your server logs for the crawler's user agent directly.

5. Put the answer in plain text, near the top

The pages that get quoted tend to state the specific answer directly, in the first few hundred words, with the concrete details that survive being lifted out of context: numbers, names, dates, the actual steps. A page that opens with three paragraphs of scene-setting, or keeps the real answer inside an image, gets passed over for one that commits to something immediately. How to rank in AI search goes through this in more depth.

6. Treat llms.txt as optional, and last

Google has confirmed outright that Search, AI Overviews included, does not use llms.txt. Perplexity, OpenAI and Anthropic have not said anything either way. In the census, 7.46% of sites had a valid one, and adoption falls the further down the rank list you go. It costs little to add and will not fix anything on its own, so do it after the five steps above, not instead of them. The generator and validator builds one from your real pages and checks the format of one you already have. What is llms.txt, and do you actually need one has the full case for and against.

7. Re-check periodically

This is not a set-and-forget list. Platforms change what they do here without much notice: Google added a dedicated AI Overviews opt-out in Search Console in 2026, Cloudflare set new default AI-crawler blocks for September 15, 2026 that can quietly take Googlebot with them on sites that are not careful, and crawler identities and behavior have shifted before and will again. Re-run the checker every so often rather than assuming a setup that was correct once stays correct.

Run the free checker to see where your own site stands on the first four items right now. The rest of the writing is at the guide index.