How to rank in AI search: ChatGPT, Perplexity and Google AI Overviews

An answer engine does not return ten links. It searches, then cites the few pages that answered the question. Here is what each engine documents about how it finds pages, then the work that helps with all three.

What ranking means when the result is an answer

Every AI search product works in two stages. First it retrieves: the engine turns the question into searches and runs them against an index, which returns candidate pages. Then it selects: a model reads those candidates and pulls out the passages that answer the question, and the citations go to the pages those passages came from. A page that ranks well but buries the answer under background wins the first stage and loses the second.

So AI search optimization has two halves: be retrievable, which is mostly ordinary SEO plus a few crawler rules, and then be the page whose text answers the question most directly.

How each engine finds pages

ChatGPT search reads the web with OAI-SearchBot, which OpenAI documents as the crawler behind its search index, and OpenAI says search also uses third-party providers, generally understood to mean Bing. ChatGPT-User fetches a page live when a user asks about it. So two indexes matter: OpenAI's own and Bing's.

Perplexity builds its index with PerplexityBot and fetches pages live as Perplexity-User. Its bot documentation states that Perplexity-User generally ignores robots.txt because a user requested the page, so a block on PerplexityBot keeps you out of the index while a block on Perplexity-User will mostly be ignored.

Google AI Overviews and AI Mode are served from Google's normal index, built by Googlebot. Google's documentation says its AI features use the same crawling and ranking systems as Search. Google-Extended is a separate token that controls only Gemini training, and Google says it does not affect Search or AI Overviews. Blocking Googlebot removes you from all of it.

1. Get crawled

A crawler follows the most specific robots.txt group that names it. If none does, the catch-all group (User-agent: *) applies. The common mistake is a file that names a few training crawlers and then disallows everything in the catch-all, so every search crawler not named is blocked while Googlebot, which nearly everyone names, gets through.

In my September census of the top 5,000 sites, 2,778 served a robots.txt. Of those, 393 block at least one AI search crawler and 726 block at least one training crawler, and 305 of the 393 still allow Googlebot, which is the catch-all pattern at work. OAI-SearchBot is blocked by 262 sites (9.43%), PerplexityBot by 382 (13.75%) and Claude-SearchBot by 267 (9.61%), against 95 for Googlebot (3.42%) and 118 for bingbot (4.25%). The full table is on the data page.

To opt out of training and stay in search, name the search crawlers and leave the catch-all open:

User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: *
Allow: /

Googlebot needs no line of its own.

2. Get read

A crawler reads what the server sends. If that is an empty shell that JavaScript fills in later, a crawler that does not run JavaScript sees nothing. Google documents that Googlebot renders JavaScript, though rendering is queued and can lag. OpenAI and Perplexity have not documented that their crawlers run it, so assume they do not.

In the census, 16.0% of 607 top-site homepages sent under 250 characters of readable text without JavaScript, and another 4.8% sent under 800. View the raw source of your page and search for a sentence from it. If it is not there, the page is built in the browser, and server-side rendering fixes it. The checker on the home page runs this test.

3. Get chosen

Ordinary SEO stops here. A model reading your page is looking for the sentence that answers the question, so put it in the first paragraph. Give each page one question to answer. Write headings in the words people ask with, since the engine's searches are built from the question. Include the details that make an answer usable: numbers, names, dates and the actual steps. Show a visible date so the engine can judge freshness.

The strongest position is being the original source of a fact. When a number exists on your page first and elsewhere as a paraphrase, the engines tend to cite the origin. In my experience, publishing even a small dataset earns more citations than summarizing other people's.

4. Get indexed

None of this matters for an unindexed page. List a sitemap in robots.txt. For ChatGPT and Bing, submit new and changed pages through IndexNow, which feeds Bing, and register in Bing Webmaster Tools. Google Search Console does the same for Google. Perplexity has no submission tool that I know of, so PerplexityBot depends on your sitemap and internal links.

5. Measure

Ask each engine, with search on, a question your page answers, and note whether your site is cited. Repeat monthly, since answers vary. In your server logs, look for OAI-SearchBot and PerplexityBot, which show you are indexed, and ChatGPT-User and Perplexity-User, which show a question led the engine to your page. In analytics, referrals from chatgpt.com and perplexity.ai appear when a reader clicks a citation.

What does not work

Keyword stuffing does nothing. The models read the way a careful person does, and text padded to match queries reads as spam to them too. Hidden text is a trick search engines learned to ignore long ago.

llms.txt is a proposed convention that none of the three engines has said it reads when choosing what to cite. It does no harm, and there is a generator and validator here, but it will not rank you.

Schema markup on its own will not either. It helps an engine understand a page it can already read, and a page nobody can read stays invisible with perfect schema.