How to show up in Perplexity

PerplexityBot is blocked by more sites than any other AI search crawler in my top-5,000 census, and robots.txt is the first place to check for it.

Perplexity gets blocked more than any other AI search crawler

On 2026-09-07 I requested robots.txt from the Tranco top 5,000 sites. 4,999 were checked and 2,771 of them served an actual robots.txt file. Every percentage here uses those 2,771 as the base.

That is 118 more sites blocking PerplexityBot than OAI-SearchBot, roughly 49 percent more. Both figures are shares of the sites that served a file. The other 2,235 sites in the census served no robots.txt at all, which means crawling is allowed on them by default. These numbers describe a population, so they do not tell you the state of any one domain. The full per-crawler breakdown is on the census page, and you can check a single domain in a few seconds with the checker.

PerplexityBot and Perplexity-User are two different things

Perplexity operates two user agents and they do separate jobs.

PerplexityBot crawls pages ahead of time so they can be retrieved and cited when someone asks a relevant question. Blocking it removes your pages from what Perplexity is able to retrieve and cite. My census measures the block itself, and it does not measure referral traffic or citation counts, so I am not going to put a number on what the block costs.

Perplexity-User fetches a page at the moment a user's question points directly at it, for example when someone pastes your URL. Perplexity's own documentation describes this agent as user-triggered and says it is generally not governed by robots.txt rules, so a robots.txt line is not a control you should rely on for it in either direction. My census recorded robots.txt contents only and says nothing about how that agent behaves, so check Perplexity's current documentation before you act on this one.

Find out what your robots.txt actually says

Two patterns from the census are worth having in mind while you look. 296 sites block at least one of the three AI search crawlers I checked, meaning OAI-SearchBot, PerplexityBot, or Claude-SearchBot, while still allowing Googlebot. 73 sites block an AI search crawler and Googlebot together. The census records the contents of each robots.txt file. No field in it captures who added a rule or why, so I cannot tell from this data whether a particular block was intended.

A wildcard group applies to Perplexity unless a more specific group exists, so a broad Disallow: / under User-agent: * keeps PerplexityBot out. A missing robots.txt means crawling is allowed. A 404, or a 200 with an empty body, leaves everything open, so an absent file is not the thing you are looking for.

The robots.txt fix

User-agent: PerplexityBot
Allow: /

Robots.txt matching picks the most specific user-agent group, so a named group like this one overrides a wildcard block for that agent. If you still want model training crawlers kept out, leave your GPTBot and ClaudeBot rules exactly where they are. Those are separate decisions, and I work through them in should I block GPTBot.

A WAF rule or a bot-management setting can reject Perplexity at the edge before robots.txt matters at all, and that block never appears in the file. If your robots.txt looks correct and you are still getting nothing, check the firewall rules and grep the server logs for the Perplexity agents.

The page has to be readable as plain HTML

On 2026-08-31 I fetched 900 homepages from Tranco ranks around 10,000 to 10,700 and got 607 of them back. Each was fetched once with JavaScript switched off. 481 (79.2%) came back readable. 97 (16.0%) returned under 250 characters of text. Another 29 (4.8%) came back under 800 characters. 58.0% carried no JSON-LD structured data and 26.4% had no meta description.

The measurement does not say why those 97 came back nearly empty. A page that assembles its content in the browser looks like that, and so does a bot wall or a redirect away from the URL I requested. I also do not know how much JavaScript Perplexity runs on any given fetch and I am not going to guess at it. The safe assumption is that the first request sees your served HTML and nothing more, so the sentences you want quoted need to be in that markup. If your product description only exists after a framework hydrates, treat it as a visibility problem. The same reasoning applies across every engine, which I cover in how to rank in AI search.

Some sites block Perplexity on purpose

A block can be a deliberate policy decision. My census reads robots.txt files and records no intent, so I cannot separate a considered choice from an inherited rule using this data. Before you copy anyone else's robots.txt, it is worth seeing which well-known sites block AI search crawlers, which I list on who blocks AI search, with per-agent detail on the crawler reference pages.

llms.txt is optional

In the same 2026-09-06 census, 373 sites (7.46%) served a valid llms.txt. Adoption stays under 12% in every rank band, and the share falls as you move down the list: 11 of the top 100 (11%), 75 of ranks 101 to 1000 (8.33%), and 287 of ranks 1001 to 5000 (7.18%). Cloudflare has one at rank 2, carrying 90 links and an llms-full.txt alongside it.

No major AI search engine has confirmed that llms.txt is used as a ranking or citation input, and it is not required for you to be cited today. I treat it as cheap and unproven. If you want one, the generator and validator will build it and check it, and it belongs at the bottom of your list.

Do not mix this up with the training crawlers

The three most-blocked crawlers in the census are all training crawlers: GPTBot at 535 sites (19.31%), ClaudeBot at 516 (18.62%), and Google-Extended at 459 (16.6%). Blocking those has no effect on whether you get cited anywhere. Google-Extended governs Gemini model training only and has no bearing on Google Search or AI Overviews, which Googlebot controls. GPTBot is OpenAI training only, while OAI-SearchBot is what feeds ChatGPT's citations. PerplexityBot sits on the citation side of that split, so a block there takes you out of answers.

If ChatGPT is also on your list, the ChatGPT guide covers the OAI-SearchBot side and does ChatGPT see my website walks through testing one domain. Everything else is indexed at the guide index.

The order I would work in:

  • Read your live robots.txt and search it for PerplexityBot and for any wildcard Disallow that would apply to it.
  • Add the named Allow group above if a block exists, then confirm the file you edited is the file being served.
  • Check the CDN or WAF for a managed AI-bot rule that robots.txt cannot override.
  • Load your own page with JavaScript disabled and read what actually comes back.
  • Leave llms.txt until the four steps above are finished.