What is Perplexity-User and should you block it?

Perplexity-User is not a crawler in the usual sense. It fetches one page because a person using Perplexity asked about it.

What Perplexity-User does

Perplexity says Perplexity-User visits a web page when a user asks a question that needs it, and that it is not used for crawling the web or collecting training content.

On robots.txt: Perplexity-User generally ignores robots.txt. Perplexity's wording: "Since a user requested the fetch, this fetcher generally ignores robots.txt rules." A robots.txt line for it is documentation, not a control.

What blocking it costs

The person who asked gets nothing from your page, or gets the answer from a competitor's page instead. There is no index to fall out of, and no training to opt out of; the only thing you block is a reader who was already interested enough to ask.

Note that a robots.txt line may not stop it anyway, for the reason the vendor gives: the fetch was requested by a person. If you need it stopped, the enforcement is a firewall rule on the user-agent string or the published address ranges, not robots.txt.

How many of the top 5,000 sites block it

In my September 2026 census of the Tranco top 5,000, 2,778 sites returned a robots.txt. 275 of them block Perplexity-User (9.9%), which makes it the 15th most-blocked of the 20 crawlers I checked. 208 files mention it by name; the rest of the mentions either allow it explicitly or restrict only part of the site.

For comparison, 384 sites block at least one of the three on-demand fetchers. Since two of the three vendors say robots.txt may not apply to these anyway, much of that blocking is on paper only. The full census is here.

The robots.txt lines

The line, for what it is worth given the vendor's note above:

User-agent: Perplexity-User
Disallow: /

How to verify a request really came from Perplexity-User

Anyone can put the string Perplexity-User in a request. Perplexity publishes the addresses it uses at www.perplexity.com/perplexity-user.json. A request from outside those ranges is an impostor, and impostors using well-known crawler names are common enough that blocking by name alone catches the honest crawler and misses the rest.

Source: Perplexity's crawler documentation. Census: the September 2026 robots.txt census, raw data at /data/census.json.

Other Perplexity agents: PerplexityBot. All 20 crawlers. Check your own robots.txt against all of them with the checker.