What is Applebot-Extended and should you block it?

Applebot-Extended is a robots.txt token from Apple. It controls what Apple may do with pages that Applebot fetched; it does not fetch anything itself.

What Applebot-Extended does

Apple says "Applebot-Extended does not crawl webpages" and "is only used to determine how to use the data crawled by the Applebot user agent". Disallowing it opts your content out of training Apple's general purpose foundation models.

On robots.txt: Applebot-Extended is honoured as a robots.txt control token; Applebot does the fetching for Spotlight, Siri and Safari.

Apple's wording on the cost: "Webpages that disallow Applebot-Extended can still be included in search results." So the opt-out is free as far as Apple's search features go.

What blocking it costs

It costs nothing in search. Disallowing Applebot-Extended tells Apple not to use your pages for the purposes the token covers. The pages are still fetched by Applebot and still appear wherever Applebot normally shows them.

How many of the top 5,000 sites block it

In my September 2026 census of the Tranco top 5,000, 2,778 sites returned a robots.txt. 432 of them block Applebot-Extended (15.6%), which makes it the 8th most-blocked of the 20 crawlers I checked. 448 files mention it by name; the rest of the mentions either allow it explicitly or restrict only part of the site.

For comparison, 726 sites (26.1%) block at least one training crawler, against 393 (14.1%) that block at least one AI search crawler. Most sites that opt out of training are doing it deliberately; most sites that block AI search are not. The full census is here.

The robots.txt lines

To opt out of the uses the token covers, with no effect on search:

User-agent: Applebot-Extended
Disallow: /

How to verify a request really came from Applebot-Extended

Anyone can put a name in a user-agent string. Since Applebot-Extended is a token and Applebot does the fetching, verify the fetcher: Apple publishes the address ranges at support.apple.com/en-us/119829. A request from outside them is not Applebot.

Source: Apple's crawler documentation. Census: the September 2026 robots.txt census, raw data at /data/census.json.

All 20 crawlers. Check your own robots.txt against all of them with the checker.