Every site in the top 5,000 that blocks an AI search crawler

368 of the 2,771 sites with a robots.txt block OAI-SearchBot, PerplexityBot or Claude-SearchBot, so they cannot be cited by that engine. The full list with each site's result, from a 2026-09-07 census.

Free, no signup. It reads your robots.txt, then asks your server for your homepage the way the AI search crawlers do, and shows what came back. If you want your site watched, monitoring repeats the check every day, free for 7 days then $19/mo, and keeps the results on a private status page with an Atom feed. You also get an email when a watched status changes.

The shorter list covers 32 names most people recognise. This is all of them. Of the 2,771 sites in the census that served a robots.txt, 368 block at least one of the three crawlers that feed AI answers, and 200 block all three. A red mark means that engine is not allowed to read the site root, so it cannot cite the site. Every domain links to its own live result.

Counted per crawler: OAI-SearchBot, which feeds ChatGPT citations, is blocked by 240 of them. PerplexityBot is blocked by 358. Claude-SearchBot is blocked by 249.

Two things worth knowing before you read anything into a row. Blocking a training crawler such as GPTBot is a separate decision and does not appear here, because it does not affect citations; the GPTBot list is its own page. And the Tranco list ranks raw hostnames, so it includes CDN and API endpoints nobody visits, such as gstatic.com and fbcdn.net. 2 of the 368 rows look like infrastructure rather than sites you would browse. They are real robots.txt results so they stay in the table, but they are not evidence about a publisher.

Measured on 2026-09-07 against each site's live robots.txt, using the same matcher Google publishes, so wildcards and dollar anchors are handled the way a real crawler handles them. The raw per-domain data is on the census page and in the open dataset.

Want your site counted in the October census re-run?

Leave your domain and address to be counted for the October census re-run. Results are posted on /data; nothing is emailed.

Rank	Site	ChatGPT	Perplexity	Claude
3	gstatic.com	✗	✗	✗
4	facebook.com	✗	✓	✗
11	instagram.com	✗	✗	✗
13	fbcdn.net	✗	✗	✗
16	twitter.com	✗	✗	✗
26	amazon.com	✗	✗	✗
32	whatsapp.net	✓	✗	✓
50	x.com	✗	✗	✗
54	tiktok.com	✗	✗	✗
56	whatsapp.com	✓	✗	✓
57	pinterest.com	✗	✗	✗
60	yahoo.com	✓	✗	✓
66	msn.com	✗	✗	✗
76	chatgpt.com	✓	✗	✓
84	baidu.com	✗	✗	✗
105	reddit.com	✗	✗	✗
120	registrar-servers.com	✗	✗	✗
122	amazonvideo.com	✓	✗	✓
153	nytimes.com	✗	✗	✗
193	cnn.com	✗	✗	✗
195	theguardian.com	✓	✗	✗
206	forbes.com	✓	✗	✓
210	t-online.de	✓	✗	✓
211	bbc.com	✗	✗	✓
224	bbc.co.uk	✗	✗	✓
231	t.co	✗	✗	✗
235	ebay.com	✓	✗	✓
254	meraki.com	✗	✗	✗
256	imdb.com	✗	✗	✗
262	launchpad.net	✗	✗	✗
274	reuters.com	✓	✗	✗
287	weibo.com	✗	✗	✗
305	weather.com	✓	✗	✓
323	amazon.co.uk	✗	✗	✗
358	washingtonpost.com	✓	✗	✓
361	wsj.com	✓	✗	✗
365	bloomberg.com	✗	✗	✗
380	amazon.de	✗	✗	✗
407	unsplash.com	✓	✓	✗
434	claude.ai	✗	✓	✓
456	sohu.com	✗	✗	✗
465	edgecdn.ru	✗	✗	✗
470	cnbc.com	✗	✗	✓
487	amazon.co.za	✗	✗	✗
490	quora.com	✗	✗	✗
491	gcdn.co	✗	✗	✗
494	aol.com	✓	✗	✓
497	amazon.co.jp	✗	✗	✗
523	ft.com	✓	✗	✓
541	patreon.com	✓	✗	✓
544	wired.com	✓	✗	✗
562	uol.com.br	✓	✗	✗
567	usatoday.com	✗	✗	✗
597	telegraph.co.uk	✗	✗	✗
611	amazon.fr	✗	✗	✗
613	threads.com	✗	✗	✗
619	sharethrough.com	✗	✗	✗
636	amazon.ca	✗	✗	✗
651	yelp.com	✗	✗	✗
659	elpais.com	✓	✗	✗
660	dailymail.co.uk	✗	✗	✗
666	primevideo.com	✓	✗	✓
668	amazon.in	✗	✗	✗
673	hstgr.net	✗	✗	✗
685	hihonorcloud.com	✗	✗	✗
693	lemonde.fr	✓	✓	✗
694	dailymail.com	✗	✗	✗
695	fb.com	✗	✗	✗
697	blackhub.team	✗	✗	✗
699	cnet.com	✗	✗	✗
700	pinimg.com	✗	✗	✗
703	amazon.es	✗	✗	✗
707	amazon.it	✗	✗	✗
713	quantserve.com	✗	✗	✗
724	spiegel.de	✗	✓	✗
732	creativecdn.com	✗	✗	✗
774	bild.de	✓	✗	✓
780	mercadolivre.com.br	✓	✗	✓
789	apnews.com	✓	✗	✗
790	adform.net	✗	✗	✗
792	welt.de	✓	✗	✓
799	people.com	✓	✗	✗
831	lefigaro.fr	✗	✓	✗
833	theverge.com	✓	✗	✗
845	amazon.com.br	✗	✗	✗
849	amazon.com.au	✗	✗	✗
858	ebay.co.uk	✓	✗	✓
859	nbcnews.com	✗	✗	✗
862	pexels.com	✓	✗	✓
875	ebay.de	✓	✗	✓
940	mailchi.mp	✗	✗	✗
947	mercadolibre.com.ar	✓	✗	✓
948	investopedia.com	✓	✗	✗
958	repubblica.it	✓	✗	✓
979	btloader.com	✗	✗	✗
983	stbid.ru	✗	✗	✗
992	elmundo.es	✗	✗	✓
1008	theatlantic.com	✓	✗	✗
1036	garmin.com	✓	✗	✓
1038	amazon.com.mx	✗	✗	✗
1043	pbs.org	✓	✗	✓
1045	theconversation.com	✗	✗	✗
1057	huffpost.com	✗	✗	✗
1061	ouest-france.fr	✗	✗	✗
1062	as.com	✓	✗	✗
1065	marca.com	✗	✗	✓
1075	bfmtv.com	✗	✗	✗
1089	idnes.cz	✓	✗	✓
1095	franceinfo.fr	✗	✗	✗
1101	namecheaphosting.com	✗	✗	✗
1110	gsmarena.com	✗	✗	✗
1113	cookpad.com	✓	✗	✓
1115	nypost.com	✓	✗	✗
1127	netcraze.io	✗	✗	✗
1133	mercadolibre.com.mx	✓	✗	✓
1136	leparisien.fr	✓	✗	✓
1151	xiaohongshu.com	✗	✗	✗
1155	economist.com	✗	✗	✗
1161	pimpbunny.com	✗	✗	✓
1180	ad.nl	✗	✗	✗
1184	aftonbladet.se	✓	✗	✗
1188	nu.nl	✗	✗	✗
1203	rocketmoney.dev	✗	✗	✗
1204	dcinside.com	✓	✗	✓
1210	lequipe.fr	✗	✗	✗
1212	vg.no	✓	✗	✗
1217	mercadolibre.com	✓	✗	✓
1230	gazzetta.it	✓	✗	✓
1237	furaffinity.net	✗	✗	✗
1247	hln.be	✗	✗	✗
1269	nrk.no	✗	✗	✗
1270	amap.com	✓	✓	✗
1274	ladepeche.fr	✓	✗	✓
1275	abc.net.au	✓	✗	✓
1290	actu.fr	✗	✗	✓
1296	f95zone.to	✗	✗	✗
1305	yr.no	✗	✗	✓
1327	huffingtonpost.com	✗	✗	✗
1344	e621.net	✓	✗	✓
1352	sport.es	✗	✗	✓
1367	dw.com	✗	✗	✓
1382	dagbladet.no	✓	✗	✓
1410	iltalehti.fi	✗	✗	✗
1411	aljazeera.com	✓	✗	✓
1444	ekstrabladet.dk	✗	✗	✗
1459	hdtube.porn	✓	✗	✓
1472	eenadu.net	✗	✗	✗
1490	usnews.com	✗	✗	✓
1499	exhentai.org	✗	✗	✗
1532	aka.ms	✗	✗	✗
1548	buzzfeed.com	✓	✗	✓
1564	ladies.de	✗	✗	✗
1597	tjk.org	✗	✗	✗
1633	artfight.net	✓	✗	✓
1649	keirin.jp	✗	✗	✗
1660	sedoparking.com	✗	✗	✗
1781	threads.net	✗	✗	✗
1809	newsweek.com	✓	✗	✓
1810	qcloud.com	✗	✗	✗
1819	attn.tv	✗	✗	✗
1820	newyorker.com	✓	✗	✗
1823	themoviedb.org	✗	✗	✗
1824	avito.st	✗	✗	✗
1831	vanityfair.it	✓	✗	✗
1851	prokerala.com	✓	✗	✓
1879	marketwatch.com	✓	✗	✗
1885	arstechnica.com	✓	✗	✗
1889	zdnet.com	✗	✗	✗
1924	adrta.com	✗	✗	✗
1927	wikihow.com	✗	✗	✗
1941	iqiyi.com	✓	✗	✓
1973	scientificamerican.com	✓	✗	✓
1990	secureserver.net	✗	✗	✗
2001	thelancet.com	✓	✗	✓
2008	fastcompany.com	✓	✗	✓
2052	lnkd.in	✓	✗	✓
2058	sureserver.com	✗	✗	✗
2064	vtb.ru	✗	✗	✗
2072	clevertap-prod.com	✗	✗	✗
2077	ign.com	✗	✗	✗
2100	sogou.com	✗	✗	✗
2109	timeanddate.com	✓	✗	✓
2116	mashable.com	✗	✗	✗
2138	thesun.co.uk	✓	✗	✗
2152	salesforceliveagent.com	✗	✗	✗
2172	ztomy.com	✗	✗	✗
2211	variety.com	✗	✗	✓
2217	mirror.co.uk	✗	✗	✓
2250	ad.gt	✗	✗	✗
2273	juicyscore.com	✗	✗	✗
2289	vox.com	✓	✗	✗
2301	nba.com	✓	✗	✓
2324	weibo.cn	✗	✗	✗
2330	mparticle.com	✗	✗	✓
2345	inc.com	✓	✗	✓
2382	congress.gov	✗	✗	✗
2395	guardian.co.uk	✓	✗	✗
2402	umd.edu	✓	✗	✓
2403	co.com	✗	✗	✗
2408	tappx.com	✗	✗	✗
2425	rollingstone.com	✗	✗	✓
2438	nejm.org	✓	✗	✓
2452	redditmedia.com	✗	✗	✗
2462	express.co.uk	✗	✗	✓
2466	sfgate.com	✓	✗	✓
2488	pcmag.com	✗	✗	✗
2529	glassdoor.com	✓	✗	✓
2539	news.com.au	✗	✗	✗
2546	hollywoodreporter.com	✗	✗	✓
2550	openssl.org	✗	✗	✗
2553	nextdoor.com	✗	✗	✗
2557	smh.com.au	✗	✗	✓
2568	zeit.de	✓	✗	✓
2571	urldefense.com	✗	✗	✗
2582	technologyreview.com	✗	✗	✓
2615	expressapisv2.net	✗	✗	✗
2624	pxf.io	✗	✗	✗
2630	thehill.com	✗	✗	✓
2640	kuaishou.com	✓	✓	✗
2643	tmdb.org	✗	✗	✗
2670	theglobeandmail.com	✗	✗	✗
2680	smi2.ru	✗	✗	✗
2689	ksapisrv.com	✗	✗	✗
2713	cell.com	✓	✗	✓
2727	newscientist.com	✗	✗	✗
2746	huntress.io	✗	✗	✗
2774	instacart.com	✓	✗	✗
2778	system-monitor.com	✗	✗	✗
2800	chicagotribune.com	✗	✗	✗
2808	indianexpress.com	✓	✗	✓
2811	france24.com	✗	✗	✗
2832	wa.gov	✗	✗	✗
2836	salesforce-scrt.com	✗	✗	✗
2850	sueddeutsche.de	✓	✗	✓
2860	yumpu.com	✓	✗	✓
2863	faz.net	✗	✗	✗
2877	jwpltx.com	✗	✗	✗
2894	mapquest.com	✗	✗	✓
2925	thetimes.co.uk	✓	✗	✗
2926	imageshack.us	✗	✗	✗
2936	asahi.com	✗	✗	✗
2963	thehindu.com	✗	✗	✓
2969	globalcdn.co	✗	✗	✗
2973	vogue.com	✓	✗	✗
2990	1drv.ms	✗	✗	✗
3000	medscape.com	✓	✗	✓
3018	ctvnews.ca	✗	✗	✓
3026	today.com	✗	✗	✗
3039	newsbreak.com	✗	✗	✗
3050	hellotalk8.com	✗	✗	✗
3098	skybridge.click	✗	✗	✗
3159	allrecipes.com	✓	✗	✗
3177	metro.co.uk	✗	✗	✗
3202	billboard.com	✗	✗	✓
3222	flipboard.com	✗	✗	✗
3238	quizlet.com	✓	✗	✓
3253	nzherald.co.nz	✗	✗	✗
3262	infojobs.net	✗	✗	✗
3273	yahoo.net	✓	✗	✓
3285	thetimes.com	✓	✗	✗
3321	permutive.app	✗	✗	✗
3341	amazon.ae	✗	✗	✗
3349	nymag.com	✓	✗	✗
3353	lifehacker.com	✗	✗	✗
3354	kicker.de	✓	✗	✗
3359	tds.net	✓	✗	✓
3376	deadline.com	✗	✗	✓
3436	rs6.net	✗	✗	✗
3450	orf.at	✓	✗	✓
3451	ladsp.com	✗	✗	✗
3467	thefreedictionary.com	✗	✗	✗
3473	lenovomm.com	✗	✗	✗
3474	skimresources.com	✗	✗	✗
3498	abc.es	✗	✗	✗
3503	cuny.edu	✗	✗	✗
3518	chron.com	✓	✗	✓
3527	glance.net	✗	✗	✗
3535	vanityfair.com	✓	✗	✗
3537	stuff.co.nz	✗	✗	✓
3576	amazon.nl	✗	✗	✗
3579	politico.eu	✗	✗	✗
3582	smugmug.com	✗	✗	✗
3629	nydailynews.com	✗	✗	✗
3637	amazon.eg	✗	✗	✗
3641	entrust.net	✗	✗	✗
3645	ansa.it	✓	✗	✓
3660	gifshow.com	✗	✗	✗
3684	gitea.com	✗	✗	✓
3687	aljazeera.net	✓	✗	✓
3693	amazon.se	✗	✗	✗
3729	realestate.com.au	✓	✗	✗
3743	liberation.fr	✗	✗	✓
3746	codeberg.org	✗	✗	✓
3763	zimbra.com	✗	✗	✗
3796	yomiuri.co.jp	✗	✗	✗
3805	pcworld.com	✓	✗	✓
3819	toyota.com	✓	✗	✓
3829	nos.nl	✓	✗	✓
3830	sportradarserving.com	✗	✗	✗
3844	news24.com	✗	✗	✗
3847	wordreference.com	✗	✗	✗
3887	geforce.com	✗	✗	✗
3920	aol.de	✓	✗	✓
3952	ilsole24ore.com	✗	✗	✓
3968	udn.com	✓	✓	✗
3977	rfi.fr	✗	✗	✗
4000	xhpingcdn.com	✗	✗	✗
4024	siteforce.com	✗	✗	✗
4028	seekingalpha.com	✓	✗	✓
4036	gmarket.co.kr	✗	✗	✗
4046	thestar.com	✗	✗	✗
4057	kompas.com	✗	✗	✗
4078	handelsblatt.com	✓	✓	✗
4081	seattletimes.com	✗	✗	✗
4114	mainichi.jp	✗	✗	✗
4119	amazon.sa	✗	✗	✗
4147	amazon.pl	✗	✗	✗
4208	braintreegateway.com	✗	✗	✗
4225	derstandard.at	✓	✗	✓
4229	tvtropes.org	✓	✗	✓
4233	ketchcdn.com	✗	✗	✗
4254	verywellhealth.com	✓	✗	✗
4255	infura-ipfs.io	✗	✗	✗
4263	snopes.com	✓	✗	✓
4273	home.blog	✗	✗	✗
4321	ebay.ca	✓	✗	✓
4325	verywellmind.com	✓	✗	✗
4348	radiofrance.fr	✗	✗	✓
4353	mcusercontent.com	✗	✗	✗
4411	southernliving.com	✓	✗	✗
4415	ugtelset.ru	✗	✗	✗
4424	d-net.pro	✗	✗	✗
4443	biorxiv.org	✓	✗	✓
4446	paycomonline.net	✗	✗	✗
4447	20minutes.fr	✗	✗	✗
4457	marthastewart.com	✓	✗	✗
4501	ebay.com.au	✓	✗	✓
4516	lastampa.it	✓	✗	✓
4543	cnki.net	✗	✗	✗
4549	huya.com	✗	✗	✗
4566	feednews.com	✗	✗	✗
4568	barrons.com	✓	✗	✗
4582	amazon.com.tr	✗	✗	✗
4621	thespruce.com	✓	✗	✗
4643	lesoir.be	✗	✗	✗
4669	eatingwell.com	✓	✗	✗
4680	msnbc.com	✗	✗	✓
4685	mooo.com	✗	✗	✗
4690	meb.gov.tr	✓	✗	✓
4700	boxofficemojo.com	✗	✗	✗
4703	polygon.com	✓	✗	✓
4746	helsinki.fi	✗	✗	✗
4757	hurriyet.com.tr	✗	✗	✓
4763	yandexadexchange.net	✗	✗	✗
4765	channel4.com	✗	✗	✓
4773	cadenaser.com	✓	✗	✗
4803	nine.com.au	✗	✗	✓
4806	telegraaf.nl	✗	✗	✗
4812	ew.com	✓	✗	✗
4858	drive2.ru	✗	✗	✗
4868	drugs.com	✓	✗	✓
4870	dr.dk	✗	✗	✗
4918	softpedia.com	✓	✗	✓
4923	arte.tv	✗	✗	✗
4939	dayforcehcm.com	✗	✗	✗
4941	deccanherald.com	✗	✗	✓
4945	lesechos.fr	✗	✗	✗
4992	imageshack.com	✗	✗	✗

Want to be counted for the October re-run?

Not on this list and want to be sure? Check your own site. If you are on it by accident, this guide covers how the mistake usually happens and how to undo it.