What is llms.txt, and do you actually need one?

I checked 4,999 sites in the Tranco top 5,000 for a valid llms.txt file on 2026-09-06 and found 373 of them, which works out to 7.46 percent.

llms.txt is a plain markdown file served at the root of a domain, at /llms.txt. It gives a language model a short map of the site: a title line, then markdown links to the pages that matter, grouped under headings. The reasoning behind it is that a model reading markdown does not have to reconstruct your information architecture from navigation menus and cookie banners. It is an emerging and optional convention.

How many sites actually have one

On 2026-09-06 I requested /llms.txt from 4,999 sites in the Tranco top 5,000. I counted a file as valid only if it returned HTTP 200, was not HTML, opened with a markdown H1, contained at least one markdown link, and was still at /llms.txt after any redirects. 373 sites passed, which is 7.46 percent.

Adoption barely moves with site size. The top 100 sites had 11 with a valid file, so 11 percent. Ranks 101 to 1000 had 75 of 900, or 8.33 percent. Ranks 1001 to 5000 had 287 of 3,999, or 7.18 percent. The biggest sites are only slightly ahead of everyone else.

Most of the failures are simple absence: 2,201 returned a 404 and 1,241 were unreachable. 416 returned 403. 347 redirected away, and 180 returned an HTML page with a 200 status, which is a soft 404. Another 122 served plain text in some other format and 119 returned some other status. Those redirects and soft 404s add up to 527 sites whose /llms.txt answers with something no parser would accept.

Of the 373 valid files, 320 included a blockquote summary line and 55 sites also served /llms-full.txt. cloudflare.com, at rank 2, is one of the fuller examples, with 90 links and an llms-full.txt alongside. The whole set is in the 5,000-site census.

What the format requires

The format is small enough to describe in one paragraph. The first line is an H1 carrying the name of the site or project. Below it goes an optional blockquote line summarizing what the site is, which most adopters include. After that come H2 sections, each holding a markdown bullet list of links, and any link can be followed by a colon and a short description. A minimal valid file looks like this:

# Example Docs

> Documentation for the Example API and its client libraries.

## Guides

- [Quickstart](https://example.com/docs/quickstart): Install the client and make your first call.
- [Authentication](https://example.com/docs/auth): API keys and token rotation.

## Reference

- [Endpoints](https://example.com/docs/endpoints): Every route with its request and response shapes.

Serve the file as plain text and make sure /llms.txt does not redirect. Both problems show up in the census: 347 requests were redirected away from /llms.txt, and 122 sites returned plain text that did not parse as an llms.txt file. llms-full.txt is a related convention that concatenates the full text of your documentation into one file, so a model can read everything in a single fetch. It only makes sense if your docs build can generate it automatically, because a hand-maintained copy goes stale.

What it does for citations

The honest answer is that nobody knows. No major AI search engine has confirmed that it uses llms.txt as a ranking or citation input. Anyone promising that adding one will get you into ChatGPT answers is guessing.

The file is cheap to produce. If your docs already build from markdown, generating one costs almost nothing, and you end up with a canonical list of your own important URLs. The mechanism with direct evidence behind it is crawler access, since blocking a search crawler removes you from that engine's answers, and that is covered in how to show up in ChatGPT.

The short version: llms.txt is optional and unproven as a citation input, and 373 of the 4,999 sites I checked, or 7.46 percent, have one. Add it if it is cheap for you, once you have confirmed that the AI search crawlers can reach your pages and that those pages carry readable text before any JavaScript runs.

Two things that matter more

The first is whether the crawlers that feed citations can reach you. In the same census, 2,771 of the 4,999 sites served a robots.txt. Among those, 241 (8.7 percent) block OAI-SearchBot, the crawler that fetches pages for ChatGPT Search citations. 359 (12.96 percent) block PerplexityBot, and 250 (9.02 percent) block Claude-SearchBot. Much of that looks accidental: 296 sites block an AI search crawler while still allowing Googlebot. The census records the counts and not the reasons, so I cannot tell you how those lines got there.

The confusion usually starts with the training crawlers, and two facts clear most of it up. Google-Extended has exactly one function, controlling whether Google uses your content to train Gemini models. Google Search and AI Overviews are governed by Googlebot, so a Google-Extended block leaves your Google Search presence untouched. GPTBot works the same way on OpenAI's side, covering model training only, while OAI-SearchBot is the crawler behind ChatGPT Search citations. In the census, 297 sites block GPTBot on its own, which is the training opt-out done correctly, while 238 block GPTBot and OAI-SearchBot together and lose their ChatGPT Search presence with it. I break each one down on the crawler reference pages, with a fuller argument in should I block GPTBot. One more detail: if you have no robots.txt, or it returns an empty 200, crawling is allowed.

The second thing is whether your pages are readable without JavaScript. On 2026-08-31 I attempted 900 homepages from Tranco ranks around 10,000 to 10,700, once each, with no JavaScript executed, and got 607 successful fetches. 481 (79.2 percent) came back with readable text. 97 (16.0 percent) returned under 250 characters, which is effectively an empty page, and 29 more (4.8 percent) under 800. Separately, 58.0 percent carried no JSON-LD structured data and 26.4 percent had no meta description. A tidy llms.txt does nothing for a site whose real pages return a shell. Does ChatGPT see my website covers how to test that on your own domain.

My recommendation

Do it if you run documentation that builds from markdown already, since the file then stays current for free.
Do it if you want one canonical list of your important URLs, which is useful to your own team regardless of any crawler.
Do not do it before fixing your robots.txt. If OAI-SearchBot is blocked, no map of your site will help.
Do not pay an agency for it. The file is a few dozen lines of markdown.

I built a generator and validator for it, and the validator is the part I would use first, given how many published files fail the format. To see which AI crawlers your robots.txt allows today, run your domain through the free checker. Well-known sites that have shut AI search engines out are listed on who blocks AI search, and the rest of the writing sits under guides.