Common Crawl checked 584,107 llms.txt files. Most of them are not doing what site owners think.

A new Common Crawl analysis, published September 8, 2026, is the largest look yet at what is actually inside the llms.txt files sites have started publishing. The headline finding is not that adoption is low. It is that a lot of the files that do exist are not doing the one job llms.txt is supposed to do.

What Common Crawl actually looked at

Common Crawl, the nonprofit that maintains the largest open web crawl most large language models are trained on, pulled every llms.txt file it could find across its crawl and published the results as an open report. The scale is real: 584,107 files, not a sample of a few dozen sites. Search Engine Journal's Matt G. Southern covered the findings the same day they went up, and the underlying data is published as an open Hugging Face Space by Common Crawl itself.

The three problems the report found

Reporting on the analysis describes three recurring patterns, not isolated examples:

• Many files are template-generated. A CMS plugin or generator produces the same boilerplate shape regardless of what the site actually is, which is the opposite of what this site's own llms.txt guide already says the format is for: a short, curated, human-written map of what actually matters on a site.
• Many files contain no links at all. llms.txt's entire proposed job is pointing an AI system at the pages worth reading. A file with no links is not a smaller version of that job. It is not doing the job.
• Some files write robots.txt-style rules into llms.txt. Allow and Disallow lines, or similar crawler-permission syntax, showing up in a file with no enforcement mechanism behind it. This is the exact confusion this site's own llms.txt vs robots.txt guide was built to head off: robots.txt is the file that can actually be checked and enforced by a well-behaved crawler. llms.txt has no such mechanism. Nothing reads a Disallow line in llms.txt and honors it.

Why this matters if you already have a file, or are about to write one

A template-generated or link-free llms.txt is not neutral. It costs you nothing to serve, but it also is not accomplishing anything, and it can create a false sense that the AI-visibility box has been checked. The real value of llms.txt, on the rare sites that use it as intended, is a short, honest list of the pages that actually explain what the site does, kept current by a person who knows which pages those are. A generated file with no links does not do that, no matter how many crawlers eventually decide to read llms.txt at all.

If you are trying to control what a crawler can access or train on, that is a robots.txt and Content-Signal question, covered in this site's GPTBot guide and Content-Signal guide. llms.txt cannot do that job, and writing it as though it can is exactly the pattern Common Crawl's analysis found at scale.

What to actually do

• If you already have an llms.txt file, open it and check honestly: does it have real links to real pages, written by someone who knows what is actually worth reading on the site? Or is it template output with a company name swapped in?
• Do not put Allow/Disallow lines or other crawler-permission syntax in llms.txt. It will not be enforced. Use robots.txt for that, and verify what is actually being served with the checker on this site.
• If you do not have one yet, do not treat this report as a reason to skip llms.txt entirely. It is a reason to write a short one by hand, with real links, rather than generate one to have something in place.

Related: What is llms.txt, and do you actually need one? and llms.txt vs robots.txt: what each one actually controls cover the format itself in more depth. Source: Common Crawl's llms.txt report, covered by Matt G. Southern at Search Engine Journal, published September 8, 2026.