llms.txt vs robots.txt: what each one actually controls

These get confused constantly because they are both small text files at the root of a domain aimed at automated readers. They do not compete and one cannot substitute for the other: robots.txt is a permission gate, llms.txt is a summary handed over after permission is already granted.

The one-line answer

robots.txt tells a crawler what it is allowed to fetch, checked before the request happens. llms.txt is a curated map of a site's own pages, read by an agent that has already decided to look at you and wants the short version instead of parsing your navigation and cookie banner. One is an access rule. The other is a table of contents.

What the spec itself says

The people behind llms.txt drew this line themselves. Their own specification puts it plainly: "robots.txt lets automated tools know what access to a site is considered acceptable, such as for search indexing bots. llms.txt information is instead used on demand, when an agent needs information about a topic while assisting a user." That is the clearest answer available, straight from the source: access control and content summary are two different jobs, done by two files that happen to share a naming convention and a folder.

llms.txt does not grant access to anything

Publishing an llms.txt does not open a door that robots.txt has closed. If a crawler's user agent is disallowed in your robots.txt, adding a well organized llms.txt changes nothing for that crawler, because the access decision already happened at the robots.txt step, earlier in the pipeline than llms.txt is ever read. See how to show up in ChatGPT for what actually controls that decision.

robots.txt can block llms.txt itself

llms.txt is not a special file type that robots.txt rules skip over. It is just another URL path, matched by the same rules as everything else on your domain. A group with "Disallow: /" blocks a crawler from every path under that domain, llms.txt included, for any crawler that group applies to. If you rely on a broad disallow rule while also publishing an llms.txt meant for that same crawler, the disallow wins: a crawler that checks robots.txt first will not go on to fetch the file you published for it. Narrow the disallow, or add an explicit "Allow: /llms.txt" if the rest of the block is intentional.

One open question, stated honestly

Whether any AI agent actually checks robots.txt before fetching /llms.txt at all is not settled. The llms.txt spec does not say either way, and no major AI vendor has published a statement specifically about llms.txt fetch behavior, as distinct from their documented crawling and citation behavior covered on the crawler reference pages. Treat "robots.txt can technically block it" as the safe engineering fact, not as a confirmed vendor behavior.

Do you need both?

• Fix robots.txt first, always. It is binding on all of your content, llms.txt included, and getting it wrong is the difference between being cited and being invisible.
• llms.txt is optional and additive. 7.46 percent of the Tranco top 5,000 have a valid one, and Google has stated outright that Search and AI Overviews ignore it entirely.
• Neither file does the other's job. Fixing your robots.txt does not require an llms.txt, and publishing an llms.txt does not fix a broken robots.txt.