Robots.txt, llms.txt and AGENTS.md: How They Work Together

Most sites treat robots.txt, llms.txt and AGENTS.md as three separate checkboxes: write one, copy a template for the next, done. In practice the three overlap, and when they disagree about which AI crawlers can read your site, the most restrictive rule usually wins by default, often silently blocking traffic you actually wanted. This article covers what each file controls, where they contradict each other, and what to do when legal wants everything blocked.

What is the difference between robots.txt, llms.txt and AGENTS.md?

Robots.txt is a decades-old plain-text protocol that tells a well-behaved crawler which URL paths it may request; it grants or denies access, and nothing else. llms.txt is a newer plain Markdown file that gives AI agents a curated map of your most important pages, written for machines that already have access and need to know where to look. AGENTS.md is an instruction file aimed at AI agents (both site-visiting bots and coding or browsing assistants) covering how to behave once they are on your site or working with your repository, such as preferred formats, rate limits, or contact details.

None of the three replace each other. Robots.txt is the only one of the three that a crawler is expected to check before requesting anything; llms.txt and AGENTS.md are read after access is already granted, so writing a generous llms.txt does nothing for a crawler robots.txt has already turned away. See what llms.txt actually does and how to implement AGENTS.md for the mechanics of each file on its own.

Why do these three files end up contradicting each other?

Because they are usually written at different times by different people. Robots.txt was often set up years ago by whoever configured SEO crawl budget and has not been touched since; llms.txt gets added later by whoever picks up the AEO project; AGENTS.md gets added by engineering without checking either of the other two. The most common conflict: robots.txt disallows a path such as /docs/ to protect crawl budget from search engines, while the new llms.txt links directly to pages under that same path as the site's best content for AI agents to cite.

A second common conflict is staleness on the crawler list itself. New AI crawlers appear faster than most teams update robots.txt, so a file written eighteen months ago may not mention a crawler that now sends meaningful traffic, and by default an unlisted user-agent falls under whatever the wildcard rule says.

How should you decide per-crawler allow or deny rules?

Decide by what the crawler does with your content, not by its name alone. Some crawlers train models, some fetch pages live in response to a single user question, and a few do both. RFC 9309, the formal robots.txt specification, only defines the syntax; the decision of what to allow is yours.

Google's own crawler documentation is explicit that Google-Extended can be blocked without affecting a site's presence in classic Google Search results, which is exactly the kind of narrow, per-crawler decision this table is built for. For the retrieval mechanics behind these categories, see how AI crawlers actually index a site.

What belongs in llms.txt that robots.txt cannot express?

llms.txt is additive, not restrictive: a short description of the brand, a curated list of the pages worth reading first, and section headers grouping docs, guides, and policies. It cannot deny access to anything, and it should never link to a path robots.txt disallows, since that promises access the crawler was already told not to take. Treat it as a table of contents for a house robots.txt has already unlocked, not a second gate. The directory of llms.txt implementation specialists and a working llms.txt template are useful starting points if you are building the file from nothing.

How should AGENTS.md handle pages that are gated or dynamic?

AGENTS.md should point agents at what changes and where the authoritative version lives, since it cannot itself grant access to anything behind a login. For gated content, state plainly that the page requires authentication and link to a public summary or API where one exists. For pages rendered client-side, note whether meaningful content depends on JavaScript execution, because not every crawler renders it, and say where a server-rendered or API equivalent is available. AGENTS.md started as a convention for coding and browsing agents working inside a repository, and site owners are now reusing the same filename for site-level guidance, so keep the two purposes from blurring together on a single page if your project uses both.

What should you do when legal wants everything blocked?

Take the concern seriously before arguing against it — a blanket block is a real, defensible business decision, not a mistake to be talked out of. The honest trade-off: blocking everything keeps content out of model training entirely, but it also removes the site from citations in AI answers, since a crawler that cannot fetch a page cannot quote it. A narrower compromise usually satisfies both goals: deny the crawlers that train models (GPTBot, ClaudeBot, CCBot, Google-Extended, Applebot-Extended) while allowing the ones that fetch a page live only when a real person asks about it (ChatGPT-User, PerplexityBot). That distinction, made per crawler rather than as a single blanket rule, is the whole reason the table above separates "trains on" from "fetches live for."

Getting that distinction written into a policy that legal will actually sign off on is where a lot of teams stall out. An AEO agency that has taken this exact conversation through legal before can usually turn it around faster than drafting the policy from a blank page internally.

How often should you audit robots.txt, llms.txt and AGENTS.md together?

Treat all three as one system with one review cadence, not three separate files on three separate schedules.

1. Diff all three files against the last reviewed version and note every change, not just the most recent one.
2. Cross-reference every path llms.txt links to against robots.txt's Disallow rules — a mismatch here is the most common conflict.
3. Recheck the crawler user-agent list against current engine documentation, since new crawlers appear faster than most review cycles.
4. Confirm your CDN or WAF bot-management rules do not contradict what robots.txt allows; see how CDN bot management silently blocks AI crawlers for why this step gets missed.
5. Re-fetch a sample page as each crawler after any CMS, CDN, or hosting change, since those changes are the most common trigger for a silent regression.
6. Log the change and who approved it, especially after a legal-driven block, so the next review does not have to guess why a rule exists.

The bottom line

Robots.txt is the only enforceable layer of the three — it is what actually stops or allows a request. llms.txt and AGENTS.md are descriptive layers sitting on top of it, and both fail quietly the moment they promise something robots.txt does not back up. Review all three together, per crawler rather than as one blanket rule, every time one of them changes.

If you would rather have someone who has already done this audit walk it through with you, get matched with a vetted AEO agency for a free crawler-access review — it takes a few minutes to submit and there is no cost to buyers.