What Is llms.txt? A Proposed Convention for AI-Readable Sites

llms.txt is a plain-text, Markdown-formatted file that a website can publish at its root — https://example.com/llms.txt — to give AI agents a curated map of the site’s most important content. The idea was proposed by Jeremy Howard and is documented at llmstxt.org: instead of forcing a language model to parse navigation menus, JavaScript-heavy pages, and hundreds of URLs, the file offers a short summary plus links to the pages the site owner considers authoritative.

That is the whole mechanism. Everything else — whether agents read it, whether it helps, whether you should bother — is where the honest answers get more nuanced.

Status: a proposed convention, not a standard

This is the point most coverage of llms.txt gets wrong, so it deserves its own section.

Label: Proposed convention. llms.txt is a community proposal. It is not an IETF RFC, not a W3C recommendation, and not part of any search engine’s documented requirements. As of this writing, no major AI provider has publicly committed to fetching or honoring llms.txt as part of its default retrieval behavior. Some tools and agent frameworks let users point at llms.txt files manually, and some sites publish them, but tool support varies and broad adoption should not be assumed.

Contrast that with its two neighbors:

• robots.txt is an informal but decades-old convention that every major crawler documents support for — see Google’s robots.txt documentation.
• sitemap.xml is a formalized protocol that search engines actively consume — see Google’s sitemap guide.

llms.txt has neither that history nor that documented uptake. Google’s own guidance on AI features and your website centers on normal crawlable content and existing controls, not llms.txt. If you add the file, add it because a curated AI-readable summary is useful on its own terms — not because you expect guaranteed pickup.

Where llms.txt fits in the agent-ready model

In the Learn Web4 framework, an agent-ready website has three levels: Discoverable (agents can find and fetch it), Understandable (agents can correctly interpret it), and Actionable (agents can safely act through it). Agent-ready is not the same as AI SEO. A site that agents can find is not necessarily a site agents can understand, and a site agents can understand is not necessarily a site they can safely act on.

llms.txt sits mostly at the boundary of Level 1 and Level 2. It can help an agent that has already found your site orient itself faster — which pages explain the basics, which are reference material, what the site is and is not about. It does nothing for Level 3, because it describes content rather than enabling actions. For the full model, see The Three Levels of an Agent-Ready Website and the practical build guide How to Build an Agent-Ready Website.

llms.txt vs robots.txt vs sitemap.xml

These three files are often confused because they all live near the site root. They answer different questions:

	robots.txt	sitemap.xml	llms.txt
Core question	”What may crawlers access?"	"Which URLs exist?"	"What should an AI read first?”
Format	Plain text directives	XML	Markdown
Primary audience	Search-engine crawlers	Search-engine crawlers	LLM agents (proposal)
Status	Established convention, documented by major crawlers	Formal protocol, actively consumed	Proposed convention, varying tool support
Controls access?	Yes (advisory; compliant crawlers honor it)	No	No
Affects ranking?	Indirectly, via crawl control	Indirectly, via discovery	No confirmed effect
Typical location	/robots.txt	/sitemap.xml	/llms.txt

A useful mental model: robots.txt is the fence, sitemap.xml is the index, llms.txt is the tour guide. The fence and the index are load-bearing infrastructure. The tour guide only matters if someone takes the tour — which is exactly what is not yet guaranteed.

A realistic example file

The llms.txt specification describes a Markdown file with an H1 site name, a short blockquote summary, and H2 sections of links. Here is a realistic example for a small learning site:

# Example Learning Site

> Example Learning Site explains web technology for AI agents in plain
> English. Content is educational and updated on a stated schedule.

## Start here

- [What is Web4?](https://example.com/articles/what-is-web4/): Core concept and working definition
- [Learning roadmap](https://example.com/articles/roadmap/): Suggested reading order for beginners

## Reference

- [Glossary](https://example.com/glossary/): Definitions of all key terms
- [Agent-ready checklist](https://example.com/tools/checklist/): Self-assessment for site owners

## Optional

- [Archive](https://example.com/archive/): Older posts, kept for history; prefer the pages above

Details worth copying:

• Curate, don’t dump. Ten to twenty links, chosen deliberately. The “Optional” section exists precisely to tell agents what they can skip.
• Absolute canonical URLs. No relative paths, no staging domains, no URLs that redirect.
• A scope note. The blockquote states what the site is and its limits — this is ambiguity reduction, a Level 2 concern.
• Maintainable size. If the file becomes a second sitemap, it will rot. Review it whenever core pages change.

The proposal also mentions an optional /llms-full.txt variant that inlines full content. For most small sites that is unnecessary maintenance; start with the short file.

When adding llms.txt is worth the effort — and when it is not

Worth it when:

• You run a documentation, API, or learning site where agents are a plausible audience, and you can name the ten pages that matter most.
• You already want a curated “start here” map for humans, and publishing it in Markdown costs you an afternoon.
• You can commit to keeping it current. A stale llms.txt pointing at dead pages is worse than none.

Not worth it when:

• Your site is small and already well-linked. A five-page site with clear internal linking gives an agent the same map for free. The file adds nothing a good homepage doesn’t.
• You expect it to fix weak content. llms.txt can point to your best page; it cannot make a thin page substantive. If agents misunderstand your site, the fix is almost always clearer pages, better headings, and explicit definitions — Level 2 work — not a new file at the root.
• You expect an SEO payoff. There is no official confirmation that any major search engine reads it. Treat ranking claims you see elsewhere as marketing, not evidence.
• You would publish and forget it. An unmaintained curated file is a liability.

The ordering matters: crawlable HTML, working internal links, robots.txt, sitemap.xml, and structured data all come first. llms.txt is the last layer, not the first. The Agent-Ready Website Checklist walks through that order.

How to test whether agents actually use it

Because adoption is uncertain, treat your llms.txt as a hypothesis and test it. Label: test method.

1. Log fetches. After publishing, watch your server or CDN logs for GET /llms.txt. Record the user agents. If nothing fetches the file for weeks, that is direct evidence about real-world uptake — at least for your site’s traffic. (Caveat: many agentic tools fetch pages on demand from user machines or third-party crawlers, so absence in logs is suggestive, not conclusive.)
2. Baseline agent answers. Ask two or three agentic tools questions your site answers well, in fresh sessions: “According to [yoursite.com], what is X?” Save the answers and note whether your site is cited.
3. Publish, wait, retest. With llms.txt live, repeat the identical questions in fresh sessions after some weeks. Compare citation rates and answer accuracy against the baseline.
4. Direct test. Some agent tools let you attach or reference a URL explicitly. Point one at your llms.txt and ask it to summarize your site. A good file produces a noticeably better-oriented summary than pointing the same tool at your homepage.

Keep the test honest: same questions, fresh sessions, written-down results. Anything less is vibes.

Limitations

Label: Limitation. Be clear-eyed about what this file cannot do:

• No guaranteed readership. The file only works for agents that choose to fetch it. You cannot make them, and today most retrieval pipelines were not built around it.
• No access control. Unlike robots.txt, it expresses no permissions. It cannot keep content out of training data or block crawling — that is what robots.txt and documented AI crawler controls are for.
• No action capability. It describes pages; it does not expose tools, APIs, or safe actions. That is Level 3 territory in the agent-ready model.
• Maintenance burden. Curated files drift. Every site restructure creates a chance for the file to mislead the exact systems it was written to help.
• Trust is unearned at scale. If llms.txt ever becomes popular, it will also become an SEO-spam target, which gives agent builders a reason to distrust it. Its usefulness depends on norms that do not exist yet.

llms.txt is a cheap, low-risk clarity layer for sites that already have their foundations in order — useful as a curated map for humans and any agent that bothers to look, and a small bet on a convention that may or may not take hold. Publish it with that framing and you cannot be disappointed by it. For the surrounding context of what agents are and how they differ from chatbots, see What Is the Agentic Web? and AI Agents vs Chatbots.

Further reading

• The /llms.txt file - llmstxt.org
• Robots.txt introduction and guide - Google Search Central
• Build and submit a sitemap - Google Search Central
• AI features and your website - Google Search Central

FAQ

Is llms.txt an official standard?

No. It is a proposed convention published at llmstxt.org. It is not an IETF or W3C standard, no major AI provider has committed to reading it, and broad adoption should not be assumed.

Does llms.txt replace robots.txt or sitemap.xml?

No. robots.txt controls crawler access and sitemap.xml lists URLs for discovery. llms.txt is a curated reading guide for language models. The three files do different jobs and can coexist.

Will llms.txt improve my Google rankings or AI Overview citations?

There is no official confirmation that Google reads llms.txt for ranking or AI features. Treat it as an optional clarity layer, not an SEO lever.

How do I know if any agent reads my llms.txt?

Check your server or CDN logs for GET requests to /llms.txt and inspect the user agents. Then compare agent answers about your site with and without the file present.

Changelog

• Jul 26, 2026: Expanded into the site's canonical llms.txt reference: added the three-file comparison table, a realistic example file, an adoption-status section, a when-it-is-not-worth-it section, a test method, and a limitations section.
• May 2, 2026: Initial publication.