The Complete llms.txt Guide (With Examples): Format, Hosting, and How AI Crawlers Use It

llms.txt is a plain-text Markdown file you place at the root of your domain (https://yourdomain.com/llms.txt) that gives AI models a clean, curated map of your most important content. It is a proposed convention — introduced by Jeremy Howard of Answer.AI in September 2024 — not an official web standard, and no major AI provider currently guarantees they read it or that having one improves your rankings.

That honesty up front matters, because most llms.txt guides oversell it. This one will show you exactly how to write the file, where to host it, how it relates to robots.txt and sitemap.xml, and what it realistically does and does not do today. If you want the broader strategic picture, read our guide on how to make your website AI-discoverable alongside this one.

What Is llms.txt and What Problem Does It Solve?

When a large language model needs information about your site, it faces two problems. First, HTML pages are noisy: navigation menus, cookie banners, ads, JavaScript, and footers bury the actual content. Second, models work within a limited context window, so they cannot ingest your entire site. Converting messy HTML into clean signal at inference time is expensive and error-prone.

llms.txt addresses this by doing the curation for the model. Instead of letting a crawler guess which pages matter, you hand it a short, structured Markdown document that lists your key resources with plain-language descriptions. Think of it as a table of contents written specifically for machines that read text, not for browsers that render pixels.

It is worth being precise about scope. llms.txt is aimed at inference-time retrieval — the moment an AI assistant is answering a question and wants to pull in relevant, current information about your product, docs, or policies. It was not designed as a training-data control mechanism. That job still belongs to robots.txt and the newer AI-specific crawler directives.

The llms.txt File Format

The specification defines a simple, human-readable Markdown structure. It has a required part and several optional parts, and the ordering is fixed so parsers can rely on it.

The format, in order:

1. An H1 with the name of the project or site. This is the only strictly required line.
2. A blockquote (>) with a short summary. One or two sentences describing what the site or project is. Optional but strongly recommended.
3. Zero or more paragraphs of additional context (no headings), giving detail a model needs to interpret the rest of the file.
4. H2 sections, each containing a Markdown list of links. Each list item is a hyperlink, optionally followed by a colon and a short description.
5. An ## Optional section. Links here are explicitly marked as skippable — a model with a tight context budget can drop them without losing anything essential.

The key design choice is that everything is standard Markdown. That means the same file is readable by a human in a text editor and trivially parseable by a model, since LLMs are already fluent in Markdown.

The Companion Pattern: llms-full.txt and .md Pages

Two related conventions often appear alongside llms.txt:

• llms-full.txt — a single, larger file that contains the actual full text content, not just links. Useful when you want a model to have everything in one fetch, at the cost of a much bigger file.
• Clean .md versions of pages — the practice of serving a Markdown copy of any HTML page at the same URL with .md appended (for example about.html.md). The links in your llms.txt can point to these clean versions so a model never has to parse HTML at all.

Worked Example Files

Here is a minimal llms.txt for a small SaaS product. This is the baseline everyone should be able to produce.

# Acme Analytics

> Acme Analytics is a privacy-first web analytics tool for small
> teams. It is cookieless, GDPR-compliant, and self-hostable.

Acme replaces heavyweight analytics suites with a single lightweight
script. Pricing is flat-rate per site with no event caps.

## Docs

- [Quick Start](https://acme.example/docs/quickstart.md): Install the
  script and see your first data in under five minutes.
- [API Reference](https://acme.example/docs/api.md): Full REST API for
  querying metrics programmatically.
- [Self-Hosting Guide](https://acme.example/docs/self-host.md): Run
  Acme on your own infrastructure with Docker.

## Product

- [Features](https://acme.example/features.md): What Acme measures and
  how it protects visitor privacy.
- [Pricing](https://acme.example/pricing.md): Flat monthly pricing per
  tracked site.

## Optional

- [Changelog](https://acme.example/changelog.md): Release history.
- [Company](https://acme.example/about.md): Team and founding story.

A second example, this time for a local service business rather than a software product. Notice how the descriptions carry the facts an AI assistant would need to answer a customer question directly.

# Harbor Dental Clinic

> Harbor Dental is a family dental practice in Portland, Oregon,
> offering general, cosmetic, and emergency dentistry since 2004.

## Services

- [General Dentistry](https://harbordental.example/general.md):
  Cleanings, fillings, crowns, and preventive care.
- [Emergency Care](https://harbordental.example/emergency.md):
  Same-day appointments for pain, breaks, and lost fillings.
- [Cosmetic Dentistry](https://harbordental.example/cosmetic.md):
  Whitening, veneers, and Invisalign.

## Visit

- [Hours & Location](https://harbordental.example/contact.md): Open
  Mon-Fri 8am-5pm at 120 Harbor St, Portland OR 97201.
- [New Patients](https://harbordental.example/new-patients.md):
  Insurance accepted, intake forms, and what to expect.

## Optional

- [Patient Reviews](https://harbordental.example/reviews.md)
- [Blog](https://harbordental.example/blog.md)

A third, stripped to the absolute minimum that is still valid — an H1 and one section. This is legitimate; you do not need every section to have a usable file.

# Jane Doe — Freelance Illustrator

> Editorial and children's book illustrator based in Berlin,
> available for commissions.

## Links

- [Portfolio](https://janedoe.example/work.md): Selected published work.
- [Rates & Contact](https://janedoe.example/contact.md): Commission
  pricing and how to get in touch.

Where to Host It

Host the file at the root of your domain: https://yourdomain.com/llms.txt. This is the single location the convention specifies, exactly like robots.txt. Do not put it in a subdirectory, and do not rename it — tooling looks for that precise path.

A few practical hosting notes:

• Serve it with the content type text/plain (or text/markdown); most static hosts do this automatically for a .txt file.
• Make sure it returns HTTP 200 at the canonical URL and is not blocked by authentication, geoblocking, or a robots.txt disallow rule.
• If you run multiple subdomains (for example docs. and app.), each subdomain can have its own llms.txt scoped to that host.
• Keep it in version control and update it when your important URLs change, the same way you would maintain a sitemap.

How llms.txt Differs From robots.txt and sitemap.xml

These three files all live at the domain root and all speak to automated clients, which is exactly why they get confused. They solve different problems and are complementary, not substitutes. Keep all three.

The mental model: sitemap.xml says "here is everything that exists," robots.txt says "here is what you may and may not access," and llms.txt says "here is what actually matters and what it means." A sitemap is exhaustive and undescribed; an llms.txt is selective and annotated.

One important clarification about crawl control. llms.txt does not block anyone from anything — it grants no permissions and revokes none. If you want to control whether AI companies crawl your site for training, that is done through robots.txt user-agent rules (for bots like GPTBot, ClaudeBot, Google-Extended) and, increasingly, a separate proposed ai.txt-style permissions layer. Do not treat llms.txt as a consent or opt-out file.

How AI Crawlers and Assistants Actually Use It

Here is where honesty is essential, because the gap between what is technically possible and what is confirmed in production is wide.

What happens today. A growing set of developer-facing tools read llms.txt. Documentation platforms, several AI coding assistants and IDE integrations, and some retrieval frameworks will fetch /llms.txt when you point them at a domain, then use it to decide which pages to pull into context. If you have ever pasted a docs URL into an AI coding tool and watched it find the right reference pages fast, an llms.txt may be why.

What is not confirmed. As of this writing, the major consumer AI search products have not publicly committed to using llms.txt as a ranking or retrieval signal in their general web answers. Public statements from people at some large providers have been skeptical, noting they already extract content from HTML at scale. So you should not expect that publishing an llms.txt will, by itself, get you cited more often in a general-purpose AI assistant's answers.

Why it can still be worth doing. The cost is near zero — it is one small Markdown file — and the practice forces a useful exercise: deciding which ten or twenty URLs on your site actually represent your value, and writing a one-line description of each. That clarity helps your HTML, your metadata, and your content strategy regardless of who reads the file. And if adoption grows, you are already positioned. This is the same low-cost, high-optionality logic behind broader generative engine optimization: you optimize for how AI systems consume content because that is where discovery is heading.

For content that AI systems parse right now with confirmed effect, structured data does more heavy lifting than llms.txt. See our breakdown of the schema markup AI search engines actually read — pairing valid schema with a clean llms.txt covers both the confirmed and the emerging channels.

Current Adoption and Limitations

To keep expectations calibrated, here is the state of play in plain terms.

• It is a proposal, not a mandate. There is no governing body, no conformance test, and no penalty for not having one. The spec lives in a public repository and evolves.
• Reading it is opt-in for consumers. Any tool or model that reads it does so by choice. Many do not.
• It is not a ranking signal. No search or AI provider has confirmed it influences visibility. Treat any claim otherwise as marketing, not fact.
• It can drift. Like a sitemap, an llms.txt that points to dead or outdated URLs is worse than none. It needs maintenance.
• It does not replace good content or structured data. It is a pointer file. If the pages it points to are thin, the pointer does not help.

None of this makes llms.txt a bad idea. It makes it a low-risk, forward-looking hygiene practice — closer to adding a sitemap in 2006 than to a guaranteed growth lever. Publish it, keep it accurate, and let the ecosystem catch up.

Frequently Asked Questions

Is llms.txt an official web standard?

No. It is a proposed convention introduced in September 2024 and maintained in a public specification, but it has not been ratified by any standards body such as the W3C or IETF. Unlike robots.txt, which is formalized as RFC 9309, llms.txt has no official status and no provider is obligated to support it.

Will adding llms.txt improve my Google or AI search rankings?

There is no evidence that it does, and no major provider has confirmed it as a ranking or retrieval signal for general web answers. You should add it for its low cost and future potential, not as an SEO tactic. Real visibility gains still come from quality content, structured data, and technical health.

Do I still need robots.txt and sitemap.xml if I have llms.txt?

Yes, absolutely. The three files serve different, complementary purposes: sitemap.xml lists all your URLs for discovery, robots.txt controls crawler access and permissions, and llms.txt curates and describes your key content for AI reading. llms.txt grants no permissions, so it can never replace robots.txt.

Where exactly should the llms.txt file live?

At the root of your domain, served at https://yourdomain.com/llms.txt, returning HTTP 200 with a plain-text or Markdown content type. It must not be in a subdirectory or behind authentication. If you have multiple subdomains, each can host its own file scoped to that host.

How long should an llms.txt file be?

Keep it short and curated — typically your ten to thirty most important URLs, each with a one-line description. The whole point is selectivity, so resist listing every page. If you want to provide full text rather than links, use a separate llms-full.txt file so the primary llms.txt stays a lean index.

You do not have to hand-write and maintain all of this alone. Tools like emax.studio can generate the AI-optimized content your llms.txt should point to — clean pages, structured summaries, and answer-first copy — so the file links to material that is actually worth citing.

Create your first AI-powered marketing campaign at emax.studio — free plan available.