Llms.txt for AI optimization is a hedge. It may help AI models understand your content, though it may not. We have limited direct evidence today about whether AI systems index these files. If these files gain adoption, early movers will benefit.
Llms.txt is an experimental standard that places a structured, LLM-friendly overview of a site's key content in plain Markdown at the root of a domain, as in Anthropic's llms.txt. It gives AI models a clean, structured summary of the site's most important content.
Cursor uses llms.txt to improve inline completions. Anthropic worked with Mintlify on a fuller version for its own docs. Mintlify rolled it out across thousands of dev sites. Perplexity maintains its own llms-full.txt.
Are AI crawlers actually accessing llms.txt?
We used our llms.txt WordPress plugin to log access from known AI crawlers across thousands of participating WordPress sites. We wanted to see whether GPTBot, ClaudeBot, and other AI crawlers accessed the llms.txt file.
Most didn't. Across 6,455 opted-in domains, GPTBot and OAI-SearchBot were the most likely to fetch the file, yet each did so on only a small share of the sites it crawled. The full numbers are in our llms.txt crawler study.
For now, the findings leave us with an uncertain picture:
- llms.txt for AI optimization is a hedge. It might help LLMs understand your content, and it might not.
- Direct evidence of real indexing behavior remains limited today.
- If these files are adopted, early movers will benefit.
If you run WordPress, you can generate your own llms.txt file with a simple plugin install. You can also log access to the file.
Who’s using llms.txt so far?
Adoption has moved quickly in separate pockets. Mintlify rolled it out across its entire doc hosting platform. Cursor, Zapier, Vercel, and Anthropic use it, while Perplexity maintains its own llms-full.txt.
Several GitHub tools have appeared to generate, validate, and simulate llms.txt ingestion. Some sites quietly link to their llms.txt from a footer or /meta directory. Others put references in developer tools or AI prompts.
It hasn't become a widespread standard yet. It's circulating in the right places, including internal copilots, RAG pipelines, and AI IDEs.
Examples worth checking:
What’s it good for?
For now, the strongest use cases for llms.txt aren't about visibility in AI answers. LLMs struggle with noisy page code and massive page structures full of irrelevant text. llms.txt provides a lightweight map of important content in plain Markdown.
That map can help:
- AI coding tools find the right docs for inline suggestions
- RAG pipelines reduce token waste
- Internal agents skip the crawl and go straight to the signal
For dev-focused companies, llms.txt is already useful. For everyone else, it's a bet on the future.
Best practices
- Keep it short. Start with 10–50 key URLs.
- Structure clearly. Use Markdown headings and add link descriptions where useful.
- Avoid spam. Keep keyword stuffing out of the file.
- Update it. Make sure your most useful content stays represented.
- Check your logs. See who requests the file.
Some ways llms.txt could be used in the future
1. A control surface for AI interaction
Beyond its role as a permissions file, llms.txt could point to a richer, structured manifest. That manifest could take the form of a model-facing API at .well-known/mcp.json. That file could specify which AI uses are permitted, including indexing, summarization, training, and fine-tuning. It could also set out how attribution should be handled, where source content is located, and how compliance can be maintained over time.
2. A monetization layer for content access
Inspired by ideas from Cloudflare, llms.txt could eventually link to licensing or billing endpoints that specify payment terms for AI systems reusing content. Those terms might include per-call pricing, usage tiers, or paywall indicators. Publishers could shift away from blocking bots outright and toward telling them, "yes, but here's how."
Background and technical overview
What is llms.txt?
Jeremy Howard, co-founder of Answer.AI, introduced llms.txt in 2024. It's a plain Markdown file at the root of a domain, such as https://yoursite.com/llms.txt. The file gives LLMs a clean, structured summary of your site's most important content.
llms.txt isn't a replacement for robots.txt or sitemap.xml. Those files focus on crawler permissions and URL discovery. llms.txt is about clarity and intent. It tells LLMs what to read and why that content matters.
Structure and formatting
The format is intentionally lightweight and includes these elements:
- One H1 containing your site name
- A brief summary written as a blockquote
- H2 sections that group content by category
- Markdown lists of links, with descriptions if you want them
- An optional "## Optional" section for content that's less critical
Example:
# Acme Docs
> Acme is an API platform for data automation.
## Guides
- [Quick Start](https://acme.com/docs/start.md): Basic setup instructions.
- [API Reference](https://acme.com/docs/api.md): Full API endpoint list.
## Optional
- [Changelog](https://acme.com/docs/changelog.md)
llms.txt vs. llms-full.txt
llms.txtprovides a curated index of high-priority content.llms-full.txtcontains a flattened Markdown export of your site's entire content corpus.
Some companies host both files. Others use only one, depending on how structured their site is.
Challenges llms.txt is trying to solve
- Context window limits: LLMs can't ingest an entire site.
llms.txtgives them a short list of content. - HTML bloat: Markdown is cleaner and easier to parse than HTML.
- Content ambiguity: Many sites don't make clear which content matters most.
llms.txthelps clarify that.
Frequently asked questions
What’s the purpose of llms.txt?
llms.txt guides AI models toward your best content. Its format is lightweight and structured.
Where should I put it?
Put the file at the root of your site. An example address is https://example.com/llms.txt.
Will it help me rank in ChatGPT or Perplexity?
It might, though nothing is guaranteed yet. It's early, and in our crawler data, most AI bots that found the file didn't fetch it.
What’s the difference between llms.txt and sitemap.xml?
Sitemaps list URLs. llms.txt explains and prioritizes them, giving AI models a structure to follow.
Do AI bots support it?
Some tools and internal systems already use llms.txt. Widespread adoption is still developing.
Can I include usage terms or metadata?
Yes. Some sites add attribution rules or update timestamps.
What kind of sites should use it?
Developer platforms, documentation sites, publishers, and product companies should use it. So can anyone with content worth summarizing or citing.
What’s the difference between llms.txt and llms-full.txt?
llms.txt curates highlights, while llms-full.txt contains everything in clean Markdown. Choose the version that fits your needs.
Can I block or gate specific sections?
Yes. Some implementations use sections or links with access guidelines, or include licensing terms and usage restrictions.
Is there a standard validator?
There's no official validator yet. Some GitHub projects can check the basic structure.
Can I automate it?
Yes, static site generators and platforms like Mintlify often generate it automatically. WordPress users can use the llms.txt plugin.
What format should I use?
Use plain Markdown for the file. Use H1/H2 headings, blockquotes, and bullet lists to organize the content cleanly.
What’s next?
Our crawler experiment is finished, and its results are in the llms.txt crawler study. We'll keep tracking how llms.txt evolves, and WordPress users can get started with our llms.txt plugin.
