Update — June 14, 2026: We’ve concluded this experiment. The figures below are the final snapshot from the last 30-day collection window and are no longer updating live. We’re grateful to everyone who ran the plugin and contributed anonymous data.
We measured how often AI bots requested a site's llms.txt file while crawling it. We focused on bots already on the site and whether they went on to request that structured guidance file.
—
Loading data…
Per-bot engagement
The table gives each AI bot’s share of crawled sites where it requested /llms.txt in the last 30 days. Highest engagement rates come first.
| Bot | Sites crawled | Sites with /llms.txt visit | Engagement | Last seen on /llms.txt |
|---|---|---|---|---|
| Loading data… | ||||
Crawl depth
Some bots revisit /llms.txt, while others check it once. The table shows mean /llms.txt hits per domain over the last 30 days for bots that engage with the file.
| Bot | Domains visited | Avg hits per domain | Pattern |
|---|---|---|---|
| Loading data… | |||
About llms.txt
llms.txt is a simple idea meant to give AI crawlers a structured, easy-to-digest version of a site's most important content. Whether it works remains an open question. Yoast added automatic llms.txt generation to its plugin, and more recently Google added llms.txt guidance in Lighthouse docs.
What is this experiment?
We followed AI crawlers on the sites of users who opted in to our llms.txt WordPress plugin. Complete SEO released the plugin in early 2025 to auto-generate llms.txt, and it grew organically past 40k active installs. Our data comes from 6,455 opted-in domains that each sent at least one recognized AI-bot signal during the final collection window. Whenever a known AI bot hits a site, an anonymous signal goes to our server and feeds our tables.
What the data showed
- Engagement stays in single digits. Even the top-engagement bot requested
/llms.txton only a small fraction of sites it crawled. When AI crawlers encountered the file, most didn't read it. - OpenAI's bots engage most. GPTBot and OAI-SearchBot were the likeliest to request
/llms.txtwhile crawling a site. ClaudeBot and Meta-ExternalAgent appeared to lead on raw volume. Their numbers fell sharply once we excluded sitemap requests and counted only canonical/llms.txtfetches. - Most fetches happen once. Bots that engaged typically fetched the file once and didn't return.
- User-action bots hardly engage. ChatGPT-User, Claude-User, and Perplexity-User accounted for a negligible share of
/llms.txttraffic. These bots fetch the specific page a user asks about. They don't fetch a directory file.
We help clients grow revenue through better visibility in AI search and organic search. If you'd like help with AI Optimization, contact us for a free consultation.
