Update — June 14, 2026: We’ve concluded this experiment. The figures below are the final snapshot from the last 30-day collection window and are no longer updating live. We’re grateful to everyone who ran the plugin and contributed anonymous data.
This experiment tracked how often AI bots actually fetch llms.txt files when they crawl a site. Not whether they crawl, but whether they reach for the structured guidance file once they’re already there.
—
Loading data…
Per-bot engagement
For each AI bot, the share of crawled sites where it also fetches an /llms.txt file. Sorted by engagement rate. Last 30 days.
| Bot | Sites crawled | Sites with /llms.txt visit | Engagement | Last seen on /llms.txt |
|---|---|---|---|---|
| Loading data… | ||||
Crawl depth
Some bots come back; others look once and leave. Average /llms.txt hits per domain, for bots that engage at all. Last 30 days.
| Bot | Domains visited | Avg hits per domain | Pattern |
|---|---|---|---|
| Loading data… | |||
About llms.txt
The idea behind llms.txt is straightforward: give AI crawlers a structured, easily digestible version of your site’s most important content. But does it work? Yoast added autogeneration of llms.txt to their plugin and recently Google added llms.txt guidance to their lighthouse docs.
What is this experiment?
We launched a plugin that auto-generates llms.txt for WordPress in early 2025 and it has grown organically to over 30k active installs. We track AI crawler behavior across opted-in users of our llms.txt WordPress plugin. Each time a known AI bot hits the site, an anonymous signal is sent to our server and contributes to the tables above.
What the data showed
- Engagement is single-digit even at the top. The most engaged bot in the dataset still only fetches
/llms.txton around 6% of the sites it crawls. Most are well under 2%. Even when AI crawlers find the file, the majority don’t bother to read it. - OpenAI’s bots lead engagement. GPTBot and OAI-SearchBot are the most likely to actually fetch
/llms.txtwhen they crawl a site. The bots that looked like leaders in raw volume (ClaudeBot, Meta-ExternalAgent) drop sharply once we exclude sitemap requests and count only canonical/llms.txtfetches. - Common Crawl (
CCBot) ignores/llms.txtentirely. The crawler behind much of the open AI training pipeline crawls thousands of sites in our sample and has not fetched/llms.txta single time in 30 days. - Most fetches are one-and-done. Bots that engage usually grab the file once and don’t return. The clear exception is Meta-ExternalAgent, which revisits an engaged domain about five times on average.
- User-action bots barely engage. ChatGPT-User, Claude-User, and Perplexity-User account for less than 1% of
/llms.txttraffic. They fetch the specific page being asked about, not a directory file. Google-Extendedis dead air. The official Google opt-out signal for AI training has registered zero/llms.txthits in the last 30 days across the entire monitored dataset. In practice, almost nobody uses it.
What gets sent
All data is anonymous. The signal sent per bot visit includes the bot’s user-agent name, the timestamp, and the page path. Nothing else. No content, no IPs, no identifying information.
Need help with AI Optimization? At Complete SEO, we help clients increase revenue through improved visibility in AI and organic search. Contact us today for a free consultation.
