This live experiment tracks how often AI bots actually fetch llms.txt files when they crawl a site. Not whether they crawl, but whether they reach for the structured guidance file once they’re already there.
—
Loading data…
Per-bot engagement
For each AI bot, the share of crawled sites where it also fetches an /llms.txt file. Sorted by engagement rate. Last 30 days.
| Bot | Sites crawled | Sites with /llms.txt visit | Engagement | Last seen on /llms.txt |
|---|---|---|---|---|
| Loading data… | ||||
Crawl depth
Some bots come back; others look once and leave. Average /llms.txt hits per domain, for bots that engage at all. Last 30 days.
| Bot | Domains visited | Avg hits per domain | Pattern |
|---|---|---|---|
| Loading data… | |||
What is this experiment?
We track AI crawler behavior across opted-in users of our llms.txt WordPress plugin. Each time a known AI bot hits the site, an anonymous signal is sent to our server and contributes to the tables above.
About llms.txt
The idea behind llms.txt is straightforward: give AI crawlers a structured, easily digestible version of your site’s most important content. But does it work? Yoast added autogeneration of llms.txt to their plugin, while John Mueller from Google publicly stated that no AI system currently uses llms.txt.
When llms.txt first “became a thing” I launched a plugin to auto-generate it for WordPress, originally as an internal tool for client and in-house sites. It has grown organically to over 20k active installs.
The plugin’s opt-in tracking feature lets us measure whether bots actually fetch the file in the wild. That’s the question this page is built to answer.
What the data is showing right now
- Engagement is single-digit even at the top. The most engaged bot in the dataset still only fetches
/llms.txton around 6% of the sites it crawls. Most are well under 2%. Even when AI crawlers find the file, the majority don’t bother to read it. - OpenAI’s bots lead engagement. GPTBot and OAI-SearchBot are the most likely to actually fetch
/llms.txtwhen they crawl a site. The bots that looked like leaders in raw volume (ClaudeBot, Meta-ExternalAgent) drop sharply once we exclude sitemap requests and count only canonical/llms.txtfetches. - Common Crawl (
CCBot) ignores/llms.txtentirely. The crawler behind much of the open AI training pipeline crawls thousands of sites in our sample and has not fetched/llms.txta single time in 30 days. - Most fetches are one-and-done. Bots that engage usually grab the file once and don’t return. The clear exception is Meta-ExternalAgent, which revisits an engaged domain about five times on average.
- User-action bots barely engage. ChatGPT-User, Claude-User, and Perplexity-User account for less than 1% of
/llms.txttraffic. They fetch the specific page being asked about, not a directory file. Google-Extendedis dead air. The official Google opt-out signal for AI training has registered zero/llms.txthits in the last 30 days across the entire monitored dataset. In practice, almost nobody uses it.
What gets sent
All data is anonymous. The signal sent per bot visit includes the bot’s user-agent name, the timestamp, and the page path. Nothing else. No content, no IPs, no identifying information.
Need help with AI Optimization? At Complete SEO, we help clients increase revenue through improved visibility in AI and organic search. Contact us today for a free consultation.
