Update — June 14, 2026: We’ve concluded this experiment. The figures below are the final snapshot from the last 30-day collection window and are no longer updating live. We’re grateful to everyone who ran the plugin and contributed anonymous data.
This experiment tracked how often AI bots actually fetch llms.txt files when they crawl a site. Not whether they crawl, but whether they reach for the structured guidance file once they’re already there.
—
Loading data…
Per-bot engagement
For each AI bot, the share of crawled sites where it also fetches an /llms.txt file. Sorted by engagement rate. Last 30 days.
| Bot | Sites crawled | Sites with /llms.txt visit | Engagement | Last seen on /llms.txt |
|---|---|---|---|---|
| Loading data… | ||||
Crawl depth
Some bots come back; others look once and leave. Average /llms.txt hits per domain, for bots that engage at all. Last 30 days.
| Bot | Domains visited | Avg hits per domain | Pattern |
|---|---|---|---|
| Loading data… | |||
About llms.txt
The idea behind llms.txt is straightforward: give AI crawlers a structured, easily digestible version of your site’s most important content. But does it work? Yoast added autogeneration of llms.txt to their plugin and recently Google added llms.txt guidance to their lighthouse docs.
What is this experiment?
We launched a plugin that auto-generates llms.txt for WordPress in early 2025 and it has grown organically to over 40k active installs. We tracked AI crawler behavior across opted-in users of our llms.txt WordPress plugin. The figures on this page come from 6,455 opted-in domains that sent at least one recognized AI-bot signal during the final collection window. Each time a known AI bot hits the site, an anonymous signal is sent to our server and contributes to the tables above.
What the data showed
- Engagement is single-digit even at the top. The most engaged bot in the dataset still fetches
/llms.txton only a small share of the sites it crawls. When AI crawlers find the file, most do not read it. - OpenAI’s bots lead engagement. GPTBot and OAI-SearchBot are the most likely to actually fetch
/llms.txtwhen they crawl a site. The bots that looked like leaders in raw volume (ClaudeBot, Meta-ExternalAgent) drop sharply once we exclude sitemap requests and count only canonical/llms.txtfetches. - Most fetches are one-and-done. Bots that engage usually grab the file once and don’t return.
- User-action bots barely engage. ChatGPT-User, Claude-User and Perplexity-User are a negligible share of
/llms.txttraffic. They fetch the specific page being asked about, not a directory file. Google-Extendedis dead air. The official Google opt-out signal for AI training has registered zero/llms.txthits in the last 30 days across the entire monitored dataset. In practice, almost nobody uses it.
Need help with AI Optimization? At Complete SEO, we help clients increase revenue through improved visibility in AI and organic search. Contact us today for a free consultation.
