Are AI bots actually reading `llms.txt` files?

October 21, 2025 4 min. read

Ryan Howard Head of Growth at Complete SEO

This live experiment tracks whether major AI bots like GPTBot, ClaudeBot, and PerplexityBot are reading llms.txt files across the web.

Each time an AI crawler accesses an llms.txt file, we log an anonymous signal from that site. This helps us understand which bots are actually checking for AI-specific permissions and content rules.

—

Websites are participating in the experiment

Bot	Sites Seen On	Last Seen
Loading data...

What is this experiment?

We're tracking AI crawler behavior via opted-in users of our llms.txt WordPress plugin to see if bots are accessing llms.txt files.

About llms.txt

The idea behind llms.txt is straightforward: give AI crawlers a structured, easily digestible version of your site's most important content. But does it work? Yoast recently added the ability to autogenerate llms.txt to their plugin, but John Muller from Google recently posted that no AI system currently uses llms.txt.

When llms.txt first "became a thing" I launched a plugin to auto-generate llms.txt for WordPress thinking it would be an internal tool for evaluation on our client and internal sites, but it has grown organically to over 20k active installs.

So that brings me to an easy experiment to answer the question: Are AI search bots actually looking at llms.txt files?

The experiment

Track which AI crawlers actually visit llms.txt when it's available. The latest version of the plugin includes an opt-in feature that logs when known AI bots hit the file. If users opt in:

Their site logs bot visits locally with the name of the bot and the date
There's a live view of site-specific logs inside WordPress
An anonymous signal is sent to our server and that data contributes to a public dashboard

All data is encrypted and anonymous. The data shared includes the bot name, timestamp, and a hashed version of your domain to track LLM crawler behavior across thousands of sites. No content or personal information is collected or stored.