Select Language:
John Mueller from Google stated that the new Content Signals robots.txt directive, introduced by Cloudflare last year, has no impact on any crawler or large language model (LLM). He emphasized that this directive merely adds unnecessary complexity and future maintenance work to your robots.txt file. According to him, no crawlers or LLMs currently utilize the content-signal robots.txt directives.
This was his response on Reddit to the question about whether Content-Signal headers and llms.txt files actually aid in disambiguating person entities.
Mueller shared the following points:
- Google does not recognize or use llms.txt or llms-author.txt files, and no other known crawlers or LLMs are confirmed to use them either, aside from some SEO tools.
- As far as he knows, no crawlers or LLMs currently support the “content-signal” directives in robots.txt. These were created by a CDN and appear to have no real effect on crawler behavior. Using them only adds clutter and complexity to your robots.txt file. Crawlers tend to support only the directives they recognize and ignore the rest.
Additionally, Cloudflare announced a deadline of September 15, 2026, by which it will set new default settings across three categories. For new domains using Cloudflare, the categories of Training and Agent will be blocked by default on pages serving ads, while Search remains allowed by default.
Currently, Cloudflare services about 21.3% of all websites as of January 2026. If your site uses Cloudflare, it’s important to review your settings before this deadline. Many site owners have already checked their configurations in anticipation.
In summary, Mueller clearly indicates that Google has not adopted the Content Signals robots.txt directive and doesn’t seem to have plans to do so.
Further discussion on this topic can be found on Reddit.
ChatGPT
Perplexity
Gemini AI
Grok AI
