Content-Signal in robots.txt: Why Google and Bing Ignore It

Quick Answer

`Content-Signal: search=yes, ai-input=yes, ai-train=yes` is a line in robots.txt that tells crawlers what they may do with your content after they fetch it: index it, feed it into live AI answers, and train models on it. It was invented by Cloudflare in September 2025, not by Google, Bing or any standards body. Google Search Console, Bing Webmaster Tools and Lighthouse flag it because their parsers only recognise four robots.txt fields, and Content-Signal is not one of them. The warning is cosmetic: the line changes nothing about how your site is crawled or ranked, and, as of July 2026, no major crawler or AI company has committed to honouring it.

If you found this line in your robots.txt and did not put it there, you are not alone, and nothing is broken. Here is the full picture.

Key Takeaways

What Content-Signal Actually Is

Classic `robots.txt` answers one question: may you fetch this URL? It has never answered the question every publisher now cares about: what may you do with the page once you have it?

Content-Signal is an attempt to answer that second question in the same file. It sits under a User-agent line and carries a comma-separated list of signals:

User-agent: *
Content-Signal: search=yes, ai-input=yes, ai-train=yes
Allow: /

Each signal covers a distinct downstream use:

Three states exist per signal, and the third one matters. `yes` grants, `no` restricts, and omitting a signal grants nothing and restricts nothing. That is why Cloudflare's default block sets search=yes, ai-train=no and deliberately leaves ai-input unstated: a neutral position on the most contested use.

The all-permissive version in the heading of this article, search=yes, ai-input=yes, ai-train=yes, is the opposite stance. It says: index me, quote me in AI answers, and train on me. Publishers who want maximum AI visibility rather than maximum control sometimes set it deliberately, which is a defensible strategy if getting cited by AI engines is worth more to you than the content itself. Just be aware you are broadcasting a preference to systems that are not reading it.

Not sure what your own robots.txt is telling crawlers right now? Get a free audit and we will show you exactly which directives are live and which are being ignored.

Who Made It, and Why

Cloudflare launched the Content Signals Policy on 24 September 2025, during its annual Birthday Week. The stated motivation was the collapse of the old bargain of the web: search engines crawled your content and sent you traffic in return, while AI systems increasingly crawl your content and answer the user directly, sending nothing back.

Three design decisions are worth knowing:

1. It is CC0. Cloudflare released the policy text into the public domain so any site, on any host, can paste it in. This was positioned as a shared standard rather than a Cloudflare feature, and the format is genuinely free to adopt.
2. It was applied by default. Cloudflare added the policy to the managed robots.txt of millions of domains on its network. Zones using managed robots.txt received the actual Content-Signal line, and free zones without a robots.txt file received the explanatory comment block. This is why so many site owners discovered the line without writing it, and why the validator warnings arrived in waves.
3. It doubles as a legal notice. The comment block that ships above the directive states, in capitals, that any restriction expressed through content signals is an express reservation of rights under Article 4 of EU Directive 2019/790 on copyright in the digital single market. That is the EU text and data mining opt-out. The intent is to make the line evidence in a copyright argument, not just a hint to a crawler.

In July 2026 Cloudflare extended the format with the use signal and began reporting through its bot verification programme on whether Verified Bots comply with declared preferences, with loss of verified status as the sanction. That is the only enforcement mechanism attached to the format anywhere, and it is Cloudflare's, not the web's.

Here is the full comment block Cloudflare injects, so you can recognise it:

# As a condition of accessing this website, you agree to abide by the following
# content signals:

# (a)  If a Content-Signal = yes, you may collect content for the corresponding
#      use.
# (b)  If a Content-Signal = no, you may not collect content for the
#      corresponding use.
# (c)  If the website operator does not include a Content-Signal for a
#      corresponding use, the website operator neither grants nor restricts
#      permission via Content-Signal with respect to the corresponding use.

Why Google Search Console Calls It an Error

This is the part that sends people looking for answers, so here is the mechanism in full.

Google's parser recognises four fields, and that is all

Google's robots.txt documentation is explicit. The supported fields are `user-agent`, `allow`, `disallow` and `sitemap`. The documentation adds that rules other than allow, disallow and user-agent are ignored by the parser, and that Google ignores invalid lines and uses only the valid ones.

`Content-Signal` is not in that list. It never was, because Google did not design it and has not adopted it. So when the Search Console robots.txt report, the Lighthouse SEO audit or PageSpeed Insights parses your file, the line falls through to the same bucket as `crawl-delay`, `noindex`, `host` and `request-rate`: unknown directive.

The behaviour is correct per RFC 9309, the 2022 standard that finally formalised the Robots Exclusion Protocol. Parsers are supposed to ignore lines they do not understand rather than choke on them. Ignoring is the specified behaviour. Reporting the ignored line is a validator being helpful, and that is what you are seeing.

The timeline that made this visible to everyone

The confusion peaked because the tooling lagged the rollout by months:

Note the split in that timeline. A tool marking the line as valid syntax is not the same as a crawler acting on it. Lighthouse stopping the warning changed a lint rule. It did not add a single crawler to the list of systems that honour the signal.

What Google has actually said

The clearest statement came from John Mueller of Google in July 2026, responding to a question about Content-Signal and llms.txt. His position was that he was not aware of any crawler or LLM using the directive, that it was created by a CDN rather than by a standards body or a search engine, and that unsupported entries add maintenance burden to robots.txt for no return.

Separately, Google has been documenting the unsupported rules it sees most often in the wild, working from HTTP Archive crawl data, with content-signal and content-usage on that list. Being formally documented as unsupported is the most official acknowledgement the directive has received from Google, and it is an acknowledgement that Google does not act on it.

Worth remembering: `Google-Extended`, Google's actual robots.txt token for AI training and grounding in Gemini, does not remove you from AI Overviews. Those are built from the regular search index. That gap is exactly the one ai-input was written to close, and it remains open. If your traffic is sliding because of AI Overviews, no robots.txt line currently fixes it.

Why Bing Does the Same Thing

Microsoft has not announced support for Content-Signal either, and Bing Webmaster Tools reports it the same way, as a line its parser does not recognise. Bing's robots.txt tester was built to validate the classic directive set, so an unrecognised field surfaces as a syntax notice.

What Bing does support is narrower and older: user-agent rules for bingbot and its AI crawler tokens, crawl-delay (which Bing honours and Google does not), sitemap declarations, and page-level meta directives such as noarchive, nocache and nosnippet that control how much of a fetched page can be displayed. Bing Webmaster Tools also added AI performance reporting that covers Copilot and Bing's own AI summaries, but that is measurement, not permission.

Two search engines, the same conclusion, for the same reason: neither of them agreed to a format a third party published.

Warnings like these are the reason most site audits get ignored: one cosmetic notice sits next to a genuine crawl-blocking rule and both look equally red. We can run a free check and tell you which of your robots.txt issues actually matter.

Does It Hurt Your SEO?

No. To be precise about the three things people worry about:

• Crawling: unaffected. Unknown lines are skipped, and the surrounding User-agent, Allow and Disallow rules are parsed normally.
• Indexing and rankings: unaffected. There is no ranking factor attached to the presence or absence of an unrecognised robots.txt field.
• AI visibility: unaffected, in both directions. Setting ai-input=no does not keep you out of AI answers, and setting ai-input=yes does not get you into them.

There is exactly one real risk, and it comes from automated editing rather than from the directive itself. Anything that writes to `robots.txt` on your behalf can produce a malformed file: a directive placed before any `User-agent` line, a missing colon, or a managed block that overwrites rules you added by hand. That breaks real rules. So the sensible response to seeing an injected `Content-Signal` line is not to panic about the warning, it is to read the whole file once and confirm your own rules survived the injection intact. This is the kind of silent regression that continuous indexing and monitoring exists to catch, because nobody re-reads robots.txt voluntarily.

What Actually Works Today

If your goal is control over how AI systems use your content, here is the honest hierarchy, strongest first:

The pattern across all five: the layers that enforce are the ones you control, and the layers you publish are requests. Content-Signal is a request published to an audience that has not agreed to listen.

Should You Keep the Line?

A short decision guide:

• Keep it if you are in the EU or serve EU users and want the Article 4 reservation of rights stated explicitly, or if you are on Cloudflare managed robots.txt and do not want to fight the platform default. Cost of keeping it: one cosmetic validator warning.
• Remove it if the warning is polluting an automated audit that your team or a client actually reads, or if you have hand-written robots.txt rules and want a file where every line does something today. Cloudflare lets you turn the managed injection off.
• Do not set it and then assume you are protected. That is the only genuinely bad outcome here: a site owner who sets ai-train=no, sees no warning after the Lighthouse fix, and concludes the content is safe from training. It is not.

Either way, the line should never be your only answer. Pair it with real controls from the list above, and treat Content-Signal as a bet on where the web's norms are heading rather than a description of where they are.

Summary

• `Content-Signal` is a Cloudflare format from September 2025 that states what crawlers may do with content after fetching it: search, ai-input, ai-train, and since July 2026 use
• Google supports four robots.txt fields and Content-Signal is not one of them, so Search Console, Bing Webmaster Tools and older Lighthouse builds report it as an unknown directive
• The warning is cosmetic. Unrecognised lines are ignored per RFC 9309, and crawling, indexing and rankings are unaffected
• Lighthouse stopped flagging it in v13.0.2 (January 2026), but that changed a lint rule, not crawler behaviour
• As of July 2026 no major crawler or LLM provider has committed to honouring it, and Google has said as much publicly
• Real control comes from network-level blocking, per-crawler Disallow rules and meta robots directives. Keep Content-Signal as a stated position, not as protection