Google announced on October 30, 2026 that it plans to support a shared AI training control in robots.txt, giving website owners a single way to express training preferences across multiple AI providers. The commitment arrived alongside same-day announcements from Microsoft and OpenAI.
How Google Will Apply the Control
The plan was laid out in a Google Search Central Blog post by John Mueller, a Google Search Advocate. The control is based on work from the AI Preferences working group at the Internet Engineering Task Force (IETF) and builds on Google-Extended, the AI training control Google says it has offered since 2023.
Instead of managing separate instructions for every AI developer, website owners will be able to use a shared policy token within robots.txt to set broad preferences across participating AI providers. When a website uses the token to opt out of training and has not explicitly specified Google-Extended, Google plans to interpret that as if the publisher had disallowed Google-Extended. Other AI providers can handle the signal similarly, so owners can opt out across all participating providers at once or pick and choose.
According to the post, existing company-specific controls will continue to function alongside the joint signal, so website owners can express a broad choice through the industry signal while still tailoring instructions to individual developers through controls such as Google-Extended. Websites that specifically allow Google-Extended are eligible to appear in responses and receive clicks through grounding in Gemini Apps.
Google said the proposal reflects feedback from publishers, civil society groups, regulators, and other stakeholders that have called for industry-wide controls on AI training, and that support across major AI providers would produce a durable standard that works across the industry. The company also said the proposal is consistent with the EU Code of Practice for general-purpose AI model providers under the EU AI Act, which recognizes robots.txt as a primary method for web publishers to express machine-readable preferences. Google aims to support the new control, along with other AI providers, in 2027 and said it will provide updates on its progress.
OpenAI made a parallel commitment the same day. OpenAI’s announcement said the company will honor the common robots.txt signal, aims to implement it in the coming months, and will keep its existing crawler controls in place and respected. OpenAI said it was joining other AI Preferences working group participants in the announcement and encouraged other AI providers to honor the signal.
The IETF Draft Behind the Signal
The mechanism is specified in an Internet-Draft titled “Machine-Readable Training Preference Signal,” authored by Kevin Kelley of Anthropic and dated October 30, 2026. The document is intended for the standards track and expires May 3, 2027; per the IETF datatracker, it is an individual submission that is not endorsed by the IETF and carries no formal standing in the standards process.
Under the draft, a service operator binds usage preferences to paths by writing ordinary robots.txt groups keyed by a per-purpose policy token, __usage_ai-training, in the User-Agent line. The draft’s example group expresses a preference against AI training for an entire site except for paths under /press-kit/. The mechanism introduces no new directive names and no new grammar: the path language, longest-match precedence, and group structure of the robots.txt standard, RFC 9309, are unchanged, so parsers already deployed across the internet will ignore policy-token groups they do not recognize. Wildcard groups, those with User-Agent: *, must not be interpreted as matching the policy token, and a robots.txt file with no __usage_ai-training group expresses no preference at all.
The token takes its meaning from the AIPREF vocabulary’s definition of AI training: “Using an asset to modify the learned parameters of a generative AI model.” The definition excludes the training of models used exclusively for non-generative tasks, even when the model is capable of generative tasks. The draft also stresses that the token states a preference about content use rather than a directive for crawling behavior, though consumers may use preferences to inform crawling; specific expressions should take precedence over general ones, and private agreements may override expressed preferences.
Kelley announced the draft on the IETF’s ai-control mailing list on October 30, 2026, writing that Anthropic has already started implementing support for it. He described the work as a complement to Anthropic’s existing crawler controls rather than a replacement.
Microsoft’s Adoption and Transition Dates
Microsoft announced its adoption the same day in a Bing blog post by Krishna Madhavan, Principal Product Manager for Microsoft Bing. Microsoft said it was the first company to provide publisher controls for AI training and grounding, pointing to the NoArchive and NoCache tags it introduced in 2023.
Microsoft is adopting the shared AI Training preference for sites and paths and extending it to the page level through meta tags and HTTP response headers, while adding a separate page-level AI Grounding preference. Alongside Allow and Disallow, the grounding preference adds a new DisallowExtended option that limits grounding use to Microsoft-operated experiences such as AI Mode, AI Overview, Copilot, and Microsoft 365 while excluding grounding services Microsoft provides to third parties, such as Web IQ and grounding through API access in Foundry. Both preferences default to Allow when no preference is published. Microsoft cautioned that selecting Disallow or DisallowExtended for grounding reduces brand visibility in AI experiences; the preferences do not change eligibility for traditional Bing search results and do not establish licensing or payment arrangements.
The new preferences will replace NoArchive and NoCache for training and grounding and take effect starting February 1, 2027, as Microsoft crawls. Temporary grounding transition preferences open in Bing Webmaster Tools on December 15, 2026 for sites using the older tags as of that date. Those sites must publish their new preferences and record a temporary grounding preference before March 1, 2027, after which NoArchive and NoCache will no longer be read as preferences for training or grounding. Temporary preferences remain in effect until Microsoft recrawls a page and discovers the new grounding preference or until September 30, 2027, whichever comes first.
Credit: Source link
