AI Giants Agree on One Opt-Out: How Publishers Can Block AI Training With a Single Setting

Four of the biggest names in artificial intelligence have agreed to honour a single opt-out signal from publishers, in a move that could finally give newsrooms, bloggers and website owners one straightforward way to say no to AI training. OpenAI, Google, Microsoft and Anthropic said on October 30, 2026, that they will support a shared policy token in robots.txt, the small text file that has told web crawlers what they may and may not do for more than 30 years.

Until now, opting out has meant keeping a growing list of company-specific crawler names up to date. A publisher that wanted to block AI training had to name OpenAI’s GPTBot, Google’s Google-Extended, Anthropic’s ClaudeBot and every other crawler separately, then repeat the exercise each time a new company or crawler appeared. Under the new approach, one shared entry is meant to be read the same way by all four companies.

The token is called __usage_ai-training. A site adds it as a user agent in robots.txt, followed by a Disallow rule for the pages it wants to keep out of training. In its simplest form, the entry looks like this:

User-Agent: __usage_ai-training
Disallow: /

Publishers that want finer control can use Allow lines to carve out exceptions, keeping some sections open while closing others. The idea is deliberately boring: it reuses a file every web team already knows, rather than asking publishers to learn a new system or sign up to a new service.

Why publishers pushed for one signal

The announcement lands after years of friction between publishers and AI companies. News organisations and independent sites have argued that their reporting, photography and analysis were swept into training data without permission or payment, while AI companies have pointed to existing controls and to fair-use arguments that are still being tested in courts.

That argument has only sharpened as AI crawlers have multiplied. A single news site can now be visited by dozens of bots, some collecting pages for search, some for archiving and some for model training. Existing robots.txt controls can already express a preference, but they have to be configured company by company, and a mistake or an outdated entry can leave a site exposed without its owner realising.

A shared token does not settle the payment question, and it does not settle the legal question. What it does is lower the practical barrier to saying no. For a small publisher without a dedicated engineering team, the difference between maintaining one rule and maintaining a dozen is the difference between opting out and giving up.

Who is backing it, and when it starts

The proposal comes from Kevin Kelley, a technical lead for standards at Anthropic, who submitted it as an individual draft to the Internet Engineering Task Force (IETF), the body that develops internet protocols. It is important to be precise about its status: an individual IETF draft is not an endorsed standard, and the draft itself is due to expire in May 2027 unless it is revised or adopted. The companies’ commitments, rather than the draft’s formal standing, are what give the token its force.

Anthropic has already begun implementing the signal. OpenAI says it aims to implement it in the coming months. Google aims to support it in 2027, with its crawler documentation pointing to early 2027. Microsoft will honour the preferences from February 1, 2027, when its Bingbot crawler revisits pages. Microsoft is also adding a more granular, page-level setting that separates training from grounding, the process of pointing an AI model at a verified source to reduce errors.

Meta is the notable holdout, for now. The company behind Facebook and Instagram has called a common signal an important step, but it has not said when, or whether, it will honour this particular token. That gap matters, because a publisher that adopts the shared token still needs a separate rule for any major crawler that has not signed up.

What the token does not do

Readers should not mistake the new signal for a delete button. It applies to future training, not to material already collected. Models trained on the open web through 2025 and 2026 are already deployed, and the token says nothing about removing existing training sets or paying publishers for material already used.

It is also a statement of preference, not a technical lock. The draft frames the token as a preference about how content is used, not as an instruction about crawling itself. A bot may still fetch a page for other purposes, such as search indexing, unless a separate rule blocks that too. And robots.txt has always worked on an honour system: it carries weight only to the extent that crawler operators choose to respect it.

Both Google and Microsoft have stressed that opting out of training will not remove a site from search results. Google has said a site that opts out with the shared token, without mentioning Google-Extended, will be treated as blocking Google-Extended, while sites that allow Google-Extended can still appear in Gemini app answers. Microsoft has likewise said websites will continue to appear in Bing search results regardless of their training preferences. For publishers worried that saying no to training means disappearing from search, that separation is a crucial reassurance.

What it means for American readers and newsrooms

For American newsrooms, the practical effect will arrive gradually, and unevenly. Large publishers with legal and engineering teams will adopt the token quickly and test whether it is honoured in practice. Smaller local papers, magazines and independent sites, which often lack those resources, stand to gain the most from a single rule they can copy, paste and forget.

For readers, the change is less visible but still real. If publishers use the token widely, future AI models may be trained on a narrower slice of the open web, with more content behind explicit permission or licensing deals. That could push AI companies towards licensed partnerships with publishers, the model already emerging in several high-profile deals, rather than broad scraping of everything publicly available.

There is also a transparency angle. A single, public signal makes it easier for researchers and journalists to check which sites have opted out, and to ask awkward questions when a company claims to respect a preference it has not implemented. The staggered timelines, however, mean that a publisher adopting the token today cannot assume all four companies are honouring it yet. Anthropic first, OpenAI in the coming months, Google and Microsoft in 2027: that is the honest timetable, and any coverage that promises instant protection would be misleading.

What to watch next

Three questions will decide whether this becomes a genuine turning point or another well-meaning note in a file. First, will Meta join, and will other model developers follow, so that one token really does cover the field? Second, will the IETF draft progress from an individual submission towards a recognised standard before it expires next May? Third, will publishers be able to verify, in practice, that crawlers are respecting the token, rather than taking company statements on trust?

Previous episodes have shown why verification matters. Publishers have found crawlers visiting pages they believed were blocked, and the difference between a stated policy and crawler behaviour is exactly what the new token is meant to close. Watchdog groups, search engineers and publishers themselves will be testing that gap in the months ahead.

For now, though, the direction of travel is clear. After years in which every AI company asked publishers to learn its own opt-out language, the four largest Western AI labs have agreed to speak one shared language instead. It is a small technical change with potentially large consequences for who controls the raw material of the AI era: the people who write it, or the companies that train on it.

Frequently asked questions

What is the new AI training opt-out?
It is a shared policy token, __usage_ai-training, that publishers can add to their robots.txt file to signal that their content should not be used to train generative AI models. OpenAI, Google, Microsoft and Anthropic have said they will honour it.

How does a publisher use it?
By adding a robots.txt entry naming __usage_ai-training as the user agent, with Disallow rules for the pages to be excluded, and Allow lines for any exceptions. Previously, publishers had to list each company’s crawler, such as GPTBot, Google-Extended and ClaudeBot, separately.

When will each company support it?
Anthropic has already begun implementation. OpenAI aims to implement it in the coming months. Microsoft will honour it from February 1, 2027. Google aims to support it in 2027, with documentation pointing to early 2027.

Does it delete content already used for AI training?
No. The token applies to future training only. It does not remove material already collected into datasets or models, and it does not require payment for material already used.

Will opting out hurt a site’s search rankings?
Google and Microsoft have both said no. Opting out of AI training is separate from search visibility, and sites will continue to appear in Google Search and Bing results.

Is Meta supporting the token?
Not yet as a commitment. Meta has called a common signal an important step, but has not said when or whether it will honour this token.

Sources and further reading

Reporting on the announcement by The Verge, as covered by The Indian Express, SiliconANGLE and other outlets; OpenAI’s post, “Making it easier for websites to express AI training preferences,” published October 30, 2026; and Microsoft’s and Google’s crawler documentation on the new token’s rollout dates.

The Indian Express: How a new standard could make it easier for publishers to opt out of AI training
The Brief: Four AI companies agree on a shared opt-out signal for web scraping
AI Weekly: OpenAI, Google, Microsoft, Anthropic back one AI opt-out token