Google, OpenAI, Microsoft and Anthropic said on October 30 that they'll honor one shared line in robots.txt telling them not to train AI on your website. Until now you had to block each company's crawler by name. If you run a site, this is the easiest opt-out yet, though it switches on at different times for each company.
robots.txt is the small text file at the root of a website that tells crawlers what to skip. The new signal is a special name you put where a crawler's name would normally go. Here's the example from the proposal. It blocks training on a whole site except its press kit.
User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/Anthropic's Kevin Kelley wrote the proposal and filed it with the Internet Engineering Task Force, the body behind many web standards. The IETF hasn't endorsed it yet. It's a draft that the four companies have chosen to follow.
When it takes effect, and what it covers
PPC Land reports that Anthropic has started building support, without a live date. OpenAI says it aims to do so "in the coming months." Microsoft will honor it from February 1, 2027, and Google says 2027. Google will also treat the line as a block on Google-Extended, its existing control for training and grounding in Gemini Apps, if you don't mention that one separately.
The signal covers training generative AI models, which the working group defines as changing a model's learned parameters. It isn't a crawling rule, so bots can still visit your pages; it only states how you want the content used. The general User-Agent: * rule doesn't count, so you have to add the group by name. It also can't pull back anything that was scraped before.
AI answers that fetch your page live are a separate question. Microsoft warns that blocking that kind of use "will significantly reduce your content's visibility in AI-generated experiences" and is adding a separate control for it. So decide two things: whether you want to feed training, and whether you want to show up in AI answers. We'd guess many publishers will say no to the first and yes to the second.
Meta isn't on the list
Meta spokesperson Andy Stone backed the idea of a common signal on X, but Meta hasn't committed to honoring it. Its own fight with publishers is now a regulatory case. On October 30, the UK's Competition and Markets Authority opened an investigation into Meta's commercial terms, which the regulator says could let Meta collect content from publishers' websites for its own products, including AI features. Meta paused the rollout on October 28 pending talks with industry groups, and says there are no grounds for an investigation.
CMA chief executive Sarah Cardell said that "where a company holds a dominant position, it must not use that position to impose unfair terms," and the case will also test whether Meta holds one. We covered what those terms said in our October 16 piece.
What we'd do with your robots.txt
If you don't want your work used for training, add the group now. The providers have announced different implementation schedules. Adding the line now prepares your preference for that rollout. Keep your existing rules for GPTBot, Google-Extended and ClaudeBot too, because the new signal sits alongside them.
Just don't mistake it for a deal. Microsoft says outright that the preferences "do not establish licensing or payment arrangements." Saying no just got easier. Getting paid is still a different conversation.
Cover: AI-generated editorial illustration by TMRW using Google, OpenAI, Microsoft and Anthropic branding. The printed page shows the example from the IETF draft.
