Google, Microsoft, Anthropic, and OpenAI have agreed on 1 unified protocol to let website publishers block automated data collection for artificial intelligence training. The system adds a shared control token to standard robots.txt files, eliminating the need to write separate rules for each individual tech firm. The framework is now before the Internet Engineering Task Force, with company rollouts scheduled to extend through 2027.
Publishers can insert the designated token into their robots.txt file to restrict training access across an entire domain or limit the block to specific folders. Site owners retain the option to set custom exceptions for individual vendors. The rule applies strictly to back end model training datasets. Consumer facing features such as instant language translation, automated document summaries, and user driven web browsing remain unaffected by the restriction.
Anthropic engineer Kevin Kelly drafted the proposal and submitted the technical framework to the Internet Engineering Task Force as a working draft. Adoption schedules vary across the 4 primary partners. Anthropic has started active deployment, while OpenAI expects to launch support within months. Microsoft plans to activate the protocol in early 2027, with Google rolling out support later in 2027. Meta endorsed the initiative publicly but has not published a specific deployment schedule.
