OpenAI will honor a shared robots.txt signal that lets website operators tell participating AI providers how they want their content treated for training.
That is the whole announcement: one signal, carried in a file sites already publish, meant for providers that agree to recognize it. The company said the goal is to help operators communicate their preferences.
Some background, which is context and not part of the announcement. Robots.txt is a plain text file, dating to the 1990s, that sits at the root of a site and tells automated crawlers which pages to skip. AI opt-outs have mostly worked company by company. A site names each crawler it wants to block, then has to keep that list current as new ones appear. A common signal would let an operator state a preference once.
The word "participating" carries a lot of weight. A shared signal is only as useful as the number of providers that agree to read it, and robots.txt has always relied on voluntary compliance rather than enforcement. Which other companies join, and how finely the signal can express a preference, will decide whether this simplifies anything for publishers or just adds one more entry to the list they already maintain.