Cloudflare has unveiled new software that lets websites block unauthorised AI crawlers and set prices for access, in a bid to help content creators monetise their work amid declining referral traffic.
The tool, announced on Tuesday, introduces a “pay per crawl” model allowing site owners to decide whether and how artificial intelligence models can use their material.
With AI firms harvesting web content to train models without sending users back to source sites, traditional ad-driven traffic has taken a hit.
Publishers including Condé Nast, the Associated Press, Reddit and Pinterest have reportedly backed the initiative.
Stephanie Cohen, Cloudflare’s chief strategy officer, said the goal was to restore balance between AI firms and online content creators.
Google’s crawler-to-visitor ratio has fallen sharply, while AI companies such as OpenAI generate far fewer referrals.
With lawsuits and licensing deals now shaping the response, Cloudflare’s tool adds a commercial layer to the debate over fair use.