A New Way to Block Bots and Earn Revenue
Cloudflare has launched a tool that allows websites to control how AI bots access their content. The move aims to give publishers more power in deciding who uses their material and how.
With this new feature, website owners can block unauthorized bots or allow access in exchange for a fee under a “pay-per-crawl” model. This gives creators a path to earn revenue from AI firms that collect data to train their models.
Changing Web Traffic, Shrinking Revenue
AI crawlers often gather data without sending users back to the original website. As a result, many websites have seen a drop in referral traffic, once a key source of ad income.
Cloudflare’s tool responds to this shift. It helps site owners regain control as AI companies increasingly use online content without credit or compensation.
Backed by Major Publishers and Platforms
Big names like Condé Nast, Associated Press, Reddit, and Pinterest support this initiative. These organizations also face challenges from AI bots scraping their content for free.
Stephanie Cohen, Cloudflare’s Chief Strategy Officer, said the tool’s purpose is to protect creators and ensure a sustainable internet. “This is just the beginning of a new model for the internet,” she said.
Crawlers Rise, Referrals Fall
Cloudflare shared striking data showing how referral traffic is falling while crawls continue.
Google’s crawl-to-referral ratio has shifted from 6:1 to 18:1 in just six months. This means that while Google bots are still crawling sites, fewer users are visiting them.
Other AI companies are worse. OpenAI’s ratio stands at 1,500:1, massive crawling with minimal traffic back to the source.
Legal Battles and Licensing Deals
Publishers have started to push back. Some have filed lawsuits for copyright violations. The New York Times is one of the major players suing AI companies over unauthorized data use.
Others are exploring licensing deals. Reddit, for instance, has sued Anthropic for scraping user comments while signing a separate licensing agreement with Google.
AI Firms Bypassing Standard Blocks
Many AI crawlers now bypass the common robots.txt file, a web standard used to control bot access. These firms claim they are not breaking any laws. But creators argue their work is being used without consent or compensation.
Building a Fairer Digital Economy
With Cloudflare’s tool, websites now have the power to say no, or ask to be paid. The goal is to strike a balance between innovation and fair use of online content.
This tool could be a first step in changing how the internet works for both creators and AI companies.
