In a move that could reshape how artificial intelligence (AI) companies gather data, connectivity cloud company Cloudflare has announced it will now block AI crawlers from accessing content without permission or compensation by default. This policy, which affects the roughly 20% of the entire internet that uses Cloudflare's services, shifts the burden from an opt-out to an opt-in model for data scraping.
Previously, website owners had to actively choose to block AI bots. In September 2024, Cloudflare introduced a one-click option to block AI crawlers, which more than one million customers adopted. The new default setting reverses this approach, requiring AI companies to obtain explicit permission before scraping. Upon signup, new Cloudflare customers will now be asked upfront if they want to allow AI crawlers.
The decision comes amid growing concerns over the legality, copyright infringement, and accuracy surrounding the practice of web scraping to train large language models (LLMs). Matthew Prince, co-founder and CEO of Cloudflare, emphasized the need to protect content creators. "If the Internet is going to survive the age of AI, we need to give publishers the control they deserve and build a new economic model that works for everyone," he stated. Prince noted that AI crawlers have been "scraping content without limits," depriving creators of revenue and traffic.
To address this, Cloudflare is also introducing an initiative called "Pay Per Crawl." This program, currently in private preview, allows publishers to set prices for access to their content. AI companies can then choose to pay for access or be denied entry. Furthermore, Cloudflare is providing more granular controls, allowing website owners to distinguish between bots based on their stated purpose, such as for training, inference, or search.
Cloudflare has developed sophisticated systems to identify and block unwanted bots. The company uses global machine learning models to identify deceptive practices, such as user agent spoofing, where a bot tries to disguise itself as a real browser. For users, enabling the block is straightforward. They can navigate to the Security section of the Cloudflare dashboard and toggle the "AI Scrapers and Crawlers" switch. This feature is available to all Cloudflare customers, including those on the free plan.
Several major publishers, including Ziff Davis, The Atlantic, BuzzFeed, and TIME, have already signed on to support the new policies. Roger Lynch, CEO of Condé Nast, called Cloudflare's approach a "game-changer" that "opens the door to sustainable innovation built on permission and partnership."
This strategic shift by Cloudflare represents a significant step toward giving content creators and website owners more control over their intellectual property. As the AI landscape continues to evolve, this move could set a precedent for more equitable and transparent data collection practices, ensuring that those who create the content that fuels AI models are fairly compensated for their work.