Cloudflare’s new policy pushes AI companies to pay for publishers’ content
Cloudflare, a leading internet infrastructure and security company, has announced a significant policy shift that will require artificial intelligence (AI) companies to either separate their web crawling activities or face default blocking from content that hosts advertisements. Effective September 15, 2026, Cloudflare's default settings will block "mixed-use" crawlers – bots that blend traditional search indexing with AI agent use and model training – from accessing ad-supported web pages. This move is poised to fundamentally reshape how AI models access and monetize online content, empowering publishers with greater control and potential revenue streams.
Cloudflare's New AI Policy: Key Details for Publishers and AI Companies
By Decode Today News
Cloudflare is introducing a new default policy to differentiate web crawlers. Starting September 15, 2026, "mixed-use" crawlers, which combine search indexing with AI training and agent functions, will be blocked by default from pages hosting ads for new and free Cloudflare customers, and new sites from existing customers. This initiative aims to give publishers more control over their intellectual property, allow them to monetize content used by AI, and conserve bandwidth by reducing unnecessary re-fetching of unchanged pages. Cloudflare is also evolving its "Pay Per Crawl" system into "Pay Per Use," enabling publishers to charge AI companies when their content generates value, not just when accessed.
The announcement, made on Wednesday, underscores a growing tension between content creators and AI developers regarding the use of web data. Cloudflare's co-founder and CEO, Matthew Prince, highlighted the urgency of the situation, noting that non-human bot traffic now constitutes the majority of internet activity, a milestone reached sooner than anticipated. "Now that the majority of traffic on the Internet is non-human, we must go further and act faster so that a sustainable ecosystem can emerge," Prince stated, emphasizing the need for clarity and transparency in bot interactions.
Under the new default policy, website owners utilizing Cloudflare will find that crawlers designed for a combination of search indexing, AI agent use, and AI model training will be automatically blocked from pages displaying advertisements. Site owners will retain the ability to adjust these settings, but the default aims to protect publishers' intellectual property and ensure their content is not freely leveraged for AI development without their explicit consent or a commercial arrangement. This change applies to new Cloudflare customers, new sites established by existing customers, and all current free customers.
The implications for AI model providers are substantial. Companies relying on broad web crawls for training large language models or powering agentic services will need to adapt their strategies. Cloudflare's objective is to encourage these mixed-use crawlers to separate their functions, allowing publishers to differentiate between traffic intended for traditional search discoverability and traffic intended for AI training or agent services. This distinction is crucial for fostering a more equitable digital ecosystem where content creators can realize the value of their contributions in the age of AI.
Addressing the "World's Largest Search Engine"
Cloudflare's announcement also contained a pointed reference to the "world's largest search engine," implicitly Google, suggesting it has a disproportionate advantage in accessing web content. Cloudflare noted that this search giant reportedly has access to "2x more information" than other AI companies, largely because its integrated approach makes it challenging for customers to remain discoverable via search without also having their content used for AI purposes. This perception has fueled ongoing discussions about fair competition and data access within the AI industry.
Google, for its part, has previously countered such generalizations. The tech giant offers a specific bot, Google Extended, which allows site owners to opt out of having their content used for AI training and AI products like Gemini Apps and Vertex AI API, without impacting their inclusion in Google Search. However, Google's flagship Googlebot is responsible for crawling content for Search, which now includes AI-powered features such as AI Overviews and AI Mode. This dual functionality is at the heart of the debate Cloudflare's new policy seeks to address, aiming to provide a clearer separation for publishers.
Evolving Monetization and Resource Management
The policy change is not solely about control; it also paves the way for new commercial opportunities for publishers. Cloudflare has been actively developing tools to empower website owners in the AI era, including initiatives to combat malicious AI bots. A notable development is the evolution of its "Pay Per Crawl" marketplace, which allowed websites to charge AI bots for scraping content, into "Pay Per Use." This advanced model enables publishers to charge AI companies not just for fetching content, but specifically when their content creates tangible value or appears in AI-generated outputs.
This "Pay Per Use" model represents a significant step towards a sustainable revenue stream for content creators from AI consumption. Cloudflare's data indicates that over 50% of crawl traffic from AI crawlers is spent re-fetching unchanged pages, an inefficient use of bandwidth and compute resources for publishers. By enabling more discerning access and monetization, the new policy could help conserve these resources, benefiting both publishers and AI model providers by promoting more efficient data acquisition practices.
To kickstart this new monetization framework, Cloudflare is initially partnering with Ceramic.ai and You.com. When publishers opt into the system, they can receive payments when their content appears in Ceramic's AI search results or when You.com accesses their premium content. Cloudflare has also indicated that other AI companies can customize this model to suit their specific operational frameworks, promoting flexibility and broader adoption.
The Broader Impact on the AI Ecosystem
Cloudflare's policy signals a maturation of the AI industry, moving beyond a largely unregulated content acquisition phase towards one that emphasizes transparency, compensation, and publisher rights. This shift is crucial for fostering trust and ensuring the long-term health of the internet as a source of high-quality, diverse content. If publishers feel their intellectual property is consistently undervalued or exploited, it could lead to a decline in accessible premium content, ultimately harming the quality and utility of future AI models.
For developers of AI agents and large language models, the impending deadline requires strategic re-evaluation of data sourcing. This could involve developing more specialized and transparent crawlers, engaging in direct licensing agreements with publishers, or adjusting business models to account for content acquisition costs. The goal, as Cloudflare suggests, is to encourage AI companies to have bots with "clear and transparent intent," fostering a more collaborative and fair environment.
Frequently Asked Questions About Cloudflare's New AI Policy
What is Cloudflare's new policy regarding AI companies and content?
Cloudflare will, by default, block "mixed-use" web crawlers (bots that combine search, AI agent use, and training) from accessing ad-hosting web pages. This policy takes effect on September 15, 2026, for new and free Cloudflare customers, and new sites set up by existing customers.
Why is Cloudflare implementing this policy?
The policy aims to protect publishers' intellectual property, give them more control over their content in the AI era, enable monetization opportunities for content used by AI, and help conserve publishers' bandwidth and compute resources by reducing unnecessary data re-fetching.
How will this affect AI companies?
AI companies using mixed-use crawlers will need to adapt by separating their crawler functions for search vs. AI training/agent use, or face being blocked by default from many ad-supported websites. This may lead to new content licensing models or direct payment arrangements.
How can publishers benefit from this change?
Publishers can gain greater control over who accesses their content for AI purposes and potentially monetize it through Cloudflare's evolving "Pay Per Use" system. This system allows them to charge AI companies when their content generates value, not just when it's accessed.
What is "Pay Per Use" and how does it work?
"Pay Per Use" is an evolution of Cloudflare's "Pay Per Crawl" marketplace. It allows publishers to charge AI companies when their content is specifically used to create value, such as appearing in AI search results or powering AI services. Cloudflare is initially partnering with Ceramic.ai and You.com for this model.
The Bigger Picture
Cloudflare's proactive stance is a landmark moment in the ongoing debate over digital intellectual property and the economic models underpinning the internet. By setting clear defaults and offering sophisticated monetization tools, Cloudflare is attempting to steer the AI industry towards a more responsible and sustainable future. This initiative could catalyze a broader shift in how content is valued and consumed by AI, empowering publishers and potentially leading to a more transparent and equitable exchange of value across the digital landscape. As the September 2026 deadline approaches, both AI innovators and content creators will be closely watching to see how this ambitious policy reshapes the foundations of the internet.
