Cloudflare Against 'Mixed-Use' AI Crawlers: Blocked Since September 15

Starting September 15, 2026, Cloudflare will block by default 'mixed-use' crawlers, which combine indexing for search, AI model training, and agent use, from pages hosting advertisements. This new feature, announced on the company's official blog, aims to give publishers more control over how their content is utilized by artificial intelligence.

The new default settings apply to new Cloudflare customers, new sites of existing customers, and all free customers who have not modified their configurations. Those who wish to continue being scanned by these bots can re-admit them manually by adjusting the settings, which will be made more granular, allowing the admission or exclusion of only certain types of bots.

The block is accompanied by a change in the compensation model: the Pay Per Crawl service evolves into Pay Per Use. Publishers will no longer be paid only when content is scanned, but when that content actually creates value. The first partners of the new model are Ceramic.ai, an AI search engine, and You.com, a search engine designed for AI agents: those who join will be paid when their content appears in the results of Ceramic.ai or when You.com accesses premium content.

Non-Human Traffic Surpasses Human Traffic

Matthew Prince, co-founder and CEO of Cloudflare, linked the move to a scale shift in internet traffic: "Now that the majority of internet traffic is no longer human, we need to move beyond and act more swiftly for a sustainable ecosystem to emerge," he stated, emphasizing that this threshold has been reached earlier than expected. Cloudflare also claims that more than half of the traffic generated by AI crawlers is spent re-scanning pages that have not changed, wasting bandwidth and publisher resources.

To give publishers more visibility into these consumptions, Cloudflare is also introducing a new Business Insights Dashboard, designed to show how bots consume content and how much traffic AI models generate.

Google, Microsoft, and Apple Potentially Involved

Among the crawlers that fall into the 'mixed-use' category, and therefore are potentially subject to the default block, are Googlebot, Bingbot from Microsoft, and Applebot. However, the three companies offer separate, specific exclusion channels for AI use, which could exempt them from the block: namely, Applebot-Extended, Google-Extended, and the 'noarchive' meta tag.

Cloudflare has doubled down, pointing a finger at what it calls "the world's largest search engine," referring to Google, claiming it would have access to roughly twice the information than other AI providers because it makes it difficult for sites to remain indexed on Search without simultaneously feeding its own AI products. Google responded by reminding that its Google-Extended bot allows sites to exclude content from training and AI products, including Gemini and Vertex APIs, without losing indexing on Google Search.