Cloudflare moves to the other side: helping ChatGPT index the web faster
Cloudflare has started a pilot research project with OpenAI to verify whether the data from its network can improve the accuracy of AI-based search. The company, which oversees more than one-fifth of the web, observes in real-time how pages change and how traffic behaves: the test channels these signals, such as content freshness and traffic quality, into OpenAI's search and response system.
The stated objective concerns content discovery. The responses generated by the models are only as good as the pages that feed them: when the index falls behind, the chatbot returns outdated or incorrect information. OpenAI aims to leverage Cloudflare's live web view to help ChatGPT discover new content more quickly.
"Having up-to-date information is important for providing accurate answers to people using ChatGPT," stated Nick Ryder, VP of research at OpenAI, framing the trial as a way to identify content more efficiently.
The pilot operates in a different space than the content agreements already signed by OpenAI to fuel ChatGPT: beyond licensing, this is about discovery, specifically how and how quickly new material is accessed.
A reversal from the past year
This move seems to be at odds with Cloudflare's recent trajectory. In the past year, the company has allowed sites to block AI crawlers unless industry companies paid publishers and has built a protocol to distinguish humans from bots. The slogan was clear: your content, your rules.
This project goes in the opposite direction. Instead of blocking access to content, Cloudflare is now helping an AI engine reach it. CEO Matthew Prince has presented it as a matter of efficiency, a way to "make AI search more efficient and help people get quality answers faster."
Search continues to shift from links to answers. Publishers are already afraid of losing traffic as chatbots summarize their work, a concern that has prompted regulators to impose opt-out mechanisms. Cloudflare now finds itself on both sides: selling the tools to block crawlers while experimenting with the setup to fuel them.