Introduction to the New System by Cloudflare
Cloudflare, a US-based company, has introduced Pay per crawl, a feature that enables content creators to charge artificial intelligence (AI) bots for accessing their online content. This new system is currently in a private beta phase and represents a shift in the AI economy by introducing a pay-per-query model that could become the new standard for monetizing access by automated agents, such as ChatGPT from OpenAI or Google’s Gemini.
The Current Landscape and Challenges
Currently, developers of AI models extract vast amounts of information from public websites without offering any compensation to the original authors or media. Meanwhile, publishers face a difficult decision: either allow unrestricted access to bots or completely block them. Pay per crawl presents an alternative solution by enabling creators to monetize their content through controlled access.
HTTP 402: Payment Required
The architecture of this new model is based on an old HTTP response code: 402 Payment Required, which has been largely ignored due to a lack of clear use cases. With Pay per crawl, Cloudflare proposes reviving this code as a mechanism for real-time negotiation of automated crawler access to online content.
The system allows domain owners to set specific rules for each AI bot. They can choose to grant free access, block it entirely, or require a fixed payment for each request. If the crawler does not pay or is not registered, access is denied with a 402 response. However, if the crawler indicates payment intent in its HTTP headers and meets authentication requirements, it can receive the content along with a charging confirmation via a 200 response.
Flexibility for Publishers
Pay per crawl offers complete flexibility to publishers. Currently, they can set a single price for all requests across their website, but Cloudflare anticipates the ability to define different prices based on sections, content types, or resource popularity in the future.
Even if a bot doesn’t have a billing relationship with Cloudflare yet, websites can configure a charging response that blocks access but informs the crawler about potential future business relationships. This way, the system also functions as an incentive for formalizing agreements between both parties.
To ensure the authenticity of bots registering for payment, Cloudflare implemented a protocol called Web Bot Auth, requiring cryptographic signatures and validated public keys. This aims to prevent unauthorized or non-paying access by malicious or impersonating crawlers.
A New Business Model
The new model redefines how content access payments are settled. Cloudflare acts as a Merchant of Record, registering each access event and adding corresponding charges to bill the bot operator and distribute earnings to the publisher.
Cloudflare envisions a future where intelligent agents, such as personal assistants or contextual search engines, negotiate access to content on behalf of users.
Users could allocate budgets for their agents to access specialized databases, scientific research, or high-quality journalistic content in real-time under clear payment schemes. This model could pave the way for new licensing types and content markets where different prices are charged for model training, response generation, indexing, or commercial use.
Although the system is currently in beta and limited to Cloudflare’s enterprise users, the company has opened a registration form for content creators and crawler operators interested in participating in this initial phase.
