Today, Cloudflare unveiled “pay per crawl,” a way for websites to charge AI crawlers for access. Currently in private beta, the program provides more control over AI bots and lets each site set a price. The company also announced it will block AI crawlers from accessing web pages by default.
“If the Internet is going to survive the age of AI, we need to give publishers the control they deserve and build a new economic model that works for everyone – creators, consumers, tomorrow’s AI founders, and the future of the web itself,” Matthew Prince, co-founder and CEO of Cloudflare, said in a statement.
Pay per crawl integrates with existing web infrastructure, leveraging HTTP status codes and established authentication mechanisms to create a framework for paid content access.
Dig deeper: AI fatigue is real, and it’s costing brands more than engagement
Cloudflare said the fledgling effort is unlikely to be the only one to address the issue of compensation.
“We expect pay per crawl to evolve significantly,” the company said in a blog post. “We believe many different types of interactions and marketplaces can and should develop simultaneously. We are excited to support these various efforts and open standards.”
Content creators interested in joining the beta can sign up here.
Granular control over bots
Cloudflare said it is working with AI companies to verify the identity and purpose of AI crawlers, specifically whether the AI bots are crawling for training, content generation or search. Knowing this will give site owners and content creators more control over which bots they want to allow and which they don’t.
Any compensation scheme depends on AI companies’ participation. Cloudflare’s decision to block AI bots by default on new accounts incentivizes them to do so. An estimated 16% of global internet traffic goes directly through it’s service, according to a 2023 report. Denying access to even a fraction of that could have profound implications for AI companies already desperate for data to train on.
Enforcing a permission-based model for AI crawlers “will fundamentally change how AI companies access web content going forward,” Cloudflare wrote. “This significant shift means that every new domain starts with the default of control, and eliminates the need for webpage owners to manually configure their settings to opt out.”
Cloudflare has been working to stop AI bots from scraping websites for several years. In 2023, the company gave domain owners a way to block AI crawlers — but only the ones that followed the robots.txt file, which is more of a request than a rule. Later that year, it let sites block all AI bots. In March, the company added a new twist: any bots trying to sneak in get sent into what they call an “AI Labyrinth” to slow them down or block them entirely.
MarTech is owned by Semrush. We remain committed to providing high-quality coverage of marketing topics. Unless otherwise noted, this page’s content was written by either an employee or a paid contractor of Semrush Inc.
