- The Midas Report
- Posts
- Cloudflare Turns Scraping Into a Billable API
Cloudflare Turns Scraping Into a Billable API
3 minute read
Midas AI Research
July 02, 2025
Cloudflare just shook things up by making AI data scraping a paid service… and it could change the web’s economic landscape. Their new Pay Per Crawl feature puts a toll booth in front of AI bots: every bit of content scraped from your site now comes with a price tag.
Right now, Pay Per Crawl is in private beta, with early adopters like The Atlantic, AP, Reddit, and Pinterest. Given that Cloudflare supports nearly 20% of global web traffic, this could quickly evolve into the standard for scraping. In essence, the era of free, unrestricted data scraping may be coming to an end. If your content has value, it can now become a revenue source, not just a source of exposure.
A New Price tag on the Open Web
This isn’t just another scraper blocking tool… it’s a business model. Cloudflare’s approach treats scraped content like an API call, one you can monetize. For publishers, SaaS companies, and knowledge businesses, it means data is becoming a real asset class. You can now earn money every time an AI model trains on your proprietary content.
Imagine training GPT style models that need your specialized reports or longform guides. Until now, all they had to do was crawl your site with impunity. With Pay Per Crawl, each page, image, or asset could be tracked and paid for. That’s an entirely new revenue stream waiting to be enabled for any content-heavy business that cares about its IP.
From Content to Asset Class
If your company produces unique content… think research papers, how to guides, or educational modules…. you need to treat it like an asset. The next step is to apply for Cloudflare’s beta, track crawl metrics for your site, and test monetization thresholds. This early access could let you set the pricing you believe reflects your content’s value… before the standard models finalize global toll rates.
And keep an eye out for dashboard tools that let you regulate who pays and how much. We’re talking the earliest stages of content rights management for AI and that means defining access instead of conceding control.