- The Midas Report
- Posts
- Stealth Tactics and Scraped Secrets as Perplexity AI Sparks Showdown with Cloudflare
Stealth Tactics and Scraped Secrets as Perplexity AI Sparks Showdown with Cloudflare
3 min read.
Midas AI
August 06, 2025
In the rapidly evolving landscape of artificial intelligence, the methods by which AI companies gather data have become a contentious issue. A recent dispute between Cloudflare, a leading internet infrastructure provider, and Perplexity, an AI powered search engine, underscores the complexities and ethical considerations surrounding web scraping practices.
Allegations of Stealth Crawling
Cloudflare has accused Perplexity of employing "stealth crawling" techniques to access website content without adhering to established protocols. According to Cloudflare's findings, Perplexity's declared bots respect the directives set by website owners.
However, when these bots encounter restrictions, Perplexity allegedly switches to undeclared crawlers that mimic regular user behavior, thereby circumventing blocks and accessing content that was explicitly off limits. This behavior was observed across tens of thousands of domains and millions of requests per day.
To substantiate these claims, Cloudflare conducted tests by creating new domains with strict robots.txt files designed to block all automated access. Despite these measures, Perplexity was able to retrieve detailed information from these protected domains, indicating the use of stealth crawling techniques and non disclosed IP addresses.
Perplexity's Response and Industry Implications
Perplexity has refuted Cloudflare's allegations, asserting that their AI agents operate differently from traditional web crawlers. The company claims that their agents fetch information in real time in response to user queries, similar to how a browser or email client functions, and that this process does not constitute unauthorized scraping. Perplexity also suggested that Cloudflare's findings might be a result of misattribution or a misunderstanding of their operational methods.
This dispute highlights a broader issue within the AI industry: the balance between data accessibility and the rights of content creators. As AI models require vast amounts of data to function effectively, the methods of acquiring this data have come under scrutiny. Unauthorized scraping not only raises ethical concerns but also poses legal risks, as evidenced by lawsuits from major media organizations against AI companies for copyright infringement.
Broader Industry Implications and the Need for Standards
The conflict between Cloudflare and Perplexity is emblematic of a larger challenge facing the AI industry: establishing clear and enforceable standards for data collection. The lack of consensus on what constitutes acceptable web scraping practices has led to tensions between AI companies and content creators. This situation underscores the need for industry wide guidelines that balance the advancement of AI technologies with respect for intellectual property rights.
Furthermore, the dispute raises questions about the effectiveness of current mechanisms, such as robots.txt files, in controlling access to web content. As AI technologies become more sophisticated, there is a pressing need for more robust and transparent methods to manage data collection practices.