Cloudflare Accuses Perplexity AI of Using Stealth Crawlers to Evade Website Blocks
Main Idea
Cloudflare has delisted Perplexity for deceptive scraping practices, observing that the AI company bypassed website blocks and used undeclared crawlers to access restricted content.
Key Points
1. Cloudflare confirmed Perplexity's crawlers were blocked on specific pages but found the company used undeclared crawlers to bypass restrictions.
2. Perplexity's crawlers employed evasion techniques, including rotating IPs and impersonating browsers, to access content despite robots.txt prohibitions.
3. Cloudflare observed that Perplexity's declared crawlers generate 20-25 million requests daily, while undeclared crawlers add another 3-6 million requests.
4. Cloudflare has introduced measures like 'Content Independence Day' to block AI crawlers by default and protect content creators from unauthorized scraping.
5. Major publishers, including the Associated Press and Time, have joined Cloudflare's initiative to block AI crawlers, reflecting broader industry concerns.
Description
The $18 billion AI search startup allegedly disguised its bots to scrape content from sites that banned it, prompting Cloudflare to kick it out of its verified bot program and roll out new anti-scraping defenses.
Latest News
- CFTC Seeks Feedback on Plan to List Spot Crypto on Registered Exchanges2025-08-05 02:07:05
- North Korean Hackers Are Using Fake Job Offers to Breach Cloud Systems, Steal Billions in Crypto2025-08-05 00:25:40
- Google to Pit Top AI Models Against Each Other in Live Chess Tournament2025-08-05 00:14:22
- Cloudflare Accuses Perplexity AI of Using Stealth Crawlers to Evade Website Blocks2025-08-04 23:01:36
- A TON Strategy? Telegram-linked Online Marketer to Raise $558 Million for Treasury2025-08-04 21:42:15