Cloudflare Hits the Kill Switch on AI Crawlers—And an Entire Industry Cheers

Key Points

1. Cloudflare CEO Matthew Prince announced that starting today, known AI bots will be blocked by default from scraping sites unless given explicit permission or payment.

2. The initiative, dubbed 'Content Independence Day,' is now the default for every new Cloudflare domain and has been supported by major media organizations like the Associated Press, Time, and Universal Music Group.

3. Cloudflare is introducing a 'pay-per-crawl' marketplace where publishers can set their own rates for AI bots, using the HTTP 402 'payment required' code.

4. According to Cloudflare, AI companies like OpenAI and Anthropic have been scraping content at ratios as high as 1,500 to 1 and 60,000 to 1, respectively, without fair compensation to publishers.

5. Existing Cloudflare customers can change their AI crawler settings anytime, while new domains are asked upfront whether they want to allow AI crawlers, with blocking as the default.