As time goes on, just saying 'no' to AI feels more and more futile. However, Cloudflare is announcing a few more tools for your anti-AI arsenal that put some of the power back in your hands. Cloudflare offers a range of network services from domain registration and content delivery to DDOS mitigation. Adding to that quiver, Cloudflare is launching the sharp and pointy Pay Per Crawl scheme, which aims to hit AI companies scraping online content where it hurts—namely, their deep pockets.
Last year, Cloudflare began allowing customers . This AI Audit tool also allows users 
to pick and choose which bots they want to block, as well as offering insights into which LLM or gen-AI is regurgitating their content. Now, that AI crawlers will be blocked by default, describing it 
as "the first step toward [[link]] a more sustainable future for both content creators and AI innovators."
Users who want to have their content scraped for free can still turn this default setting off, but Cloudflare's second announcement seemingly asks the question, 'Why would you want to?'
, a scheme announced last year that allows users to charge AI companies wanting to scrape their content, . To date, Cloudflare has not revealed the names of any Pay [[link]] Per Crawl participants aside from , the startup behind AI search engine .
Founder and CEO Bill Gross offered comment as part of Cloudflare's announcement regarding AI crawlers, saying, "Our whole mission is to protect and elevate human creativity in the AI era. That’s why we’re proud to be one of the first AI companies to participate in Cloudflare’s initiative to create a new permission-based model for the Internet. We believe that creators and publishers deserve to be fairly compensated for the value they bring, and we’re thrilled to support Cloudflare in helping make that vision a reality."
It doesn't seem unlikely that bigger AI companies would go for a scheme like this. After all, OpenAI has already struck content partnerships with a number of media companies, . Pay Per Crawl feels like an attempt to level the playing field, giving smaller content creators a leg to stand on, who would otherwise simply not have an entire team to dedicate to working out the terms of a content partnership.
But let's reverse for a second, especially because, as , not all web crawlers are bad. For instance, the Internet Archive's preservation work in particular relies heavily on them. Last year, the Pew 
Research Center found that a quarter of all webpages that existed between 2013 and 2023 . Web crawlers are one way to preserve pages like that before they disappear in a puff of broken links and long-forgotten hosting services.
AI crawlers, however, are a bit of a menace as they scrape without permission from the owner of the content, and can end up overly taxing servers by behaving not unlike a DDOS attack. As previously mentioned, Cloudflare is no stranger to DDOS attacks, telling our Nick last month that their network blocks about '.' The company also shared that its network recently mitigated the largest attack ever recorded,.
As such, Cloudflare's AI blocking tools leverage a mix of machine learning and behavioural analysis to differentiate between these two types of web crawlers. Cloudflare has also found some success fingerprinting naughty bots by . Now that's what I call fighting fire with fire…or a flinging slop fight. At any rate, would it be naive to feel just a little optimistic about Cloudflare's [[link]] push for a 'Permission-Based Model for the Internet'? Time will tell.
👉👈
1. Best overall:
2. Best budget:
3. Best compact:
4. Alienware:
5. Best mini PC:
