The bargain between content makers and crawlers has broken down ai-pocalypse Cloudflare has started blocking AI web crawlers by default in a bid to become the internet's gatekeeper.… The term "gatekeeper" has been applied in a pejorative sense to platform companies like Apple and Google that use their contractual and technical control over operating systems to extract monopoly rents from developers within the platform ecosystem.

Cloudflare is testing a new way for websites to charge AI companies that use their content to train models, run inference, or power search. The new feature, called Pay per Crawl, lets website owners set their own terms for how and when AI bots can access their sites.

An internet infrastructure firm that routes 16% of global web traffic will block AI bots from new domains it hosts unless it has explicit permission from the owner.