Internet firm Cloudflare has started blocking AI web crawlers to prevent them from "accessing content without permission or compensation," by default according to an announcement on Tuesday.

The bargain between content makers and crawlers has broken down ai-pocalypse Cloudflare has started blocking AI web crawlers by default in a bid to become the internet's gatekeeper.… The term "gatekeeper" has been applied in a pejorative sense to platform companies like Apple and Google that use their contractual and technical control over operating systems to extract monopoly rents from developers within the platform ecosystem.

Cloudflare is testing a new way for websites to charge AI companies that use their content to train models, run inference, or power search. The new feature, called Pay per Crawl, lets website owners set their own terms for how and when AI bots can access their sites.