Cloudflare announced that, as of September 15, 2026, its default settings will prevent "mixed‑use" crawlers from accessing any page that serves ads. The change forces a clear split between bots used for traditional search and those employed by AI agents or for model training, unless site owners manually override the setting.
The policy applies to all new Cloudflare customers, any new site added by existing users, and all free‑tier accounts. Cloudflare says the move protects publishers who want their content searchable but not freely harvested for AI training. The company singled out the dominant search engine—widely understood to be Google—as having roughly twice the data access of other AI firms, noting that Google’s own extended bot lets sites opt out of training use without hurting search rankings.
Alongside the crawler restriction, Cloudflare is piloting a "Pay Per Use" system that lets publishers earn fees when AI services such as Ceramic.ai or You.com display their premium content. By charging only when content creates value, the model aims to curb unnecessary bandwidth consumption and encourage AI providers to adopt transparent, purpose‑driven bots.
