Cloudflare Flips the Defaults on AI Crawlers in Sept 2026

Cloudflare has quietly rewritten how it handles bots, and the shift is more interesting than the usual toggle-switch update. Instead of a single allow-or-block choice, site owners can now sort AI traffic into three categories: Search (crawlers that index and link back), Agent (bots acting on behalf of a user), and Training (the ones vacuuming content to feed models). Each bucket is governed separately, and the controls are on every plan, free tier included.

The date to circle: 15 September 2026

From that day, new domains behind Cloudflare will block Training and Agent crawlers by default on any page that shows ads. Search crawlers stay welcome. The wrinkle is multi-purpose bots. Googlebot, Applebot and BingBot get evaluated against both policies, so blocking Training means blocking them full stop, even if you were quite happy to be indexed. Existing customers who like the current behaviour can opt out through their security settings before the deadline.

BotBase and a content-use layer for robots.txt

Alongside the defaults, Cloudflare is launching BotBase, a searchable directory of known bots for Enterprise Bot Management customers. It is also introducing a content-use framework layered onto the ageing robots.txt standard. Site owners can declare whether a given bot may use their content:

• Immediately, with no storage
• By Reference, meaning excerpts and links
• In Full, allowing summaries or reproduction

robots.txt still has no teeth of its own. It cannot force anything. But Cloudflare says it will publicly track which Verified Bots actually respect the preferences declared by sites. Ignore them, or reproduce content wholesale, and a bot risks losing its Verified badge. That is not a legal remedy, but reputational pressure is often the only lever anyone has with well-funded scrapers.

Trying to see through the middleman

There is also a proposal for a transitive trust model using the HTTP Forwarded header. The idea is that when an AI agent acts through an intermediary, the site can see who is really behind the request and apply policies to that operator rather than the layer sitting in front. Cloudflare freely admits it will not work everywhere, especially where anonymity is part of the design.

What this means in practice

If you run a site behind Cloudflare, the calendar is the important bit. September 2026 sounds like a comfortable distance away, but the default flip is a real change in behaviour. The opt-out lives in security settings, and once the date passes, new domains inherit the stricter posture whether the owner has thought about it or not.

The broader shift matters too. For years, the deal for content on the open web was implicit: crawlers took what they wanted, publishers hoped for a bit of referral traffic in return. AI training broke that arrangement because there is no return trip, no link back, no visitor. Cloudflare is not solving that on its own, but sorting traffic by intent, and publishing which bots behave, is a more honest starting point than pretending robots.txt was ever enough.

Worth a look at your settings now, rather than the week the defaults change.