I switched on Cloudflare's Block AI bots last year. What happens on September 15?

Under the official rules, your site will block multi-purpose crawlers that combine search and training, such as Googlebot, Applebot, and Bingbot, following the scope of your current configuration, all pages or ad pages only. To keep search crawlers, mark the opt-out in your Security settings before September 15.

My site has no ads. Do I still need to care?

The ad-page default does not apply to you, but the multi-purpose crawler rule has nothing to do with ads. As long as your training block is on, Googlebot gets blocked. Sites without ads should still check the legacy toggle.

I want to block training crawlers but keep Googlebot. What do I do?

Before September 15, mark in Security settings that you want no changes on training crawlers that also crawl for search, in other words, opt out. Single-purpose training crawlers stay blocked. Domains onboarding after September 15 can also adjust the Search, Agent, and Training presets manually.

Can a Google-Extended line in robots.txt replace Cloudflare's blocking?

No. Google-Extended only states a preference. It controls whether content can be used for Gemini training, does not affect search inclusion, is not a ranking signal, and has no separate user agent. Cloudflare's block runs at the request layer. They are different tools, and teams that want both a training preference and full indexing can use the two layers separately.

How do I tell whether my site is being wrongly blocked?

Check three places. Cloudflare Security Events for Googlebot-related block records; Search Console crawl stats for a sudden drop after September 15 with unchanged content; and server logs for 403s on Googlebot requests.

Cloudflare's September 15 Rules Block Googlebot When You Block AI Training

One site owner hit this earlier than the official timeline. In early July, he started seeing verified Googlebot requests come back with 403s, and Cloudflare’s security event log labeled the action “Blocked by ‘Block AI training crawlers’.” He posted about it on the Cloudflare community in late August. The date Cloudflare has set for the new rules to take effect is September 15, two months after his first 403.

I went through Cloudflare’s official blog post, its press release, and the community threads to get the whole story straight. With three weeks left before September 15, this is the last real window to check your settings.

The rules themselves are simple enough. On July 1 this year, Cloudflare announced at its second Content Independence Day that it would sort AI crawlers into three behavior categories. Search indexes pages so they can answer questions later. Agent acts in real time on a person’s behalf, like chat fetches and browser-driving agents. Training pulls content to train or fine-tune models. The three categories can be allowed or blocked separately, including on the free plan. Two default rules land on September 15.

The Two Rules Taking Effect on September 15

The first targets pages with ads. For new domains onboarding Cloudflare and new sites added by existing customers, Training and Agent will be blocked by default on pages that display ads, while Search stays allowed. The same day, existing free accounts that never changed their settings get moved to these defaults too.

The second rule reaches further. Cloudflare will judge multi-purpose crawlers by the most restrictive applicable rule. The official post puts it plainly: multi-purpose crawlers “such as Googlebot, Applebot, and BingBot will be blocked by customers who have selected to block Training”, whether through the new category options or the legacy Block AI bots service. In other words, if any version of an “AI blocking” switch is still on, Google’s crawler gets stopped at the door after September 15.

Anyone who wants to keep Googlebot can mark the opt-out in their Security settings any time before September 15, confirming they want no changes on training crawlers that also crawl for search. Cloudflare says it will keep sending notifications as the date approaches.

Why Cloudflare Put the Bill on Google’s Desk

Cloudflare’s own numbers explain the move. By its count, more than half of all requests on its network now come from machines. Training crawlers account for the majority of crawler requests, up from around a fifth in spring 2025 to 52% this June. Daily AI agent requests rose more than 1,700% over the past year. What irritates it even more is duplicate fetching: over half of AI crawler traffic re-fetches pages that have not changed.

Last July Cloudflare ran an even sharper calculation. For every real human visitor returned, Google made 14 crawls, OpenAI 1,700, and Anthropic 73,000. Behind that math is the judgment it repeats everywhere. If you run a small site, you have two problems: someone is training models on your content, and nobody can find you. So site owners face a Faustian bargain, show up in search and let AI train on you, or protect your content and lose discoverability.

Cloudflare aims this at the big labs that bundle search and training into one crawler. The blog post calls on any company that builds search indexes, acts as an agent, and collects data for training to split its automation into three separate crawlers. Matthew Prince put it more politely in the press release, saying he hopes the new defaults encourage mixed-use crawlers to separate search from agent use and training. The press release also did the arithmetic: the largest search engine has access to about 2X more information than leading AI companies, because it makes it hard for customers to stay discoverable without also being used for AI.

Google Keeps a Different Set of Books

Google’s arrangement is indeed the other kind. Googlebot is a single user agent that both indexes for search and crawls for AI features. Anyone who wants to refuse training has to use a Google-Extended directive in robots.txt. According to Google’s own crawler documentation, this directive controls whether content may be used to train future generations of Gemini models. It does not affect a site’s inclusion in Google Search, and it is not a ranking signal. Google-Extended has no separate user agent either; the crawling is still done by Googlebot, and the directive itself is only a control signal.

That is where the two systems part ways. At the robots.txt layer, a site owner can keep search and refuse training, because the two preferences can be separated after the fetch. Cloudflare enforces at the request layer. When a crawler arrives at the door, it is either let in or turned away, and half a Googlebot cannot be blocked. So as long as Google does not split search and training into two crawlers, blocking training crawlers on Cloudflare costs you Googlebot. So far, Google has shown no sign of splitting.

Back to the site owner from the opening. The 403s he saw from early July came two months before the official date, and Cloudflare has not given a confirmed explanation. Another site owner reported similar intermittent blocking in late July. These reports are a reminder that rule boundaries wobble in real deployments. Waiting until after September 15 to check the logs may be too late.

Who Actually Gets Hit

The first group is anyone who switched on Block AI bots last year and never touched it again. In July 2025, when the first wave of “block AI” discussion peaked, plenty of sites flipped the toggle as a risk-free defensive move and forgot about it. After September 15, that toggle will take Googlebot down with it.

The second group is domains that onboard Cloudflare after September 15. Training and Agent get blocked on ad pages by default.

The third is free accounts that never changed their settings. They get moved to the new defaults that day.

The reverse cases are unaffected. An existing paid zone with no AI blocking enabled changes nothing. A site with no ads anywhere is out of reach of the ad-page default, though the multi-purpose rule still applies if your training block is on. And if your site is not behind Cloudflare, none of this touches you.

Four Things Export Sites Should Check Now

First, confirm whether your site is behind Cloudflare at all. Plenty of SEO teams do not know, because IT set it up years ago. Check the DNS, or just ask.

Second, open the bot traffic section in Security settings. Look at the current value of the legacy Block AI bots toggle: all pages, ad pages only, or off. Note who switched it on. The new Search, Agent, and Training presets live here too.

Third, confirm whether any page on your site carries ads. Sponsored placements, partner banners, and ads on resource pages all count. This decides whether the ad-page default can reach you.

Fourth, take a baseline. Export crawl stats from Search Console, screenshot the bot settings, and save a copy of the current robots.txt. If crawl volume drops after September 15 while content stays the same, that is the sign of a misfire.

Three Paths, Three Prices

For most export sites, the question comes down to one choice.

The first path is the easiest: opt out. Before September 15, mark in Security settings that you want to keep training crawlers that also crawl for search, and keep blocking single-purpose training crawlers. Any site that sells things should take this path, keeping both SEO and GEO open.

The second path is to accept the new rules, keep blocking Training, and let Googlebot go with it. That suits publishers who treat content as a bargaining chip and have other traffic sources. Most commercial sites cannot afford to vanish from search results.

The third is a middle way: block Training only on pages with ads, and watch how crawling develops.

For a site that sells things, blocking Search crawlers is close to indefensible, because citations inside AI answers are already a discovery channel in their own right. Teams that really care about training data can use the two layers separately. Google-Extended in robots.txt and the Content Signals use= field Cloudflare began testing in July state your preferences; WAF rules and bot management enforce them. Preferences do not cost you indexing. Enforcement is what keeps crawlers out.

September 15 is almost here. Cloudflare says it will keep notifying customers, but those emails usually land with whoever registered the domain four years ago. Three weeks are left. Opening the dashboard yourself is more reliable than waiting for a notice addressed to nobody you know.