1d:I[70616,["/_next/static/chunks/0kcf5dt6yxdox.js","/_next/static/chunks/1su5n1uv1ti4s.js","/_next/static/chunks/0gvke1wv6udfg.js","/_next/static/chunks/1dfrle3jgmlzh.js","/_next/static/chunks/162lgvduyr-fa.js","/_next/static/chunks/2gcnnopam8e9w.js","/_next/static/chunks/2g60iab5a1pkg.js"],"Breadcrumb"] 1e:I[90762,["/_next/static/chunks/0kcf5dt6yxdox.js","/_next/static/chunks/1su5n1uv1ti4s.js","/_next/static/chunks/0gvke1wv6udfg.js","/_next/static/chunks/1dfrle3jgmlzh.js","/_next/static/chunks/162lgvduyr-fa.js","/_next/static/chunks/2gcnnopam8e9w.js","/_next/static/chunks/2g60iab5a1pkg.js"],"ArticleGrid"] 1f:T1482,

Three of the largest crawler operators on the internet have agreed to a set of conditions written by a network provider. Cloudflare announced on September 15 that Apple, Google and Microsoft meet, or have committed to dates for meeting, a designation it calls Accountable - and alongside it shipped a setting that lets a site refuse AI training without dropping out of search.

Key takeaways

Seventeen percent of Cloudflare sites already block AI training, while under one percent block search - a gap mixed-use crawlers had made impossible to act on.
Earning Accountable status requires training and summary opt-outs, URL-level reporting on what was used, and a guarantee that refusing training will not move search rankings.
Microsoft's robots.txt support lands in early 2027, so the new setting conveys nothing to Bing until then.

The bind mixed-use crawlers created

A mixed-use crawler fetches a page once and puts it toward two ends: building a search index, and training a model. Publishers had no way to split those. Turning the crawler away to protect content from training also removed the pages from search, which for an ad-funded or subscription business is the expensive half of the bargain.

The numbers show how lopsided the preference is. Almost nobody turns down search, and roughly one site in six has reached for some mechanism against training. Cloudflare's new option, Disallow AI Training, writes a no-training directive into robots.txt through Bot Preference Sync while leaving Applebot, Bingbot and Googlebot free to index.

Why a file alone could not settle it

Publishing a directive and having it obeyed are different things. A robots.txt cannot identify who is fetching a page, determine the purpose, or stop a crawler that disregards it. Sitting in front of the origin, Cloudflare can do all three, then publish what each operator actually did on its Radar service - which is the enforcement the designation leans on.

Training-only crawlers were never the hard case. Amazon, Anthropic, Meta and OpenAI run separate bots for search and training, so the training one can be blocked with no cost to discoverability. Those operators are also counted as Accountable.

Where each operator stands

Google honors a Disallow on Google-Extended, offers a generative-results toggle in its webmaster portal, and says URL-level transparency tools are weeks away. Apple respects Applebot-Extended and reads nosnippet for summaries, but has no inspection tool yet and showed Cloudflare one in progress for next year. Both companies state that refusing training does not affect search rank.

Microsoft is the gap. Bing accepts a NOARCHIVE meta tag today, and its robots.txt handling of a no-training preference is targeted for early 2027. Site owners who want out of Bing training before then are pointed at NOARCHIVE or the content removal tools.

Summaries, not training, are the next argument

Cloudflare separates the two questions deliberately. Training asks whether content can build a model; summarization decides whether anyone visits the site at all. Its own figures make that tension concrete: more than half of consumers read search summaries, and those readers are over 40% more likely to stop searching afterward - yet visitors who do arrive from AI search convert at three to five times the rate of ordinary search traffic.

A site-wide yes or no is too blunt for that tradeoff, so the stated goal for early next year is control over how much content a summary may draw on, set once at the network instead of operator by operator, expressed through the emerging ai-prefs standard.

For Cloudflare, which has argued the web needs rebuilding around agent traffic rather than human traffic, writing the rules that Apple, Google and Microsoft then sign is the more durable win. The controls are live on every plan today.

FAQ

Does disallowing AI training remove my site from Google Search?

No. Accountable mixed-use crawlers such as Googlebot and Applebot keep crawling for search under the new setting, and both Google and Apple have stated that opting out of training does not affect search ranking. Choosing Block instead now stops those crawlers entirely, search included.

Which crawlers does Disallow AI Training stop?

All training crawlers except Accountable mixed-use ones, including the training-only bots run by Amazon, Anthropic, Meta and OpenAI. Blocking those carries no search penalty because each company operates a separate crawler for indexing.

Is the setting available on free plans?

Yes. Cloudflare says the controls reach all customers on all plans, configured per domain in Security Settings. Domains onboarded from September 15 are offered preset configurations that differ depending on whether the site serves advertising.

a:["$","div",null,{"className":"mx-auto max-w-7xl px-4 py-8 sm:px-6","children":[["$","script",null,{"type":"application/ld+json","dangerouslySetInnerHTML":{"__html":"{\"@context\":\"https://schema.org\",\"@type\":\"BreadcrumbList\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https://ainewsway.com\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"#cloudflare\",\"item\":\"https://ainewsway.com/tag/cloudflare\"}]}"}}],["$","script",null,{"type":"application/ld+json","dangerouslySetInnerHTML":{"__html":"{\"@context\":\"https://schema.org\",\"@type\":\"ItemList\",\"@id\":\"https://ainewsway.com/tag/cloudflare#itemlist\",\"name\":\"#cloudflare Articles\",\"numberOfItems\":1,\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"url\":\"https://ainewsway.com/articles/apple-google-and-microsoft-sign-on-to-cloudflares-crawler-rules-mu3ekhiz\",\"name\":\"Apple, Google and Microsoft Sign On to Cloudflare's Crawler Rules\"}]}"}}],["$","$L1d",null,{"items":[{"label":"# cloudflare"}]}],["$","h1",null,{"className":"mb-8 text-3xl font-bold text-zinc-900 dark:text-zinc-100","children":["#","cloudflare"]}],[["$","$L1e",null,{"articles":[{"id":"12dba881-c645-4efb-be0d-a99f90cd5149","slug":"apple-google-and-microsoft-sign-on-to-cloudflares-crawler-rules-mu3ekhiz","title":"Apple, Google and Microsoft Sign On to Cloudflare's Crawler Rules","subtitle":"A new Disallow AI Training setting ends the choice between search visibility and model training","content":"$1f","excerpt":"Cloudflare's Disallow AI Training setting lets sites refuse model training and stay in search, with Apple, Google and Microsoft signed on.","coverImage":"https://cdn.ainewsway.com/uploads/2026/09/2ae0806e-bb60-4af0-a49a-11dcb611fb24.webp","coverImageCaption":"The entrance to a Cloudflare office, where the company drafted the crawler conditions Apple, Google and Microsoft have now agreed to meet","authorId":"6773f67d-d7fa-4b19-8f38-384f10ac761b","status":"PUBLISHED","publishedAt":"2026-09-16T01:09:35.057Z","viewCount":0,"likeCount":0,"readingTime":4,"isFeatured":false,"isBreakingNews":false,"metaTitle":"Apple, Google, Microsoft Sign Cloudflare's Crawler Rules","metaDescription":"Cloudflare's Disallow AI Training keeps sites indexed while refusing model training. Apple and Google honor it; Bing support waits until 2027.","metaKeywords":["Cloudflare","Disallow AI Training","AI crawlers","robots.txt","Accountable crawlers","AI summaries","Googlebot","publisher controls"],"ogImage":null,"aiSummary":"Cloudflare launched a Disallow AI Training setting on September 15, 2026, letting sites refuse AI model training while staying indexed by crawlers that serve both search and training. Apple, Google and Microsoft meet its new Accountable designation, which requires opt-outs, URL-level reporting and a promise that refusing training will not hurt search rank. Microsoft's robots.txt support arrives in early 2027, and summary-level controls are promised for next year.","aiTags":[],"sentiment":null,"articleType":"NEWS","videoEmbed":null,"videoType":null,"seoScore":85,"isAutoGenerated":true,"isBackfill":false,"pipelineStage":null,"copyrightScore":null,"similarityScore":null,"matchedSourceId":null,"createdAt":"2026-09-16T01:09:09.180Z","updatedAt":"2026-09-16T15:13:36.890Z","contentUpdatedAt":"2026-09-16T15:13:36.890Z","deletedAt":null,"author":{"id":"6773f67d-d7fa-4b19-8f38-384f10ac761b","name":"Seung Jung","username":"Seung Jung","avatar":null},"categories":[{"id":"8126f4b3-ce4a-464f-bc70-7c3c39d5540b","name":"Tech & Business","slug":"tech-business"}],"tags":[{"id":"b9caafc7-568e-48b3-97f7-d056a6e74871","name":"Cloudflare","slug":"cloudflare"},{"id":"f0abfea2-e16b-4c20-9cb1-0eec21c9a80f","name":"AI Crawlers","slug":"ai-crawlers"},{"id":"16a8f98d-4428-4244-b66e-5067b426bf88","name":"robots.txt","slug":"robots-txt"},{"id":"e864e19d-e1f6-4979-a19e-72dd66308f7e","name":"AI Summaries","slug":"ai-summaries"},{"id":"fedf64b9-a5d4-4be3-a9ec-c73924e5025e","name":"Web Publishing","slug":"web-publishing"}],"translations":[{"id":"127a3838-4f45-45fd-9595-57ec85521683","language":"ko","title":"애플·구글·MS, 클라우드플레어가 쓴 크롤러 규칙에 서명했다","subtitle":"새 'AI 학습 거부' 설정으로 검색 노출과 모델 학습 사이에서 택일할 필요가 사라졌다","excerpt":"클라우드플레어의 'Disallow AI Training' 설정은 검색 노출을 유지한 채 모델 학습만 거부하게 해준다. 애플과 구글, 마이크로소프트가 이 기준을 받아들였다.","metaTitle":"애플·구글·MS, 클라우드플레어 크롤러 규칙 수용","metaDescription":"클라우드플레어의 'Disallow AI Training'은 검색 색인을 유지하면서 모델 학습만 거부한다. 애플과 구글은 이미 준수하고, 빙 지원은 2027년으로 미뤄졌다."},{"id":"251434d0-eced-477f-8fe8-95b1f6d3e0c5","language":"ja","title":"Apple、Google、MicrosoftがCloudflareのクローラー規約を受け入れ","subtitle":"新しい「AI学習を拒否」設定により、検索での露出とモデル学習のどちらかを選ぶ必要がなくなる","excerpt":"Cloudflareの「Disallow AI Training」設定は、検索から外れることなくモデル学習だけを拒否できるようにします。Apple、Google、Microsoftがこの基準を受け入れました。","metaTitle":"Apple・Google・MSがCloudflareのクローラー規約を受諾","metaDescription":"Cloudflareの「Disallow AI Training」は検索インデックスを保ったままモデル学習を拒否します。AppleとGoogleは対応済み、Bingの対応は2027年まで待つことになります。"},{"id":"1a9aca8c-b8df-4856-b936-ac83b3acc003","language":"zh","title":"Apple、Google 與微軟簽下 Cloudflare 的爬蟲規則","subtitle":"新的「拒絕 AI 訓練」設定，終結了在搜尋曝光與模型訓練之間二選一的處境","excerpt":"Cloudflare 的「Disallow AI Training」設定，讓網站能在保留搜尋曝光的同時拒絕模型訓練，Apple、Google 與微軟均已接受這套標準。","metaTitle":"Apple、Google、微軟簽下 Cloudflare 爬蟲規則","metaDescription":"Cloudflare 的「Disallow AI Training」讓網站保留索引卻能拒絕模型訓練。Apple 與 Google 已遵守，Bing 的支援要等到 2027 年。"},{"id":"a66097f2-8fae-4e82-8925-cd6c0c2351f4","language":"es","title":"Apple, Google y Microsoft firman las reglas de rastreo de Cloudflare","subtitle":"Una nueva opción para rechazar el entrenamiento de IA elimina la disyuntiva entre aparecer en búsquedas y alimentar modelos","excerpt":"La opción Disallow AI Training de Cloudflare permite a los sitios rechazar el entrenamiento de modelos sin salir del buscador, con Apple, Google y Microsoft ya adheridos.","metaTitle":"Apple, Google y Microsoft firman las reglas de Cloudflare","metaDescription":"Disallow AI Training de Cloudflare mantiene los sitios indexados pese a rechazar el entrenamiento. Apple y Google ya lo respetan; Bing, en 2027."}],"celebrities":[{"id":"a0544b08-282e-422d-9c24-40ec5a248380","nameKo":"클라우드플레어","nameEn":"Cloudflare","slug":"cloudflare","profileImage":null},{"id":"5c572908-2f5b-47b3-95e2-13ed3e298bf7","nameKo":"구글","nameEn":"Google","slug":"google","profileImage":null}]}]}],"$L20"]]}]

#cloudflare

Apple、Google、MicrosoftがCloudflareのクローラー規約を受け入れ

ニュースレター購読

#cloudflare

Apple、Google、MicrosoftがCloudflareのクローラー規約を受け入れ