1d:I[70616,["/_next/static/chunks/0kcf5dt6yxdox.js","/_next/static/chunks/1su5n1uv1ti4s.js","/_next/static/chunks/0gvke1wv6udfg.js","/_next/static/chunks/1dfrle3jgmlzh.js","/_next/static/chunks/162lgvduyr-fa.js","/_next/static/chunks/2gcnnopam8e9w.js","/_next/static/chunks/2g60iab5a1pkg.js"],"Breadcrumb"] 1e:I[90762,["/_next/static/chunks/0kcf5dt6yxdox.js","/_next/static/chunks/1su5n1uv1ti4s.js","/_next/static/chunks/0gvke1wv6udfg.js","/_next/static/chunks/1dfrle3jgmlzh.js","/_next/static/chunks/162lgvduyr-fa.js","/_next/static/chunks/2gcnnopam8e9w.js","/_next/static/chunks/2g60iab5a1pkg.js"],"ArticleGrid"] 1f:T1482,

Three of the largest crawler operators on the internet have agreed to a set of conditions written by a network provider. Cloudflare announced on September 15 that Apple, Google and Microsoft meet, or have committed to dates for meeting, a designation it calls Accountable - and alongside it shipped a setting that lets a site refuse AI training without dropping out of search.

Key takeaways

Seventeen percent of Cloudflare sites already block AI training, while under one percent block search - a gap mixed-use crawlers had made impossible to act on.
Earning Accountable status requires training and summary opt-outs, URL-level reporting on what was used, and a guarantee that refusing training will not move search rankings.
Microsoft's robots.txt support lands in early 2027, so the new setting conveys nothing to Bing until then.

The bind mixed-use crawlers created

A mixed-use crawler fetches a page once and puts it toward two ends: building a search index, and training a model. Publishers had no way to split those. Turning the crawler away to protect content from training also removed the pages from search, which for an ad-funded or subscription business is the expensive half of the bargain.

The numbers show how lopsided the preference is. Almost nobody turns down search, and roughly one site in six has reached for some mechanism against training. Cloudflare's new option, Disallow AI Training, writes a no-training directive into robots.txt through Bot Preference Sync while leaving Applebot, Bingbot and Googlebot free to index.

Why a file alone could not settle it

Publishing a directive and having it obeyed are different things. A robots.txt cannot identify who is fetching a page, determine the purpose, or stop a crawler that disregards it. Sitting in front of the origin, Cloudflare can do all three, then publish what each operator actually did on its Radar service - which is the enforcement the designation leans on.

Training-only crawlers were never the hard case. Amazon, Anthropic, Meta and OpenAI run separate bots for search and training, so the training one can be blocked with no cost to discoverability. Those operators are also counted as Accountable.

Where each operator stands

Google honors a Disallow on Google-Extended, offers a generative-results toggle in its webmaster portal, and says URL-level transparency tools are weeks away. Apple respects Applebot-Extended and reads nosnippet for summaries, but has no inspection tool yet and showed Cloudflare one in progress for next year. Both companies state that refusing training does not affect search rank.

Microsoft is the gap. Bing accepts a NOARCHIVE meta tag today, and its robots.txt handling of a no-training preference is targeted for early 2027. Site owners who want out of Bing training before then are pointed at NOARCHIVE or the content removal tools.

Summaries, not training, are the next argument

Cloudflare separates the two questions deliberately. Training asks whether content can build a model; summarization decides whether anyone visits the site at all. Its own figures make that tension concrete: more than half of consumers read search summaries, and those readers are over 40% more likely to stop searching afterward - yet visitors who do arrive from AI search convert at three to five times the rate of ordinary search traffic.

A site-wide yes or no is too blunt for that tradeoff, so the stated goal for early next year is control over how much content a summary may draw on, set once at the network instead of operator by operator, expressed through the emerging ai-prefs standard.

For Cloudflare, which has argued the web needs rebuilding around agent traffic rather than human traffic, writing the rules that Apple, Google and Microsoft then sign is the more durable win. The controls are live on every plan today.

FAQ

Does disallowing AI training remove my site from Google Search?

No. Accountable mixed-use crawlers such as Googlebot and Applebot keep crawling for search under the new setting, and both Google and Apple have stated that opting out of training does not affect search ranking. Choosing Block instead now stops those crawlers entirely, search included.

Which crawlers does Disallow AI Training stop?

All training crawlers except Accountable mixed-use ones, including the training-only bots run by Amazon, Anthropic, Meta and OpenAI. Blocking those carries no search penalty because each company operates a separate crawler for indexing.

Is the setting available on free plans?

Yes. Cloudflare says the controls reach all customers on all plans, configured per domain in Security Settings. Domains onboarded from September 15 are offered preset configurations that differ depending on whether the site serves advertising.

a:["$","div",null,{"className":"mx-auto max-w-7xl px-4 py-8 sm:px-6","children":[["$","script",null,{"type":"application/ld+json","dangerouslySetInnerHTML":{"__html":"{\"@context\":\"https://schema.org\",\"@type\":\"BreadcrumbList\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https://ainewsway.com\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"#robots-txt\",\"item\":\"https://ainewsway.com/tag/robots-txt\"}]}"}}],["$","script",null,{"type":"application/ld+json","dangerouslySetInnerHTML":{"__html":"{\"@context\":\"https://schema.org\",\"@type\":\"ItemList\",\"@id\":\"https://ainewsway.com/tag/robots-txt#itemlist\",\"name\":\"#robots-txt Articles\",\"numberOfItems\":1,\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"url\":\"https://ainewsway.com/articles/apple-google-and-microsoft-sign-on-to-cloudflares-crawler-rules-mu3ekhiz\",\"name\":\"Apple, Google and Microsoft Sign On to Cloudflare's Crawler Rules\"}]}"}}],["$","$L1d",null,{"items":[{"label":"# robots-txt"}]}],["$","h1",null,{"className":"mb-8 text-3xl font-bold text-zinc-900 dark:text-zinc-100","children":["#","robots-txt"]}],[["$","$L1e",null,{"articles":[{"id":"12dba881-c645-4efb-be0d-a99f90cd5149","slug":"apple-google-and-microsoft-sign-on-to-cloudflares-crawler-rules-mu3ekhiz","title":"Apple, Google and Microsoft Sign On to Cloudflare's Crawler Rules","subtitle":"A new Disallow AI Training setting ends the choice between search visibility and model training","content":"$1f","excerpt":"Cloudflare's Disallow AI Training setting lets sites refuse model training and stay in search, with Apple, Google and Microsoft signed on.","coverImage":"https://cdn.ainewsway.com/uploads/2026/09/2ae0806e-bb60-4af0-a49a-11dcb611fb24.webp","coverImageCaption":"The entrance to a Cloudflare office, where the company drafted the crawler conditions Apple, Google and Microsoft have now agreed to meet","authorId":"6773f67d-d7fa-4b19-8f38-384f10ac761b","status":"PUBLISHED","publishedAt":"2026-09-16T01:09:35.057Z","viewCount":0,"likeCount":0,"readingTime":4,"isFeatured":false,"isBreakingNews":false,"metaTitle":"Apple, Google, Microsoft Sign Cloudflare's Crawler Rules","metaDescription":"Cloudflare's Disallow AI Training keeps sites indexed while refusing model training. Apple and Google honor it; Bing support waits until 2027.","metaKeywords":["Cloudflare","Disallow AI Training","AI crawlers","robots.txt","Accountable crawlers","AI summaries","Googlebot","publisher controls"],"ogImage":null,"aiSummary":"Cloudflare launched a Disallow AI Training setting on September 15, 2026, letting sites refuse AI model training while staying indexed by crawlers that serve both search and training. Apple, Google and Microsoft meet its new Accountable designation, which requires opt-outs, URL-level reporting and a promise that refusing training will not hurt search rank. Microsoft's robots.txt support arrives in early 2027, and summary-level controls are promised for next year.","aiTags":[],"sentiment":null,"articleType":"NEWS","videoEmbed":null,"videoType":null,"seoScore":85,"isAutoGenerated":true,"isBackfill":false,"pipelineStage":null,"copyrightScore":null,"similarityScore":null,"matchedSourceId":null,"createdAt":"2026-09-16T01:09:09.180Z","updatedAt":"2026-09-16T01:09:35.058Z","contentUpdatedAt":null,"deletedAt":null,"author":{"id":"6773f67d-d7fa-4b19-8f38-384f10ac761b","name":"Seung Jung","username":"Seung Jung","avatar":null},"categories":[{"id":"8126f4b3-ce4a-464f-bc70-7c3c39d5540b","name":"Tech & Business","slug":"tech-business"}],"tags":[{"id":"b9caafc7-568e-48b3-97f7-d056a6e74871","name":"Cloudflare","slug":"cloudflare"},{"id":"f0abfea2-e16b-4c20-9cb1-0eec21c9a80f","name":"AI Crawlers","slug":"ai-crawlers"},{"id":"16a8f98d-4428-4244-b66e-5067b426bf88","name":"robots.txt","slug":"robots-txt"},{"id":"e864e19d-e1f6-4979-a19e-72dd66308f7e","name":"AI Summaries","slug":"ai-summaries"},{"id":"fedf64b9-a5d4-4be3-a9ec-c73924e5025e","name":"Web Publishing","slug":"web-publishing"}],"translations":[],"celebrities":[{"id":"a0544b08-282e-422d-9c24-40ec5a248380","nameKo":"클라우드플레어","nameEn":"Cloudflare","slug":"cloudflare","profileImage":null},{"id":"5c572908-2f5b-47b3-95e2-13ed3e298bf7","nameKo":"구글","nameEn":"Google","slug":"google","profileImage":null}]}]}],"$L20"]]}]

Subscribe to Newsletter