A joint NSA, CISA and FBI advisory published on September 8 tells US AI providers to make targeted changes to the answers they return to accounts suspected of mining their models for training data. The agencies are specific about how: reduce the depth of the reasoning, present correct information by a different route, or let stylistic consistency slip โ alterations that survive casual inspection while making the output worth less as training material. It is a recommendation that would have read as sabotage a year ago, and the tactics carry uncomfortable side effects for paying customers.
Key takeaways
- Advisory AA26-251A asks providers to make targeted changes to responses for high-confidence distillation requests โ reduced reasoning depth, a different path to the same correct answer, stylistic inconsistency โ to cut their training value.
- The recommended detection signal is commercial rather than technical: accounts that sit at maximum usage immediately after signup, and subscription-to-usage ratios that do not match human behavior.
- The extraction tradecraft described is procurement fraud, not intrusion โ bulk premium subscriptions, gray-market proxies and metadata-scrubbing API aggregators.
Why the defenses target billing data, not traffic
The mitigations listed in the joint NSA, CISA and FBI advisory say a lot about where the agencies think the leverage is. Request-level inspection barely features. Instead the guidance points providers at their own billing systems: watch the ratio of subscriptions purchased to usage consumed, and flag new accounts that immediately run at their ceiling.
That follows from the tradecraft. What the agencies describe is not a breach. Requests were spread across fraudulently created accounts with obfuscated metadata, routed through gray-market "transfer stations" that defeat geographic restrictions, and funneled through third-party API aggregators that automatically scrub identifying details. Bulk purchases of premium subscriptions supplied the throughput, with per-domain request volumes ranging from thousands to millions.
Every one of those requests is individually legitimate. The pattern only resolves at the account and payment layer, which is exactly where the advisory sends defenders.
What altered answers would cost
The response-alteration recommendation is the genuinely novel item. Changing what a suspected extraction account receives devalues the synthetic corpus a distiller is assembling, and it does so without the attacker learning they have been detected โ the advantage a hard block gives up immediately.
The advisory stops short of asking providers to return false information. Its worked examples keep the answer correct and attack the material around it: shallower reasoning, an alternate derivation, inconsistent style. That distinction matters legally and commercially, because a provider that knowingly served wrong answers to a paying account would be defending a very different claim.
The cost still lands on false positives. A research lab batching evaluations, an enterprise running a nightly pipeline and a distillation campaign all look similar in usage-ratio terms. Reasoning depth is precisely what a legitimate customer is paying for, and silently serving less of it is a product decision with no clean remediation path โ which is presumably why no major provider has said publicly that it does this.
The agencies also flagged prompt injection aimed at eliciting chain-of-thought traces. Reasoning traces are far more valuable as training material than final answers, since they expose the path rather than its conclusion โ which is part of why several labs now summarize or withhold them by default, and why the advisory's countermeasures target reasoning first.
The accusation behind the guidance
The advisory names six China-based companies โ DeepSeek, Moonshot AI, Alibaba, MiniMax, StepFun and Z.AI โ and estimates billions of tokens were extracted across millions of exchanges with models from Anthropic, OpenAI, Google and xAI since at least late 2024. Distillation, the agencies wrote, is the critical core of these companies' development rather than a supplement to it.
Anthropic reached a similar conclusion on its own when it documented roughly 200 million exchanges earlier. A vendor complaint invites the obvious discount; an intelligence-community advisory carries different weight, and it arrives with an operational checklist attached.
Outlook
The practical question for developers is whether the friction lands on them. Query volume quotas, adversarial input detection and cross-organization indicator sharing all raise the cost of legitimate high-volume API work, and none of the named companies had publicly denied the allegations at the time of publication. If providers adopt the full checklist, the visible change will not be an announcement โ it will be tighter rate limits and slower onboarding for anyone whose traffic looks unusual.
FAQ
What is knowledge distillation in AI?
Distillation trains a smaller or newer model on the outputs of a larger, more capable one, transferring behavior without access to the original weights. The technique is standard and legitimate when applied to a model you own. It becomes contested when the outputs come from a competitor's commercial API in violation of its terms of service.
Does the advisory tell providers to give wrong answers?
No. AA26-251A recommends "targeted changes in response to high-confidence malicious distillation requests," and its examples are reduced reasoning depth, presenting correct information with different reasoning, and stylistic inconsistency. The stated goal is to reduce training usefulness while evading detection, not to serve incorrect information.
Does this advisory create any legal obligation for AI companies?
No. AA26-251A is advisory guidance, not a regulation, so its mitigations are voluntary. Its practical weight comes from being a joint NSA, CISA and FBI product, which makes it difficult for providers and their enterprise customers to treat the underlying threat assessment as a competitive talking point.
