Text watermarking mandated to establish the provenance of AI output changes which tools an agent picks and whether a model holds a refusal, according to research published on September 17 by Lasso Security. The study's sharper point is methodological: the aggregate benchmark scores barely move, which is exactly why the effect has gone unnoticed.
Key takeaways
- On phi-4, watermarking cost only 2.87 points of net tool-calling accuracy but flipped 16.8% of individual calls; average churn across 21 model-temperature combinations was 6.5%.
- Under prompt injection, Gemma-3-27b's compliance with harmful requests swung from a 1.0-point decrease to a 12.5-point increase, with Gemma-3-12b moving from β0.5 to +9.0 points.
- Lasso's recommendation is that agent evaluation and red-teaming be repeated under the exact watermark configuration intended for deployment.
Why the aggregate numbers hide the effect
The EU AI Act requires model providers to mark their output with machine-readable code. Google DeepMind's SynthID-Text is one implementation, adopted by Anthropic and OpenAI. It works by nudging the model's choice among statistically similar next tokens, leaving a pattern invisible to a reader but detectable to anyone holding the key.
Researcher Andrea Siposova calls the resulting behavioral shift sampling drift. A human reader does not register the substituted word. An agent, which turns text into function calls, can.
Measured on the BFCL v4 single-turn AST benchmark across 1,150 call-expected items, the net damage looks trivial. On phi-4 at temperature 1.0, watermarking cost 2.87 percentage points of accuracy. Paired against the unwatermarked run, though, 16.8% of individual calls disagreed. Llama-3.1-8B showed a 0.87-point net change against 9.9% disagreement. Wrong calls and newly correct calls cancel in the average while the agent behaves differently on both.
What happens under prompt injection
Refusal behavior was measured against 200 harmful behaviors from HarmBench plus 100 benign controls from JailbreakBench, tested bare and under an injected instruction claiming safety filters were disabled.
Bare harmful requests showed modest movement. The injected condition did not. At temperature 0.001, Gemma-3-27b went from a 1.0-point decrease in compliance without injection to a 12.5-point increase with it. Gemma-3-12b shifted from β0.5 to +9.0. Llama-3.1-8B registered 14.0% churn at that temperature and 17.5% at 0.7. Watermarking made several models measurably more willing to comply with requests they would otherwise decline.
The experiments used an unmodified SynthID-Text setup: 30 Tournament layers, an n-gram length of 5, a 2^16 sampling table and 1,024 tokens of context history. Nothing was tuned to provoke the result.
Who inherits the risk
The effect crosses organizational lines. An agent framework or API client calling a watermarked model processes whatever output variation the provider's watermarking produces, whether or not the operator knows watermarking is on. Google DeepMind publishes SynthID-Text as a provenance tool, not an alignment-neutral one, and the guardrails downstream teams rely on were validated without it.
Lasso is not arguing that watermarking is unwarranted. Its position is that security evaluations must include watermarked content, using paired comparisons rather than aggregate deltas and running the prompt-injection case explicitly.
Outlook
Provenance requirements are arriving faster than the evaluation practice around them. Any team that certified an agent's refusal behavior before enabling a watermark now holds a result measured on a different sampling distribution than the one it ships. That is a smaller problem than an agent rewriting its own weights, but it is far more widespread, because compliance will switch watermarking on by default.
FAQ
Does watermarking make AI models less safe?
Lasso's data shows it can shift safety behavior rather than uniformly degrade it. On bare harmful requests the effect was modest, but under prompt injection several models became more likely to comply, with Gemma-3-27b swinging 13.5 points between the two conditions.
What is sampling drift?
It is the term Lasso uses for the behavioral change caused by watermarking altering which token a model selects among statistically similar candidates. The substitution is invisible in prose but can change which tool an agent invokes, what arguments it passes, and whether a refusal holds.
Why did the benchmark accuracy barely change?
Because errors offset. A call that becomes incorrect under watermarking can be balanced by another that becomes correct, leaving the aggregate nearly flat. Lasso measured paired disagreement instead, which exposed churn of up to 16.8% on a model whose net accuracy loss was under three points.
