LLM Watermarking: Invisible Tags and Their Effect on AI Agent Decisions

When Anthropic announced that its upcoming Claude models will embed an invisible watermark, the AI community realized that a technique once used only for provenance is now entering the regulatory arena. The EU AI Act already obliges providers to mark synthetic text in a machine‑readable way, and the new watermarking method, based on Google DeepMind's SynthID‑Text, does exactly that. But beyond labeling, the watermark subtly reshapes how large language models (LLM) behave, influencing both what they say and what they do.

Why Watermarking Matters Now

Article 50(2) of the EU AI Act requires AI systems that generate synthetic text to embed detectable signals without compromising effectiveness. This legal pressure pushes developers to adopt watermarking at the model level rather than as a post‑processing step. By integrating the signal directly into the token generation process, providers can claim compliance while keeping the output indistinguishable to end users. However, the integration occurs at the core of the generation algorithm, where safety‑critical decisions are made.

Technical Mechanism of SynthID‑Text

SynthID‑Text implements a technique called tournament sampling. Instead of sampling tokens purely from the raw probability distribution, the algorithm introduces a hidden bias that nudges the model toward a subset of tokens that encode a watermark. The process is non‑distortionary in expectation: over many generations with random keys, the overall token distribution remains unchanged. Yet each individual generation, governed by a fixed secret key, can select a slightly different token than it would have without the watermark.

Key parameters of the implementation used in the study include:

• 30 tournament layers
• n‑gram length of 5
• sampling table size 216
• context history of 1,024 tokens

The only component that differs between a watermarked and an unwatermarked run is the SynthIDTextWatermarkLogitsProcessor from HuggingFace, which applies the bias during logits calculation.

Behavioral Drift: From Token Choice to Tool Use

In a plain text generation scenario, a slight shift in token probability may appear harmless. In structured outputs—such as JSON objects, function calls, or tool‑invocation arguments—the same shift can have outsized consequences. For example, a model that must produce a JSON field "amount": 100 may instead emit "amount": 10 because the watermark nudged the token "1" over "0". When an AI agent uses these outputs to call external tools, a single altered token can change the selected tool, the arguments passed, or even the entire execution path.

Because the watermark operates at the token level, the phenomenon is called sampling drift. It is not a deterministic bug; rather, it is a probabilistic deviation that depends on the secret key and the model's uncertainty at each step.

Empirical Findings: Refusal and Tool‑Calling Shifts

The authors conducted two paired experiments. The first measured refusal behavior on 200 harmful prompts from HarmBench and 100 benign prompts from JailbreakBench, both with and without a fixed prompt‑injection attack. The second evaluated tool‑calling accuracy on the BFCL v4 single‑turn AST benchmark.

Results showed that:

• Six of seven models exhibited a drop in refusal accuracy when watermarked, with four models showing statistically significant declines.
• Tool‑calling correctness decreased on five of seven models, often because the watermark altered argument strings while leaving the tool name intact.
• The direction of change varied by watermark key; some keys caused more refusals, others less, highlighting key‑dependence.
• Aggregated scores sometimes masked the effect because improvements on some items cancelled out deteriorations on others.

These findings confirm that sampling drift is not merely theoretical; it manifests in real‑world safety metrics.

Comparison with Other Watermark Approaches

Traditional watermarking methods fall into two categories: post‑processing tags (e.g., adding a hidden string after generation) and generation‑time logits‑biasing. Post‑processing does not affect model behavior because the text is already produced. Logits‑biasing methods that introduce a fixed bias can degrade quality or alter semantics more predictably. SynthID‑Text aims to preserve quality by keeping the bias non‑distortive in expectation, but the study shows that even this subtle bias can ripple through downstream agent logic.

Open Questions and Security Implications

Several uncertainties remain:

• Key management: If an attacker discovers the watermark key, they could craft prompts that exploit the bias to force desired token selections.
• Interaction with prompt injection: A weakened refusal combined with tool use creates a higher‑impact attack surface, as the agent may execute harmful actions even when the model nominally rejects the request.
• Scalability: The study examined a limited set of models and temperatures; larger models or different sampling temperatures may exhibit different drift patterns.
• Detection vs. mitigation: While the watermark fulfills regulatory detection requirements, it does not mitigate the safety impact of the drift itself.

Addressing these questions is crucial for regulators and developers who rely on watermarking as a compliance shortcut.

Guidelines for Developers

Developers integrating watermarked LLMs into autonomous agents should consider the following practices:

• Run paired evaluations (watermarked vs. unwatermarked) on the exact prompts and tool‑calling pipelines used in production.
• Monitor refusal rates and tool‑calling accuracy over time; sudden shifts may indicate key‑related drift.
• Implement redundancy: verify critical arguments (e.g., amounts, file paths) with secondary checks before invoking external tools.
• Prefer models that offer configurable watermark intensity, allowing a trade‑off between detection robustness and behavioral stability.
• Stay informed about updates to the EU AI Act and related standards, as future revisions may require not only detection but also minimal impact on safety.

In summary, while invisible watermarks satisfy a growing legal demand for provenance, they also introduce a subtle yet measurable source of behavioral change in AI agents. Understanding and mitigating sampling drift will be essential to preserve both compliance and safety.

For the full technical report, see the original blog post at Lasso Security.

Asl manba: lasso.security