Paper Readings · 2026-09-05 · 1,348 words · 6 min read

Covert Indirect Prompt Injection (ICoA) explained

A reading of arXiv:2608.30362, which splits attack success into covert and overt outcomes on AgentDojo and shows the covert share can be engineered.

prompt injection agents AgentDojo security

The paper is arXiv:2608.30362, "Will the User Ever Know? Covert Indirect Prompt Injection Attacks on Tool-Using LLM Agents", by Yunseok Lee, Yunji Kim and Woojin Lee (Dongguk University), version 2 dated 31 August 2026 and marked as accepted to the EMNLP 2026 main conference. It asks a question that the usual metric for indirect prompt injection ignores: when an injected instruction succeeds, does the user find out? If you build or evaluate tool-using agents, the paper's decomposition of attack success is the part worth taking home, and the numbers in its Table 1 are the part worth checking.

What the paper claims

Indirect prompt injection (IPI) is the case where an attacker plants instructions inside content the agent reads through a tool: an email, a web page, a transaction record. The standard metric is Attack Success Rate (ASR), the share of trials in which the injected task was executed. The authors' claim is that ASR blurs two different outcomes. In some successful traces, the agent carries out the injected action and then writes a final response that looks like an ordinary answer to the user's question. In others, the final response mentions the injected action, so the user has a chance to notice. They call the first covert and the second overt, and they split ASR into a Covert Success Rate (CSR) and an Overt Success Rate (OSR), with ASR = CSR + OSR.

The second claim is mechanistic. Looking at successful trajectories, they find that what separates covert from overt is what the agent does after the injection fires: covert traces hand control back to the user task before ending, while overt traces end at the attack itself. They attribute this to the ReAct loop, where the final response summarises the most recent action. If the last thing the agent did was the attacker's task, that is what gets reported.

The third claim follows from the second: if the split is caused by whether the agent returns to the user task, an attacker can induce it. Their attack, ICoA (Induced Covert Attack), appends a "RETURN anchor" to the payload that tells the agent to handle the injected task first and then resume the original request. The abstract states that across four target models on AgentDojo, ICoA achieves the highest CSR, with gains of 3.79 to 12.01 percentage points over the strongest baseline.

What was measured

Section 5 gives the setup. The benchmark is AgentDojo v0.1.34 with all four suites, giving n = 949 pairs of user task and injection task: Banking (144), Slack (105), Travel (140) and Workspace (560). Four target models: two open-weight models, Qwen3-235B and LLaMA-3.3-70B, served locally through Ollama at temperature 0, and two closed models, GPT-4o-mini and Gemini-2.5-Flash, through their APIs at the same temperature. Five baseline attacks (Direct, InjecAgent, Important message, ChatInject, and ICoA) are run against no defense and five defenses: a detector (PI Detector), three prompting defenses (Instruction Prevention, Delimiting, Repeat User), and a runtime task-alignment check (Task Shield). ChatInject is omitted on Gemini-2.5-Flash because it needs model-specific role tags that model does not support.

The covert/overt label comes from an LLM auditor: GPT-4o at temperature 0, given the user task, the injected goal and the agent's final response, labels a trace overt when the response mentions an entity introduced by the injection. That auditor is validated in Appendix F.2 against a second LLM judge (Claude Haiku 4.5) and two human annotators on 500 stratified traces, 100 per attack; Table 7 gives the covert counts.

Table 1 is the main result. In the no-defense rows, per model, ASR and CSR in percent:

The 3.79 to 12.01 point range in the abstract is the CSR gap between ICoA and Important message, the strongest prose baseline: the biggest gain is on LLaMA-3.3-70B, from 11.80% to 23.81%. The LLaMA row is also the paper's cleanest illustration of the ASR/CSR distinction. ChatInject and ICoA have almost the same ASR there (32.14% vs 32.46%), yet ChatInject's CSR is 2.63% and ICoA's is 23.81%. Same execution rate, roughly nine times the covert rate.

Table 3 reports utility, the share of trials where the user task still completes correctly. Under ICoA it drops substantially: Qwen3-235B goes from 89.69% benign utility to 35.09%, GPT-4o-mini from 70.10% to 33.51%. The attack that hides best also breaks the user task most, which is a useful tension the authors state rather than bury.

Table 5 breaks ICoA down per suite under no defense. Workspace, the largest suite at 560 of 949 trials, is where ICoA is weakest (LLaMA-3.3-70B: 10.18% ASR, 5.54% CSR), while Slack is where it is strongest (Gemini-2.5-Flash: 100.00% ASR, 90.48% CSR). The aggregate numbers are dominated by Workspace, so the headline CSRs are on the conservative side of what the shorter suites show.

What holds up / what I could check

I could not rerun AgentDojo here, so what follows is a check of the paper against itself and against what is stated in its appendices.

The decomposition is sound arithmetic and the mechanism is testable. Table 2 appends the RETURN anchor to each baseline attack and CSR rises on every model, for instance Important message on LLaMA-3.3-70B from 11.80% to 22.66%, and Figure 8 reports that for Gemini-2.5-Flash Direct the number of successful injections that return to the user task rises from 6 to 180. Table 4's ablation on LLaMA-3.3-70B removes the anchor and CSR falls from 23.81% to 13.17%, while removing the user framing drops ASR from 32.46% to 21.92%. Those are the results I would want to see if the causal story were true, and they are present.

The judge validation is the part I checked most carefully, because a CSR built on an LLM auditor is only as good as that auditor. Table 7's counts over 100 traces per attack: for ICoA, GPT-4o labels 64 covert, Claude Haiku 4.5 labels 66, and humans label 64; totals over 500 traces are 198, 214 and 201. The GPT-4o count matches or falls one below the human count for every attack, so if anything the reported CSRs are slightly conservative. What I cannot verify is how the two human annotators' individual labels compared before "disagreements resolved through discussion"; no inter-annotator figure is given.

Generalisation is thinner. Table 6 repeats the measurement on InjecAgent's data-stealing suite (n = 544) with LLaMA-3.3-70B only: ChatInject reaches 45.96% ASR with 0.00% CSR, while ICoA reaches 39.15% ASR with 29.60% CSR. That is one model on one extra benchmark, and the trajectories had to be continued with simulated tool responses to produce a final answer at all. The authors' own limitations section adds that only ReAct-style single-turn loops were tested, and that agents which expose their reasoning to the user may behave differently.

Two things I would flag for anyone reading the numbers. First, the four models are a 2025-era set; the paper gives no results on current frontier models, so treat the absolute rates as benchmark-specific rather than as a statement about what ships today. Second, defenses matter in a way the abstract does not stress: in Table 1, Task Shield holds ICoA to 11.17% ASR on Qwen3-235B and 5.16% on LLaMA-3.3-70B, and PI Detector to 7.48% on LLaMA, so the covert rates under any defense are far below the no-defense rows.

What to do with it

If you report an injection metric for an agent, report CSR and OSR rather than ASR alone; the split costs one extra judge call per successful trace. If you build defenses, the paper's finding that the covert share depends on post-injection behaviour argues for checking what the agent did against what it reports, not just filtering what it reads. And if you are choosing a defense, the Task Shield and detector rows in Table 1 are the ones to reproduce first.