Back to Paper
cs.CRcs.AIcs.CLcs.ET
Local ID: 2606.02240v2
AI Summary: gemma4:e4b
AgentRedBench: Dynamic Redteaming and Integration-Aware Defense for LLM Agents over SaaS Integrations
By Hiskias Dingeto, William Leeney
Revision History Timeline
v16/1/2026
6/1/2026
No submitter comment provided.
v26/2/2026
6/2/2026
No submitter comment provided.
★ Version indexed in Explorerv37/17/2026
7/17/2026
No submitter comment provided.
Comparing v2 vs v3
Green = Added • Red = Removed
Title Comparison
AgentRedBench: Dynamic Redteaming and Integration-Aware Defense for LLM Agents over SaaS Integrations
Authors Comparison
No author changes.
v2 Comment
No comment for this version.
v3 Comment
No comment for this version.
Abstract Word Diff
Indirect prompt injection in tool-use agents is a concrete production threat: LLM agents read from integrations (third-party services such as Gmail, Salesforce, or Jira accessed through tool calls) whose response content the user neither writes nor controls. Existing benchmarks under-measure the threat: most cover only a handful of integrations with the same attack payload replayed across runs, and open-source guards are trained on chat-style data rather than tool-response content. We introduce AGENTREDBENCH, a dynamic LLM-driven redteaming benchmark of 215 subtle underspecified authorization (attacks at the boundary of what the user's request authorises)underspecified-authorization scenarios across 24 enterprise integrations in nine functional families and five attack types. Across an eight-model panel (Anthropic, OpenAI, Google), no-guard ASR (attackattack success rate)rate ranges from 32% (Claude Sonnet 4.6) to 81% (Gemini 3 Flash).81%. To keep the scenario set out of training corpora and preserve headline ASR meaning over time, we release the codebase, integration schemas, and AGENTREDGUARD model openly; the canonical scenarios are evaluated through a maintainer-mediated channel with immutable versioning. We release AGENTREDGUARD alongside the benchmark:cuts aonline guardattack trainedsuccess onby an75-77pp integration-diverseacross corpusthree oftarget adversarialmodel tool-responsefamilies content.(Haiku, AGENTREDGUARDGPT-5.4-mini, cutsGemini-3-flash) panelat ASR0.0% fromreal-benign 69.9%false-positive torate 2.4%(0.2% aton 0.37%a false-positivesynthetic-benign rate,corpus), outperforming every open-source baseline with non-trivial detection (Llama Guard, PromptGuard 2, ProtectAI) on both axes. Cross-integration and cross-attack typecross-attacker holdouts both(two independent attacker families held out from training) confirm the gain transfers beyond the training subset.