Back to Paper
cs.CRcs.AI
Local ID: 2605.16626v2
AI Summary: gemma4:e4b
SLEIGHT-Bench: A Benchmark of Evasion Attacks Against Agent Monitors
By Elle Najt, Colin Toft, Tyler Tracy, Fabien Roger, Joe Benton
Revision History Timeline
v15/15/2026
5/15/2026
No submitter comment provided.
v25/19/2026
5/19/2026
No submitter comment provided.
★ Version indexed in ExplorerComparing v1 vs v2
Green = Added • Red = Removed
Title Comparison
SLEIGHT-Bench: A Benchmark of Evasion Attacks Against Agent Monitors
Authors Comparison
No author changes.
v1 Comment
No comment for this version.
v2 Comment
No comment for this version.
Abstract Word Diff
Since autonomous coding agents generate complex behaviors at high-volume, we may want to use other LLMs to monitor actions to reduce the risk from dangerous misaligned behavior. To better understand the limitations of such monitors against the diverse attack strategies that a coding agent could use, we present SLEIGHT-Bench (Subtle Low-itEration Insight-Guided Harmful Transcripts), a benchmark of synthetic transcripts containing 40 attacks across 11 categories, each showing a coding agent covertly pursuing a harmful objective (e.g. weight exfiltration, credential theft, unmonitored rogue deployment). We find that 2420 of 40 attacks are never caught by an Opus 4.6 monitor with extended thinking across 10 trials at a 1% false-positive rate, while the overall catch rate is 23%.32%. We identify several evasion strategies such as exploiting model priors, user instruction ambiguity, and state manipulation, which facilitate creating attacks that evade frontier monitors. We also elicit stronger monitor performance using coding agents as monitors versus regular prompted monitors, and for some evasion strategies show improved catch rates with targeted monitor prompts. Our dataset and evaluation framework are available at https://github.com/safety-research/sleight-bench and https://huggingface.co/datasets/sleightbench/SLEIGHT-Bench.