cs.SEcs.AIEmpirical
MM-IssueLoc: A Controlled Benchmark for Evaluating Visual Evidence in Multimodal Repository-Level Issue Localization
Jul 16, 2026
This paper introduces MM-IssueLoc, a benchmark and evaluation protocol for repository-level issue localization with visual evidence.
The authors created a new benchmark for multimodal repository-level issue localization, which is different from existing text-only benchmarks.
Before reading this…
Applications
To understand this paper, make sure you know these concepts first:
Real repository issues routinely include visual evidence such as screenshots, error dialogs, rendered UI states, and logs, yet repository-level issue localization is evaluated mostly as a text-only task. Existing multimodal SE benchmarks evaluate end-to-end repair, entangling localization with patch synthesis and obscuring whether visual input helped, hurt, or was ignored. We introduce \textbf{MM-IssueLoc}, a controlled benchmark and evaluation protocol for repository-level localization with visual evidence. MM-IssueLoc contains 652 issue-PR instances across 23 languages, with annotations for 7 image categories and 4 relevance levels. It provides file-level and function-level gold labels, paired text-only and with-image evaluation, and VCE-based diagnostics that convert images into structured textual evidence. We evaluate LLM-based and retrieval-based systems, including MM-IssueLoc-VL-Emb as a controlled multimodal retriever. Results show that existing systems remain far from reliable multimodal repository localization: the strongest agent reaches 38.96 file Acc@5 and 22.45 function Acc@10, while the strongest retriever reaches 33.86 function Acc@10. Cross-benchmark comparisons show that high localization scores on text-dominant SWE benchmarks do not transfer cleanly to multimodal issue localization. MM-IssueLoc turns visual evidence into an explicit evaluation variable, enabling future work to test whether systems improve by using visual evidence for localization, rather than by relying on text-only cues or downstream patch-generation effects.
HoloGeo: Mitigating Landmark Bias in Geo-localization via Evidence-Driven Reasoning
This paper introduces HoloGeo, an evidence-driven reasoning framework to mitigat…
ProjAgent: Procedural Similarity Retrieval for Repository-Level Code Generation
Proposed ProjAgent, a repository-level code generation system using procedural s…
Vision as Unified Multimodal Generation
The paper presents SenseNova-Vision, a unified multimodal model for computer vis…
Scalable Visual Pretraining for Language Intelligence
This paper presents the benefits of visual pretraining for foundation model inte…
Who Broke the System? Failure Localization in LLM-Based Multi-Agent Systems
This paper presents AgentLocate, a framework for failure localization in large l…
Perceive-to-Reason: Decoupling Perception and Reasoning for Fine-Grained Visual Reasoning
This paper proposes Perceive-to-Reason (P2R), a framework for fine-grained visua…
Antaeus: Hunting Repository-Level Logic Vulnerabilities via Context-Grounded LLM Reasoning
Antaeus is a framework that uses repository-level code context to detect logic v…
DynaKRAG: A Unified Framework for Learnable Evidence Control in Multi-Hop Retrieval-Augmented Genera…
This paper introduces DynaKRAG, a method for multi-hop retrieval-augmented gener…