~ similar to 2608.13867· 20 results
This paper revisits N-version programming with AI coding agents and finds substantial common-mode failures but also practical benefits.
The paper presents a benchmark for evaluating the performance of large language models as embedded coding agents for firmware development.
The authors argue that coding agents can replace traditional human code review in software development.
This paper examines the instruction stacks of five coding agents to understand their composition and evolution.
Yangshen Deng, Jiaqi Ruan, Yeqi Huang, Nick Lau +2 more
The paper introduces InfraDev, a benchmark for evaluating AI infrastructure development, and shows that existing benchmarks are inadequate for assessing real-world tasks.
The paper argues that current 'on-the-fly' AI agent design lacks necessary software engineering rigor and proposes an 'AI Workflow Store' to provide hardened, reusable, and reliable agent workflows.
James C. Davis, Paschal C. Amusuo, Tanmay Singla, Berk Çakar +1 more
This paper presents a case study on using AI coding agents in software engineering and develops a process model for governing high-velocity agentic implementation.
One clear sentence describing what this paper does
Qi Hu, Yifeng Tang, Qinghua Wang, Lanyang Zhao +6 more
The paper introduces SABER, a new benchmark that evaluates the operational safety of LLM coding agents in complex, stateful project environments, finding that current models have a high rate of harmfu…
The paper introduces a data-centric optimization pipeline to improve coding agents' ability to interact with a branching lakehouse, showing significant accuracy gains by treating agent evaluation as a…
This paper characterizes oracle signals in test files of agent-authored pull requests and assesses their impact on merge outcomes.
Coding agents are not just models, but complex systems with multiple components that can have significant impact on benchmark scores
Xinyu Che, Yunfei Ge, Shihao Li, Yanchen Liu +8 more
This paper introduces GameLogicBench, a benchmark for evaluating the ability of coding agents to implement gameplay rules in game development.
Zhe Ye, Hantao Lou, Yuechun Sun, Peiyang Song +7 more
The paper introduces Vero, the first benchmark for evaluating joint implementation and proof synthesis at the repository level for AI-generated software.
Shiping Chen, Qin Wang, Guangsheng Yu, Xu Wang +1 more
This paper systematizes the security challenges of open agentic systems, concluding that while attack characterization is mature, the field lacks robust guidelines for operational governance, memory i…
DeepSWE is a new benchmark for evaluating coding agents with original tasks written from scratch and hand-graded verifiers.