Compact Documentation for Coding Agents: A Benchmark, an Optimizer, and Why It Does Not Transfer
This paper investigates the effectiveness of natural-language documentation in helping coding agents resolve software issues.
The paper presents a novel benchmark and optimization approach to evaluate and generate accurate code descriptions.
Before reading this…
Applications
- →Improving software development tools and processes
To understand this paper, make sure you know these concepts first:
- Understanding of software development and coding agentsfind papers →
Abstract
More Like ThisWe investigate whether natural-language documentation helps coding agents resolve software issues, and we build the tools to construct and evaluate it. We introduce a roundtrip benchmark that scores code descriptions by whether code regenerated from them passes the original tests, and show that completeness, not length, drives a description's fidelity. Using the benchmark as an optimization signal, we discover a description-writing prompt that reaches full fidelity and generalizes to unseen files. We then test the hypothesis that motivated the work: that better documentation helps an agent resolve real repository issues. Across two model families and ten repositories, and against a positive control confirming that our evaluation can detect a genuine improvement, we find that it does not. When the source is present, neither static compact documentation nor retrieved context beats the issue alone. We report this negative result together with the benchmark and the optimizer, and we characterize the boundary at which documentation helps.