SCRBench: Beyond Equivalence Checking for Semantic Change Reasoning in Code
The paper evaluates the ability of language models to reason about code changes and their semantic effects.
The paper introduces a novel dataset, SCRBench, and a new evaluation framework for language models' ability to reason about code changes.
Before reading this…
Applications
- →Improving language models' ability to reason about code changes
To understand this paper, make sure you know these concepts first:
- Understanding of language models and program semanticsfind papers →
Abstract
More Like ThisUnderstanding how code changes affect program semantics is a core aspect of code reasoning in language models. It is commonly evaluated by asking whether two programs are semantically equivalent, but a yes/no answer does not reveal whether a model knows which code change caused a semantic difference, or which sequence of code changes produced the modified program. We study two tasks that ask these questions directly, diagnosing semantic change and reconstructing sequences of code changes, alongside detecting semantic equivalence. We build SCRBench, 67,658 Python program pairs generated by applying sequences of code changes that preserve or alter program semantics, and evaluate five language models on a stratified subset of 4,500 pairs. Detecting semantic equivalence exposes weaker models, whose accuracy declines as changes accumulate, but the strongest models remain near the ceiling. Diagnosing semantic change reveals weaknesses that equivalence accuracy hides: a model with 94.71% equivalence accuracy diagnoses only 54.44% of semantic changes. Reconstructing sequences of code changes is much harder: the best model reconstructs the exact sequence for only 39.27% of pairs, and no model exceeds 1.11% when five changes are applied. High equivalence accuracy therefore overestimates how well models reason about the semantic effects of code changes.