ExeBench-FHD: Addressing Fidelity Issues in Decompiler Evaluation
The paper identifies issues with the ExeBench benchmark for evaluating LLM-based decompilers and proposes a revised version called ExeBench-FHD.
The paper introduces a revised version of ExeBench with additional fields and warning flags, which is a novel contribution to the field of decompiler benchmarking.
Before reading this…
Applications
- →Improving the accuracy of decompilers
- →Benchmarking LLM-based decompilers
To understand this paper, make sure you know these concepts first:
- Knowledge of decompilers and benchmarkingfind papers →
- Understanding of LLM-based decompilersfind papers →
Abstract
More Like ThisExeBench is one of the most widely used benchmarks for evaluating LLM-based decompilers. We analyze ExeBench and identify nine issues that affect how faithfully a test measures a decompiler. Three of them prevent LLM decompilers from recovering C code from ExeBench's function. These are ambiguous parameter order when integer and floating-point types are mixed, unused parameters that optimized assembly may drop, and constants stored outside the function's assembly code. Six more affect test validity: non-void functions without a return, placeholder types, static or inline functions removed by the optimizer, functions named main, trivial bodies and truncated definitions. We present ExeBench-FHD, a backward-compatible revision of ExeBench that adds 16 new fields: call-site assembly, data sections, test wrappers, linked-binary disassembly and five warning flags. In ExeBench, 38.9% of functions have at least one of the five flags; dropping records with any of the four warning flags (all except unused_param, which marks functions that the new wrappers make evaluable) keeps 63.9% of the corpus. In an evaluation at -O2 on four architectures, the new fields raise GPT 6 Sol's re-execution rate by up to 26.5%, 19.1% and 21.0% on functions that read data sections, mix parameter types and have unused parameters, respectively.