VMSpec: A Decompiler-based Approach against Virtual Machine Obfuscation
This paper proposes a novel decompiler-based approach called VMSpec to automate the construction of VM processor specifications, enabling existing decompilers to analyze VM-obfuscated code.
This paper is the first to propose a decompiler-based approach to automating VM processor specification construction.
Before reading this…
Applications
- →Reverse engineering
- →Malware analysis
To understand this paper, make sure you know these concepts first:
- Understanding of decompilers and VMsfind papers →
- Knowledge of programming languagesfind papers →
Abstract
More Like ThisMalicious actors frequently obfuscate their code to hinder investigative reverse engineering efforts. Among the most formidable techniques is virtual machine (VM) obfuscation: programs are translated into bytecode for a custom VM, often with a randomized virtual instruction set and distributed alongside a corresponding interpreter. Manual analysis and deobfuscation scale poorly due to the complexity of VM obfuscation and the diversification techniques that render each VM and interpreter unique. Existing methods primarily focus on simplifying execution traces and identifying virtual execution boundaries. However, comparatively little attention has been given to recovering higher-level program semantics from VM-obfuscated code. To address this gap, we propose VMSpec, a novel decompiler-based approach that automates the construction of VM processor specifications, enabling existing, well-known decompilers to analyze the VM-obfuscated code. We evaluate our prototype of VMSpec on 147 programs from an established obfuscation benchmark, comprising hash functions, standard algorithms, and randomized functions, obfuscated with different configurations of the Tigress and Code Virtualizer obfuscators. To demonstrate real-world applicability, we additionally include six domain generation algorithms stemming from real-world malware. VMSpec completely strips VM-obfuscation from nearly all obfuscated samples, producing output comparable to Ghidra's decompilation of the original code. Furthermore, it is the first method capable of reconstructing the complete program behavior of VM-obfuscated programs with infinite, input-dependent paths and recursive calls.