Learn from Your Mistakes: Tree-like Self-Play for Secure Code LLMs

While Large Language Models (LLMs) excel in code generation, they remain prone to replicating subtle yet critical vulnerabilities endemic to their training data. Current alignment techniques, such as Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL), typically apply coarse-grained optimization at the sequence level. This approach often fails to address the localized nature of security flaws, where a single incorrect token choice can compromise an entire program. To bridge this gap, we introduce Tree-like Self-Play (TSP), a framework that reframes secure code generation as a fine-grained sequential decision process. Unlike standard methods that blindly maximize likelihood, TSP constructs a decision tree where the model explores branching trajectories--generating both secure "golden paths" and vulnerable variants. By treating code generation as a self-play game, the model learns to strictly discriminate against its own localized errors. This provides a dense, on-policy learning signal that forces self-correction precisely at the critical decision nodes where vulnerabilities typically emerge. Our experiments demonstrate that TSP fundamentally enhances model reliability. In Python security benchmarks, TSP boosts CodeLlama-7B's pass rate (SPR@1) to 75.8%, significantly outperforming SFT (57.0%) and unstructured self-play baselines. Crucially, TSP induces robust out-of-distribution generalization: the model not only reduces vulnerabilities in unseen categories (CWEs) by 24.5% but also successfully transfers security principles learned from C/C++ to diverse languages, including Python, Go, and JavaScript. This suggests that TSP does not merely memorize patches, but internalizes abstract, language-agnostic security logic.

• 01Low16%

  Does Teaming-Up LLMs Improve Secure Code Generation? A Comprehensive Evaluation with Multi-LLMSecCod…

  The paper evaluates multi-LLM strategies for secure code generation, finding tha…
• 02Low16%

  SecPI: Secure Code Generation with Reasoning Models via Security Reasoning Internalization

  The paper introduces SecPI, a fine-tuning pipeline that teaches reasoning langua…
• 03Low15%

  Silent Guardians: Independent and Secure Decision Tree Evaluation Without Chatter

  The paper proposes $\sf PVODTE$, a novel two-server protocol for private and ver…
• 04Low15%

  Detecting Data Poisoning in Code Generation LLMs via Black-Box, Vulnerability-Oriented Scanning

  The paper introduces CodeScan, a novel black-box framework that detects data poi…
• 05Low14%

  Learning from Mistakes: Can LLM Self-Recover after Misalignment?

  This paper shifts the focus of LLM safety from preventing misalignment to invest…
• 06Low13%

  Bridging Code Property Graphs and Language Models for Program Analysis

  The paper introduces codebadger, a Model Context Protocol (MCP) server that inte…
• 07Low13%

  Obfuscating Code Vulnerabilities against Static Analysis in JavaScript Code

  This paper empirically demonstrates that current Static Application Security Tes…
• 08Low13%

  Invisible Threats from Model Context Protocol: Generating Stealthy Injection Payload via Tree-based…

  The paper introduces Tree structured Injection for Payloads (TIP), a novel black…