Defending Against Harmful Supervision Hidden in Benign Samples
The paper proposes Dual-Reference SFT to mitigate harmful fine-tuning of language models by embedding harmful QA pairs in benign training samples.
Proposed Embedded Attack and Dual-Reference SFT for fine-tuning language models to mitigate harmful content
Before reading this…
Applications
- →Natural Language Processing, Machine Learning, Language Models
To understand this paper, make sure you know these concepts first:
- Understanding of language models, fine-tuning, and contrastive objective designfind papers →
Abstract
More Like ThisExisting defenses are effective when harmful content is explicitly mixed into downstream fine-tuning data, but crafted samples can instead hide harmful supervision inside benign tasks. We propose Embedded Attack, where harmful QA pairs are embedded within benign training samples, and show that representative guardrails often fail to detect them at the example level. To address this, we propose Dual-Reference SFT (DR-SFT), which adapts DPO-style contrastive objective design to SFT through token-level regularization, mitigating harmful fine-tuning beyond coarse data filtering.