CDAE: Enhancing Perturbation Robustness in Pretrained Language Models with Contrastive Denoising
This paper proposes a Contrastive Denoising Autoencoder (CDAE) to refine pre-trained BERT embeddings and make them more robust to semantic preserving textual perturbations.
Proposed a new method called CDAE to optimize contrastive and reconstruction objectives for perturbation-invariant sentence representations.
Before reading this…
Applications
- →Natural language processing, information retrieval, text classification
To understand this paper, make sure you know these concepts first:
- Understanding of pre-trained language models, contrastive learning, denoising autoencodersfind papers →
Abstract
More Like ThisPre-trained language models have significantly improved sentence representation learning, yet their embedding remain sensitive to semantic preserving textual perturbations such as synonym substitution, masking and word dropout. This work proposes a lightweight Contrastive Denoising Autoencoder (CDAE) that refines pre-trained BERT embedding by jointly optimizing contrastive and reconstruction objective to learn perturbation-invariant representation. We evaluate the proposed framework using multiple perturbation strategies with varying strengths and compare it against the original BERT embeddings and SimCSE. Experimental results show that CDAE consistently preserves higher embedding similarity under perturbations, with the improvements becoming more pronounced as framework effectively enhances representation stability while preserving semantic information, highlighting perturbation-invariant learning as a promising direction for improving sentence embeddings. The source code is publicly available at: https://github.com/ComputationIASBS/CDAE