Beyond SAEs: Consistent Concept Discovery with Sparse Coding
The paper proposes Consistent Sparse Dictionary Learning (ConSDL) to improve the reliability and reproducibility of sparse autoencoders.
The paper proposes a novel approach to dictionary learning that uses per-sample sparse coding to improve the reliability and reproducibility of sparse autoencoders.
Before reading this…
Applications
- →vision
- →language
To understand this paper, make sure you know these concepts first:
- sparse autoencodersfind papers →
- dictionary learningfind papers →
Abstract
More Like ThisSparse autoencoders (SAEs) explain foundation model representations by decomposing activations into sparse combinations of learned dictionary atoms. However, the learned dictionaries can vary across training runs, limiting the reliability and reproducibility of SAE-based analyses. We propose Consistent Sparse Dictionary Learning (ConSDL), which replaces the standard linear encoder with per-sample sparse coding. This is motivated by our synthetic analysis showing that coefficients from a single linear encoder pass are suboptimal compared with those obtained by per-sample optimization, suggesting that sparse coding is a reliable mechanism for dictionary learning. Across 16 synthetic settings, ConSDL achieves near-perfect dictionary and coefficient recovery. These gains translate to state-of-the-art cross-seed consistency across four vision and language settings, where ConSDL improves dictionary and coefficient consistency by 17.6% and 35.9%, respectively, over the average of six SAE baselines.