Beyond a Bag of Features: Set-Level Instability in Sparse Autoencoders
The authors re-evaluate a previous study on LLM representations using a different similarity measure and find that SAE activation sets do not recover human category boundaries or typicality as effectively as dense embeddings or residual-stream states.
The authors propose a new way to measure similarity between sets of activations using overlap over SAE latent sets and find that it does not recover human category boundaries as effectively as other methods.
Before reading this…
Applications
- →Natural language processing, machine learning, artificial intelligence
To understand this paper, make sure you know these concepts first:
- Understanding of language model representations, cosine similarity, sparse autoencodersfind papers →
Abstract
More Like ThisShani et al. (2026) show that LLM representations broadly recover human category boundaries, while failing to reflect fine-grained typicality structure. Their analysis uses cosine similarity over dense model representations. We revisit their approach using overlap over active sparse autoencoder (SAE) latent sets as a more interpretable similarity measure. We first verify that this set-level measure is meaningful: SAE latent sets can recover union-like compositional structure in controlled toy models and induce semantically coherent neighborhoods in natural text. Extending the human-concepts analysis to SAE set similarities, we find that SAE activation sets do not recover human category boundaries or within-category typicality more faithfully than dense embeddings or residual-stream states, but instead track model-internal similarity structure. To probe this gap further, we study active latent sets under well-controlled semantic modifications, revealing a substantial mismatch between human judgements of conceptual change and change in the SAE active set. We interpret this as evidence that, outside idealised settings, SAE features do not compose via simple bag-of-features semantics.