On Subquadratic Architectures: From Applications to Principles
This paper compares three subquadratic architectures (xLSTM, Mamba-2, and Gated DeltaNet) for sequence modeling and finds that xLSTM performs the best on complex tasks.
This paper presents a comparative study of subquadratic architectures for sequence modeling, which is a novel approach compared to traditional transformer-based models.
Before reading this…
Applications
- →code-model pre-training
- →distillation of code models from large language models
- →pre-training of time-series foundation models
To understand this paper, make sure you know these concepts first:
- sequence modelingfind papers →
- transformer modelsfind papers →
Abstract
More Like ThisTransformers dominate modern sequence modeling, but their quadratic attention incurs substantial computational cost. Subquadratic architectures offer a scalable alternative. However, it remains unclear which designs yield the most effective sequence models. We compare three leading approaches: xLSTM, Mamba-2, and Gated DeltaNet. We evaluate these models on tasks with complex dependencies: (1) code-model pre-training, (2) distillation of code models from large language models, and (3) pre-training of time-series foundation models. Across these settings, xLSTM delivers the strongest overall performance. To explain xLSTM's advantage, we present a unified formulation and analyze the underlying architectural mechanisms, focusing on state tracking and memory dynamics. Our results show that xLSTM enables more flexible and stable memory correction via its gating scheme. We corroborate these findings on controlled synthetic length-generalization tasks. Overall, our findings indicate that xLSTM's gains on complex tasks stem from robust state tracking and accumulation.