Competing Learning in Hybrid Architectures
The paper proposes a staged training procedure to improve the performance of hybrid language models by suppressing the learning of recurrence layers during training.
The paper reveals a new phenomenon in the learning dynamics of hybrid language models and proposes a novel training procedure to address this issue.
Before reading this…
Applications
- →language modeling
- →common-sense reasoning
- →retrieval-related tasks
To understand this paper, make sure you know these concepts first:
- language modelsfind papers →
- hybrid architecturesfind papers →
- softmax attentionfind papers →
Abstract
More Like ThisMany modern language models have adopted hybrid architectures that incorporate both linear-time recurrence and softmax attention. Modern recurrence layers provide a highly expressive mixing mechanism while ensuring high throughput by compressing contextual information into a fixed-size state, but underperform softmax attention in contextual information retrieval. In this paper, we reveal a Competing Learning phenomenon where different kinds of mixers compete to learn the same function, and when any layer learns a function, it suppresses the corresponding learning signal for other layers. Layer attribution analysis reveals that under typical data distributions, modern recurrence learns retrieval functions faster than softmax attention, causing attention to fail to learn retrieval in most cases. Stopping weight updates in recurrence in the middle of training helps attention learn retrieval. Based on these results, we propose a staged training procedure for pre-training hybrid architectures on natural language. Empirical results on models up to 1B parameters show that our method results in substantial improvement in language modeling loss, common-sense reasoning, as well as a wide range of retrieval-related tasks across different context lengths.