cs.LGcs.AI
Positional versus Symbolic Attention Heads: Learning Dynamics, RoPE Geometry, and Length Generalization
May 29, 2026
The paper analyzes the distinct computational roles of positional versus symbolic attention heads in Transformers, demonstrating that symbolic mechanisms generalize more reliably to longer sequences than positional ones.
Transformer-based language models are widespread in today's society. As such, understanding the mechanisms by which they solve structured tasks and predicting how they may behave in novel scenarios is of great importance for safe deployment. We study the learning dynamics of attention heads in a controlled setting by training a decoder-only Transformer (GPT-J) on two structurally equivalent multi-hop reasoning tasks: a number task requiring positional reasoning and a letter task requiring symbolic reasoning. Using a recently introduced metric that classifies attention-head behavior as positional or symbolic for a given prompt, we show that successful learning is associated with the emergence of pure heads, i.e., heads that express themselves as either positional or symbolic. Despite the tasks' structural equivalence, they impose different mechanistic demands: the number task requires both positional and symbolic heads, whereas the letter task requires only symbolic heads. We then identify the computational roles of these heads, characterize the basic functions they implement, and give theoretical constructions showing how single-layer RoPE-based attention can realize these functions through geometrically interpretable query, key, and value operations. This analysis yields a quantitative separation between positional and symbolic mechanisms in their robustness to longer sequences, formalized through a novel notion of discrepancy. We empirically validate the resulting predictions in both controlled and real-world models, showing that symbolic mechanisms extrapolate more reliably to longer sequences while positional mechanisms face sharper limitations.
Give it Space! Explicit Disentangling of Positional and Semantic Representations in Encoders
The paper proposes explicitly disentangling positional and semantic representati…
Periodic RoPE for Infinite Context LLMs
The paper proposes Periodic RoPE (P-RoPE) combined with a dual-layer attention m…
Beyond Sinusoids: A Morlet Wavelet Framework for Transformer Positional Encoding
The paper introduces Morlet Positional Encoding (MoPE), a novel wavelet-based po…
Benchmarking Positional Encoding Strategies for Transformer-Based EEG Foundation Models
This paper benchmarks five positional encoding strategies for transformer-based…
Revealing Algorithmic Deductive Circuits for Logical Reasoning
This paper localizes the attention heads within LLMs responsible for specific re…
Distilling Neuro-Symbolic Programs into 3D Multi-modal LLMs
The paper introduces APEIRIA, a neuro-symbolic 3D Multi-modal LLM that bridges t…
Better Later Than Sooner: Neuro-Symbolic Knowledge Graph Construction via Ontology-grounded Post-ext…
The paper proposes a neuro-symbolic framework to construct highly consistent kno…
Cultural Binding Heads in Language Models
The paper identifies specific attention heads in LLMs responsible for 'cultural…