The paper introduces Loopie, two Mixture-of-Experts models that outperform vanilla Transformer baselines in looped Transformers, with extensive ablation studies and a strong reasoning pipeline.

The paper analyzes the expressivity of padded transformers, proving that their computational power is primarily determined by model depth and numeric precision, rather than attention type or width.

The paper demonstrates that positional encodings are not necessary for transformers to achieve universal computation, showing that the inherent mechanism of sliding context windows already provides su…

This paper isolates the effect of state update design in causal self-attention and introduces structural interventions to reduce approximation errors, outperforming prior post hoc baselines on long-co…

The paper introduces Morlet Positional Encoding (MoPE), a novel wavelet-based positional encoding that models position and locality simultaneously, outperforming standard sinusoidal and RoPE methods.

This paper proposes a new architecture for transformer-based language models called 'former', which allocates capacity non-uniformly across network depth by maintaining wider early and late layers whi…

This book provides a compact, derivation-oriented mathematical primer that connects major families of generative AI models, showing their underlying structural relationships.

The authors propose a Transformer variant that separates state prediction from next token prediction to improve language modeling performance.

The paper presents an initial implementation of a quantized, integer-only transformer for jet tagging on the AMD Versal AI Engine using a reusable software framework.

The paper introduces Transformers with Temporal Middle-Layer Recurrence (T2MLR), a transformers-based latent reasoning architecture that enables abstract intermediate computation to persist across dec…

This paper studies the distribution of reinforcement learning (RL) adaptation across transformer layers in large language models and finds that training a single layer can recover most of the gains ob…

This paper explains how Rotary Position Embeddings (RoPE) frequencies in transformer models correspond to the relative-distance structure of training data, and the implications for long-context genera…

Introduces Posterior Prefix Tuning (PPT) for eliciting high-utility continuations from Bayes-filtered transformers using a latent posterior model.

This paper proposes FPRM, a Transformer-based model using fixed-point convergence as an end-to-end halting mechanism in a looped architecture to address signal propagation issues in looped architectur…