The paper introduces Loopie, two Mixture-of-Experts models that outperform vanilla Transformer baselines in looped Transformers, with extensive ablation studies and a strong reasoning pipeline.

The paper analyzes the expressivity of padded transformers, proving that their computational power is primarily determined by model depth and numeric precision, rather than attention type or width.

The paper introduces a retraining-free method to elastify pre-trained transformers into a single model that adapts to varying computational budgets.

This paper constructs a family of constant-bit-size looped transformers that interpolates between chain-of-thought and latent thought.

This paper studies how transformers exploit task-relevant structure in geometrically heterogeneous data and provides a new estimator and generalization bound for in-context learning.

The authors propose a Transformer variant that separates state prediction from next token prediction to improve language modeling performance.

This paper introduces DeVIT, an acceleration method for vision transformers using differential computation for multiplier-less matrix multiplication and quantization.

The paper introduces Morlet Positional Encoding (MoPE), a novel wavelet-based positional encoding that models position and locality simultaneously, outperforming standard sinusoidal and RoPE methods.

This paper presents NeuronFabric, a software reference architecture for future FPGA and ASIC implementations of transformer training with local Adam updates.

This paper demonstrates that transformer-based policies can provably learn complex tree search mechanisms, such as depth-first search, purely through reinforcement learning in a stochastic environment…

The paper presents an initial implementation of a quantized, integer-only transformer for jet tagging on the AMD Versal AI Engine using a reusable software framework.

This paper presents a framework for accelerating transformer-based language models on resource-constrained IoT devices.