The paper introduces Loopie, two Mixture-of-Experts models that outperform vanilla Transformer baselines in looped Transformers, with extensive ablation studies and a strong reasoning pipeline.

The paper analyzes the expressivity of padded transformers, proving that their computational power is primarily determined by model depth and numeric precision, rather than attention type or width.

This paper proposes FPRM, a Transformer-based model using fixed-point convergence as an end-to-end halting mechanism in a looped architecture to address signal propagation issues in looped architectur…

This paper proposes a new architecture for transformer-based language models called 'former', which allocates capacity non-uniformly across network depth by maintaining wider early and late layers whi…

The paper introduces Chimera, a highly efficient and scalable MCU designed for ultra-low-power edge AI inference, achieving 3.1 TOPS/W by integrating a dedicated transformer accelerator and a QoS-guar…

This paper isolates the effect of state update design in causal self-attention and introduces structural interventions to reduce approximation errors, outperforming prior post hoc baselines on long-co…

The paper proposes a hybrid ASIC+eFPGA architecture to enhance the security and resilience of edge LLM inference accelerators against both runtime and supply-chain attacks.

This paper studies the distribution of reinforcement learning (RL) adaptation across transformer layers in large language models and finds that training a single layer can recover most of the gains ob…

This paper proposes a transition-aware backend dispatch approach for efficient large language model inference on edge platforms, reducing latency, energy, and energy-delay product by an average of 17.…

The authors propose a Transformer variant that separates state prediction from next token prediction to improve language modeling performance.

The paper presents an initial implementation of a quantized, integer-only transformer for jet tagging on the AMD Versal AI Engine using a reusable software framework.

The paper introduces Morlet Positional Encoding (MoPE), a novel wavelet-based positional encoding that models position and locality simultaneously, outperforming standard sinusoidal and RoPE methods.

The paper proposes a Ferroelectric Charge-Domain Compute Cell (FCDC) using HZO memcapacitors to perform attention computation, achieving significant energy efficiency gains, especially for long-reside…

The paper demonstrates that positional encodings are not necessary for transformers to achieve universal computation, showing that the inherent mechanism of sliding context windows already provides su…