This paper proposes a preconditioning layer for stable weight conditioning in LLM training.

The paper proposes PALS, a method for adjusting per-layer sparsity based on activation magnitudes in transformer models, achieving better performance than uniform one-shot pruning methods.

This paper compares three neuromorphic core designs for handling weight sparsity in event-driven neural networks and quantifies the performance and energy costs.

The paper introduces SLORR, a simple and stateless framework for in-training low-rank regularization of neural networks using two variants based on the Hoyer sparsity metric and the nuclear norm.

This paper proves that for any finite abelian group, there exists a spectral sparsifier for its Cayley graph with log(|G|) generators. This result improves upon previous work for constructing code spa…

This paper introduces a structural theorem for the sparsifiability of real-valued codes, which generalizes both combinatorial and continuous notions of sparsification.

A new solver-free parallel spectral sparsification algorithm for weighted graphs is presented, relying on low-diameter decompositions and independent sampling, eliminating dependence on target approxi…

This paper introduces a new scaling law for sparse language models trained with limited data, demonstrating that sparsity can significantly improve performance and delay data saturation during multi-e…

The paper refutes Steurer's conjecture regarding the existence of large constant-separated sets within families of unit-norm vectors with low average correlation, using high-dimensional expanders to s…

The paper theoretically analyzes the properties that optimal sparse autoencoder (SAE) dictionaries must satisfy, deriving constraints that explain observed SAE behaviors like hierarchical splitting an…

The paper introduces Single-stage Sparse Retrieval (SSR), a method that replaces computationally expensive vector clustering with sparse autoencoding to achieve highly efficient multi-vector retrieval…

The paper proposes FOAM, an adaptive damping method that stabilizes the Shampoo optimization algorithm by dynamically controlling damping and eigendecomposition frequency, thereby reducing staleness-i…

The paper systematically characterizes column-level activation sparsity across various diffusion model architectures, demonstrating that element-level sparsity metrics significantly overestimate the a…