The paper proposes Randomized YaRN, a training method that improves length generalization in large language models by exposing them to out-of-distribution positional representations during training on…

The paper introduces Loong, a novel human-like agent that significantly improves long document translation by adaptively selecting and utilizing optimal historical context using a specialized memory m…

This paper introduces KGERMAR, a framework that constructs dynamic, context-specific knowledge graphs during inference for long-context language modeling, achieving lower perplexity and better memory…

LongAttnComp introduces a novel, two-stage fine-tuning framework for context compression that significantly improves long-context reasoning performance, matching or exceeding full-context accuracy on…

This paper proposes RECONTEXT, a training-free inference method for improving long-context reasoning in large language models using model-internal relevance signals and recursive evidence replay.

The paper proposes Periodic RoPE (P-RoPE) combined with a dual-layer attention mechanism to overcome the positional encoding limitations of LLMs, enabling theoretically infinite context understanding.

The paper proposes an aggressive, parameter-efficient method to prune non-essential experts from Mixture-of-Experts (MoE) LLMs, significantly compressing the model while maintaining high machine trans…

The paper introduces MIDI, a novel multilingual dataset that embeds idioms in realistic sentence and conversational contexts across diverse resource levels, revealing that idiom comprehension is signi…

The paper proposes EPIC, an efficient and parallel decoding framework that significantly speeds up the process of constraining diffusion language model outputs using Context-Free Grammars (CFG).

The paper introduces RHELM, a new benchmark designed to test LLMs' long-term memory by simulating realistic, complex, and evolving dialogues that integrate multiple heterogeneous data sources.

This paper introduces SkMTEB, a comprehensive text embedding benchmark for Slovak, and develops efficient, locally-deployable Slovak embeddings.

LongTraceRL addresses long-context reasoning challenges by generating highly challenging training data and introducing a fine-grained rubric reward, significantly improving evidence-grounded reasoning…

This paper introduces a new scaling law for sparse language models trained with limited data, demonstrating that sparsity can significantly improve performance and delay data saturation during multi-e…

Soft-NBCE introduces soft entropy-weighted chunk fusion to overcome the semantic fragmentation caused by hard chunk selection in long-context LLMs, significantly improving performance on multi-hop ben…

This paper analyzes speech-text interleaved language models and reveals that they go through an implicit transcription phase in which spoken words become decodable as text in intermediate layers.

This paper introduces BamiBERT, a new Vietnamese language model based on BERT that addresses limitations of PhoBERT and sets a new state-of-the-art among base-sized Vietnamese encoders.

This paper compares the preservation of linguistic content in different text encodings (tokens, bytes, pixels) using a shared bottleneck, revealing their distinct strengths in surface form preservatio…

The paper proposes LaSR, a context-aware training paradigm that uses latent reasoning to significantly improve speech recognition, especially for specialized terminology, without adding latency.

The paper proposes a graph-constrained approach to scale multi-hop training data by decoupling path discovery from path verbalization, significantly expanding the usable corpus size for LLMs.