This paper proposes G2Rec, a scalable framework for industrial-scale generative recommendation that unifies graph-based user co-engagement modeling and semantic tokenization.

The paper proposes a robust, multi-stage pipeline combining rule-based classification and machine learning to map noisy retail product names to standardized consumption categories, finding that simple…

The paper introduces an efficient, novel algorithm for incremental Byte Pair Encoding (BPE) tokenization that processes input text prefix by prefix, achieving significant speedups and enabling streami…

This paper introduces Byte-Prefix Marginalization (BPM), a method for consolidating open-weight language models through on-policy distillation while preserving teacher probability mass and maintaining…

This study benchmarks token-optimized formats (TOON and TRON) against JSON in end-to-end agentic AI systems, finding that TRON significantly reduces token overhead with minimal performance degradation…

This study systematically evaluates a wide range of chunking methods for Retrieval-Augmented Generation (RAG) to assess their effectiveness and highlight the overlooked challenges associated with chun…

The paper proposes lightweight fine-tuning as an alternative to pooling-aware training for reducing vector count in late interaction models, achieving broad gains and high compression rates.

The paper introduces CFGzip, an offline token space compression technique that significantly reduces the computational overhead of constrained decoding, making complex grammar enforcement feasible at…

HoliTok introduces a novel continuous holistic tokenization model that provides a unified, high-fidelity latent representation for simultaneously supporting both speech generation and speech understan…

This paper proposes CompRank, a token-efficient reranking framework for large language models that reduces redundant computation and achieves strong reranking performance.

The paper introduces Latent Terms, a method that shows dense retrieval models implicitly learn sparse, Zipfian vocabularies that can be used for classical BM25-style sparse scoring without requiring s…

This paper introduces SVD-RAG, a cost-efficient and content-adaptive summarization method for hierarchical Retrieval-Augmented Generation systems using Singular Value Decomposition on dense sentence e…

The paper proposes a low-cost and interpretable fine-tuning extraction strategy for automatic term extraction, demonstrating consistent and balanced performance on the ATE Shared Task.

This paper evaluates the effectiveness of cluster-based semantic chunking compared to fixed-size and recursive chunking in Retrieval-Augmented Generation systems using the Retrieval Augmented Generati…

The paper introduces grapheme-kit, an open-source Python library that extends lexical metrics to operate on grapheme clusters instead of Unicode code points, and provides improved grapheme processing…