Improving Long-Context Retrieval with Multi-Prefix Embedding
The paper proposes Multi-Prefix Embedding (MPE), a method that encodes documents into single embeddings while retaining cross-chunk context.
The paper presents a novel approach to document encoding that balances the trade-off between fine-grained detail and storage requirements.
Before reading this…
Applications
- →Information retrieval
- →Document similarity measurement
To understand this paper, make sure you know these concepts first:
- Document encodingfind papers →
- Information retrievalfind papers →
Abstract
More Like ThisLong-context retrieval exposes a tension: single-vector embeddings lose fine-grained detail, while token-level multi-vector methods incur prohibitive storage. We propose Multi-Prefix Embedding (MPE), which partitions a document into chunks separated by EOS tokens, encodes the full sequence in a single causal forward pass, and extracts one embedding at each prefix boundary. MPE retains cross-chunk context, enables chunk-level MaxSim matching, and trains with only document-level relevance labels. Experiments on MLDR-en, BrowseComp-Plus, and LongEmbed show that MPE is competitive with or outperforms single-vector, independent-chunk, and multi-vector baselines, while providing a natural source attribution mechanism for locating evidence chunks.