How BERT-based Cross-Encoders Model Relevance in Information Retrieval?
This paper investigates the behavior of cross-encoders in information retrieval and explores their relevance prediction process, particularly in out-of-domain scenarios.
This paper extends existing knowledge on cross-encoder behavior by exploring the role of matching signals and evaluating their performance in out-of-domain scenarios.
Before reading this…
Applications
- →Improving information retrieval systems
- →Developing more efficient cross-encoder models
To understand this paper, make sure you know these concepts first:
- Understanding of information retrieval and cross-encoder modelsfind papers →
- Knowledge of BERT and its applicationsfind papers →
Abstract
More Like ThisCross-encoders are the most effective but also the least efficient models in Information Retrieval. Despite their success, the mechanisms underlying their relevance predictions remain poorly understood, particularly how they process to match queries and documents. This lack of interpretability limits their optimization, can lead to suboptimal retrieval, especially out-of-domain, and raises efficiency concerns for over-parameterized models. To address these challenges, this paper investigates two under-explored aspects of cross-encoder behavior. First, to understand the relevance prediction process, we trace the flow of information at the different stages of the process and between different input segments. We study the nature of the signals that compose this flow and that allow cross-encoders to predict the relevance of a passage to a query. Our work particularly focuses on the matching performed by the cross-encoders between queries and documents. Second, we explore how cross-encoders behave with out-of-domain documents and/or queries. Through a series of interpretability experiments, we evaluate three BERT-based cross-encoder models, varying in size and pre-training strategies, across both in-domain and out-of-domain datasets. Overall, this study both consolidates what is known about how cross-encoders model relevance and extends it by exploring the role of matching signals.