Understanding and Debugging Failures in N-Gram-Based Generative Retrieval
This paper explores failure modes in Generative Retrieval (GR) systems and introduces a new tool to analyze generated ngrams.
This paper introduces a new taxonomy of GR failure modes and a new tool for analyzing generated ngrams, which is novel compared to prior work.
Before reading this…
Applications
- →Information Retrieval
- →Language Models
To understand this paper, make sure you know these concepts first:
- Generative Retrievalfind papers →
- Information Retrievalfind papers →
Abstract
More Like ThisGenerative Retrieval (GR) is an emerging Information Retrieval (IR) paradigm that is motivated by increasingly capable language models. In GR, a model directly generates identifiers for relevant documents. While these systems offer unique advantages, they also introduce distinct failure mechanisms. We explore these failure modes in three contributions: (1) We present a taxonomy of GR failure modes based on GR literature. (2) We empirically investigate failure in a subset of GR: ngram-based methods, more specifically, SEAL and MINDER. Our analysis reveals common issues, such as ambiguous docids, low identifier diversity, and the disproportionate impact of specific identifiers. (3) We introduce a new web-based tool that helps the IR community analyze generated ngrams and their respective contribution to the final ranking, providing an intuitive interface to identify where such GR methods go wrong.