How Fine-Grained Should a RAG Benchmark Be? A Hierarchical Framework for Synthetic Question Generation
This paper presents a hierarchical framework for studying granularity in retrieval-augmented generation benchmark construction.
This paper introduces a new framework and metric for evaluating the granularity of RAG benchmarks, which is novel compared to prior work.
Before reading this…
Applications
- →Improving the accuracy of retrieval-augmented generation systems
To understand this paper, make sure you know these concepts first:
- Knowledge of retrieval-augmented generation and benchmark constructionfind papers →
Abstract
More Like ThisEvaluating retrieval-augmented generation (RAG) systems requires benchmarks that capture diverse question characteristics, yet practitioners lack empirical guidance on which dimensions to vary and at what granularity. We present HieraRAG, a hierarchical framework for studying granularity in RAG benchmark construction, defining optimal granularity as the level that maximizes discriminative power (the standard deviation of generation quality across categories) within a given RAG configuration. As a case study, we generate 5,872 synthetic question-answer (QA) pairs from FineWeb-10BT across 3 dimensions (Question Complexity, Answer Type, Linguistic Variation) at 3 granularity levels (2, 4, and 8 categories). With a BM25+Falcon-3-10B pipeline, optimal granularity varies by dimension: complexity benefits from fine-grained distinctions (discriminative power: 0.053) while answer type and linguistic variation peak at medium granularity. We introduce a Coherence Ratio metric to quantify whether fine-grained splits cleanly subdivide parent categories, revealing structural differences across dimensions (Question Complexity: 0.40 vs. Answer Type: 1.44). Human evaluation of 110 stratified QA pairs confirms synthetic quality. While these specific findings reflect a single configuration, HieraRAG provides a portable procedure and validation metric for practitioners to determine evaluation granularity within their own RAG settings.