ASK-NN: An Asymmetric Nearest-Neighbor Test that detects Distribution Drifts in Natural Language
The paper introduces ASK-NN, an asymmetric two-sample test for detecting hallucinations in LLM-generated outputs based on the directed k-nearest-neighbor graph.
Proposed an asymmetric two-sample test, ASK-NN, to detect LLM hallucinations based on the directed k-nearest-neighbor graph.
Before reading this…
Applications
- →Detecting hallucinations in large language model outputs.
To understand this paper, make sure you know these concepts first:
- Understanding of large language models (LLMs)find papers →
- Knowledge of two-sample tests and k-nearest-neighbor graphsfind papers →
Abstract
More Like ThisHallucinations and artificial text in LLM-generated outputs often appear as distributional deviations between prompt and response hidden-state distributions. Since prompts or retrieved contexts typically serve as reference samples and responses as query samples, with major differences in length, these asymmetries motivate the use of change test statistics that treat the two samples differently. We consider an asymmetric two-sample test ASK-NN based on the directed k-nearest-neighbor graph. Our statistic counts reference points whose nearest neighbor in the pooled sample is also a reference point. Under the permutation null, it admits an exact finite-sample conditional mean and variance; we further establish asymptotic normality and consistency under fixed alternatives. ASK-NN is computationally effective and easy to implement. Empirically, it is competitive with kernel and graph-based baselines on synthetic benchmarks, artificial-text detection, and LLM hallucination detection from token-level hidden states.