20 results for “Familiarity with validation concepts”
CS papers onlyHybrid search: Keyword + semantic, ranked by combined score.ⓘ
Want pure semantic search? Try claim verification →
This paper evaluates the performance of a Large Language Model (LLM) in a high-stakes context by comparing it to human experts and measuring variance and error magnitude.
The paper evaluates the inconsistency of using LLMs as automated judges for multi-dimensional safety evaluations, finding that LLMs are unreliable for nuanced safety issues like financial advice but m…
This paper investigates the challenges of Competency Questions (CQs) in CQ-verification and proposes solutions to enhance users' performance.
This paper proposes Adaptive Matrix Validation (AMV), a method for validating AI-assisted interview data in surveys using statistical adjustment.
The paper proposes a trust schema and verification framework to ensure that agent skills, which augment LLMs, are rigorously verified before deployment, thereby making human-in-the-loop oversight scal…
This paper surveys 257 papers on validating agentic AI systems, identifying mature areas and gaps in behavioral, safety, temporal, regulatory, and multi-agent concerns.
The paper introduces an Item Response Theory (IRT)-based indicator that effectively identifies likely mislabeled items in existing LLM benchmarks, revealing systematic errors in labeling and model spe…
This paper tests the assumption that evaluation is easier than generation in LLM-as-a-Judge and self-evaluation pipelines using a controlled in-context QA setting and reveals that evaluation attends t…
The paper presents Certify, a search-augmented model that decides whether its evidence suffices to answer a question by making sufficiency structural and using a per-query requirement hypergraph.
This paper characterizes the label complexity of certifying small missed mass in an empirical pipeline and shows that auditing the excluded pool is minimax rate-optimal.
Fine-grained taxonomy of schema-conformance failures and a deterministic classifier for predicting failure categories from model responses and Pydantic validation errors
The paper demonstrates that models can acquire 'evaluation meta-knowledge' from training data describing evaluation practices, leading to inflated safety benchmark performance that is independent of e…
This paper introduces SPLIT, a method to optimize the allocation of compute resources for language models and agents to achieve accurate results.
The paper introduces the Rule Violation Score (RVS) metric to evaluate the logical consistency of predictive models.
This paper evaluates the ability of large language models to identify Schwartz's ten basic values in Russian situational texts and identifies recurring directed confusions.
The paper introduces CyberCertBench, a new benchmark suite for evaluating LLMs against industry cybersecurity certifications, finding that while frontier models perform well on general knowledge, thei…
The paper introduces FinVerBench, a comprehensive benchmark for financial statement verification, concluding that successful verification requires calibrated judgment under realistic observational con…