ArXivCSExplorer
☆☆Bookmarks🏆RSSHow to UseFAQ
Built with and by Teycir Ben Soltane•
How to Use•FAQ•GitHub•arXiv.org•
Share:

20 results for “Familiarity with validation concepts”

CS papers only

Hybrid search: Keyword + semantic, ranked by combined score.ⓘ

Want pure semantic search? Try claim verification →

stat.OTcs.AIEmpiricalRecentJun 9, 2026

Flaws in the LLM Automation Narrative

George Perrett, Javae Elliott, Jennifer Hill, Marc Scott

This paper evaluates the performance of a Large Language Model (LLM) in a high-stakes context by comparing it to human experts and measuring variance and error magnitude.

View →
cs.CLRecentMay 29, 2026

LLM Judges Inconsistently Disagree Across Safety Criteria and Harm Categories

Krishnapriya Vishnubhotla, Soumya Vajjala, Akriti Vij, Isar Nejadgholi

The paper evaluates the inconsistency of using LLMs as automated judges for multi-dimensional safety evaluations, finding that LLMs are unreliable for nuanced safety issues like financial advice but m…

View →
cs.AIEmpiricalRecentJun 23, 2026

When CQs Go Wrong: Challenges in CQ Verification with OE-Assist

Anna Sofia Lippolis, Mohammad Javad Saeedizade, Robin Keskisärkkä, Aldo Gangemi +2 more

This paper investigates the challenges of Competency Questions (CQs) in CQ-verification and proposes solutions to enhance users' performance.

View →
stat.MEcs.CYcs.HCEmpiricalRecentJun 23, 2026

When Surveys Become Conversations: Adaptive Matrix Validation for AI-Assisted Interviews

Tyler H. McCormick

This paper proposes Adaptive Matrix Validation (AMV), a method for validating AI-assisted interview data in surveys using statistical adjustment.

View →
cs.CRcs.AIcs.MAMay 1, 2026

Skills as Verifiable Artifacts: A Trust Schema and a Biconditional Correctness Criterion for Human-in-the-Loop Agent Runtimes

Alfredo Metere

The paper proposes a trust schema and verification framework to ensure that agent skills, which augment LLMs, are rigorously verified before deployment, thereby making human-in-the-loop oversight scal…

View →
cs.AIcs.MAcs.SESurveyRecentJul 31, 2026

Beyond Component Testing: Validating Agentic AI Systems

Fabio Orazio Mirto, Luca D'Agati, Giuseppe Tricomi, Stefano Silvestri +3 more

This paper surveys 257 papers on validating agentic AI systems, identifying mature areas and gaps in behavioral, safety, temporal, regulatory, and multi-agent concerns.

View →
cs.CLRecentMay 28, 2026

Auditing LLM Benchmarks with Item Response Theory

Sander Land, Daniel M. Bikel

The paper introduces an Item Response Theory (IRT)-based indicator that effectively identifies likely mislabeled items in existing LLM benchmarks, revealing systematic errors in labeling and model spe…

View →
cs.CLcs.AIEmpiricalRecentJun 26, 2026

Can LLMs Judge Better Than They Generate? Evaluating Task Asymmetry, Mechanistic Interpretability and Transferability for In-Context QA

Sambaran Bandyopadhyay

This paper tests the assumption that evaluation is easier than generation in LLM-as-a-Judge and self-evaluation pipelines using a controlled in-context QA setting and reveals that evaluation attends t…

View →
cs.IRcs.AIcs.CLEmpiricalRecentOct 26, 2026

Knowing When the Evidence Is Enough: Requirement-Directed Search with Horn-Closure Certificates

Xinyu Zhang, Weiwei Sun

The paper presents Certify, a search-augmented model that decides whether its evidence suffices to answer a question by making sufficiency structural and using a per-query requirement hypergraph.

View →
cs.LGstat.MLEmpiricalRecentJul 23, 2026

Finite-Sample Coverage Audits for High-Recall Candidate Generation: Certification and Learning-Theoretic Design

Martin Anthony, Kaveh Salehzadeh Nobari

This paper characterizes the label complexity of certifying small missed mass in an empirical pipeline and shows that auditing the excluded pool is minimax rate-optimal.

View →
cs.SEcs.CLcs.LGEmpiricalRecentOct 24, 2026

Why Does Structured Output Fail? An Empirical Study of Prompt-Enforced Pydantic Schemas Across Nine LLMs

Amit Verma

Fine-grained taxonomy of schema-conformance failures and a deterministic classifier for predicting failure categories from model responses and Pydantic validation errors

View →
cs.CLcs.AIRecentMay 27, 2026

Models That Know How Evaluations Are Designed Score Safer

Katharina Deckenbach, Haritz Puerto, Jonas Geiping, Sahar Abdelnabi

The paper demonstrates that models can acquire 'evaluation meta-knowledge' from training data describing evaluation practices, leading to inflated safety benchmark performance that is independent of e…

View →
cs.LGcs.CLEmpiricalRecentOct 16, 2026

Sample, Mix, or Verify? Allocating Test-Time Compute under Imperfect Verifiers

Srinivasan Iyengar

This paper introduces SPLIT, a method to optimize the allocation of compute resources for language models and agents to achieve accurate results.

View →
cs.AIcs.DBcs.NEEmpiricalRecentJun 18, 2026

Beyond Accuracy: Measuring Logical Compliance of Predictive Models

Guillaume Olivier Delplanque, Pierre Genevès, Nabil Layaïda, Zephirin Faure

The paper introduces the Rule Violation Score (RVS) metric to evaluate the logical consistency of predictive models.

View →
cs.CLEmpiricalRecentJul 22, 2026

Which Values Do LLMs Confuse? A Schwartz-Based Recognition Study

Andrei Chetvergov, Stepan Ukolov, Timofei Sivoraksha, Alexander Evseev +4 more

This paper evaluates the ability of large language models to identify Schwartz's ten basic values in Russian situational texts and identifies recurring directed confusions.

View →
cs.CRcs.AIApr 22, 2026

CyberCertBench: Evaluating LLMs in Cybersecurity Certification Knowledge

Gustav Keppler, Ghada Elbez, Veit Hagenmeyer

The paper introduces CyberCertBench, a new benchmark suite for evaluating LLMs against industry cybersecurity certifications, finding that while frontier models perform well on general knowledge, thei…

View →
cs.AIRecentMay 28, 2026

FinVerBench: Benchmark Validity and Calibration in Large Language Model Financial Statement Verification

Silu Panda

The paper introduces FinVerBench, a comprehensive benchmark for financial statement verification, concluding that successful verification requires calibrated judgment under realistic observational con…

View →