ArXivCSExplorer
☆☆Bookmarks🏆RSSHow to UseFAQ
Built with and by Teycir Ben Soltane•
How to Use•FAQ•GitHub•arXiv.org•
Share:

20 results for “Familiarity with evaluation frameworks”

CS papers only

Hybrid search: Keyword + semantic, ranked by combined score.ⓘ

Want pure semantic search? Try claim verification →

cs.AIcs.CLcs.CYEmpiricalRecentJun 12, 2026

Every Eval Ever: A Unifying Schema and Community Repository for AI Evaluation Results

Jan Batzner, Sree Harsha Nelaturu, Anastassia Kornilova, Jon Crall +43 more

The paper introduces Every Eval Ever, a shared schema and community-crowdsourced repository for AI evaluation results to standardize and compare AI evaluations.

View →
cs.AIRecentMay 27, 2026

Benchmarking AI for low-resource contexts: Thinking beyond leaderboards

Aakash Pant, Kavya Shah, Apoorv Agnihotri, Sneha Nikam +2 more

The paper critiques current AI benchmarking practices for low-resource settings, arguing that evaluation must shift focus from isolated model performance to the holistic performance of the deployed sy…

View →
cs.CLcs.AIcs.IREmpiricalRecentOct 15, 2026

AuditKIT: A Unified Evaluation Spine for Benchmarking, Judging, and Comparing Large Language Models and Agentic Systems

Shreeyans Arora, Utsav Avaiya, Zera Lyngkhoi, Hem Gosalia +3 more

The paper presents AuditKIT, a library for evaluating language models and agents by recording various measurements in one result representation.

View →
cs.CLcs.AIRecentMay 27, 2026

Models That Know How Evaluations Are Designed Score Safer

Katharina Deckenbach, Haritz Puerto, Jonas Geiping, Sahar Abdelnabi

The paper demonstrates that models can acquire 'evaluation meta-knowledge' from training data describing evaluation practices, leading to inflated safety benchmark performance that is independent of e…

View →
cs.CLcs.AIEmpiricalRecentJun 26, 2026

Can LLMs Judge Better Than They Generate? Evaluating Task Asymmetry, Mechanistic Interpretability and Transferability for In-Context QA

Sambaran Bandyopadhyay

This paper tests the assumption that evaluation is easier than generation in LLM-as-a-Judge and self-evaluation pipelines using a controlled in-context QA setting and reveals that evaluation attends t…

View →
cs.SEcs.AIEmpiricalRecentNov 6, 2026

DRAFT-Bench: Evaluating Frontier Agents Across the Professional Design Lifecycle

Rong Pan, Jianqiang Wang

This paper introduces DRAFT-Bench, a lifecycle-oriented evaluation framework for assessing the capabilities of frontier general-purpose agents in design tasks.

View →
cs.CLRecentMay 29, 2026

LLM Judges Inconsistently Disagree Across Safety Criteria and Harm Categories

Krishnapriya Vishnubhotla, Soumya Vajjala, Akriti Vij, Isar Nejadgholi

The paper evaluates the inconsistency of using LLMs as automated judges for multi-dimensional safety evaluations, finding that LLMs are unreliable for nuanced safety issues like financial advice but m…

View →
cs.CLcs.AIEmpiricalRecentAug 10, 2026

From Values to Benchmarks: Evaluating Large Language Models for Governmental Use in Dutch

Laurens Samson, Iva Gornishka, Gossa Lô, Yuki M. Asano +1 more

The paper introduces the 'Grip on LLMs' framework, a systematic evaluation suite for Dutch governmental use of large language models, identifying six evaluation dimensions and operationalising them in…

View →
cs.AIcs.LGcs.SERecentMay 27, 2026

From paper to benchmark: agentic, framework-based reproduction of under-specified methods in machine health intelligence

Raffael Theiler, Ludovico Comito, David Leko, Leandro Von Krannichfeldt +2 more

The paper introduces an agentic, framework-based system to transform under-specified academic papers into standardized, comparable, and executable benchmarks for industrial Prognostics and Health Mana…

View →
cs.SEEmpiricalRecentJul 13, 2026

Predicting Program Comprehension with Foundation Models of Human Cognition

Yannick Lehmen, Marvin Wyrich, Anna-Maria Maurer, Norman Peitek +1 more

This paper evaluates Centaur, a foundation model trained on psychological experiments, for program comprehension tasks and compares its performance to Llama 3.1.

View →
cs.SEcs.AIcs.CLEmpiricalRecentJun 16, 2026

A Framework for Evaluating Agentic Skills at Scale

Maksim Shaposhnikov, Nicolas Fortuin, Simon Stipcich, Maria I. Gorinova +2 more

The authors present an evaluation framework for agent skills that assesses their utility and provides a reusable methodology for evaluating individual skills.

View →
cs.SEcs.AIRecentMay 28, 2026

Code-QA-Bench: Separating Code Reasoning from Documentation Memorization in Repository-Level QA

Jun Zhang, JianYing Qu, Hanwen Du, Zhongkai Sun +2 more

The paper introduces Code-QA-Bench, a novel framework that rigorously separates genuine code reasoning from mere documentation memorization in repository-level code understanding benchmarks.

View →
cs.AIRecentMay 27, 2026

A Unified Framework for the Evaluation of LLM Agentic Capabilities

Pengyu Zhu, Lijun Li, Yaxing Lyu, Qianxin Luo +7 more

The paper introduces a unified framework to fairly evaluate LLM agentic capabilities by standardizing diverse benchmarks and separating the effects of the LLM model from the surrounding framework and…

View →
cs.SEcs.AIRecentMay 28, 2026

CodeGolf Bench: A Multi-Language Benchmark for Evaluating Concise Code Generation Capabilities of Large Language Models

Vedant Padwal

The paper introduces CodeGolf Bench, a novel multi-language benchmark using code golf to measure LLMs' ability to generate highly concise and efficient code, showing that reasoning models significantl…

View →
cs.AIRecentJun 1, 2026

BADGER: Bridging Agentic and Deterministic Evaluation for Generative Enterprise Reasoning

Shannon Serrao, Soumitra Chatterjee, Dorina Strori, Abhishek Sharma +1 more

BADGER is a unified, production-grade evaluation framework that integrates text-to-SQL assessment with agentic behavior evaluation, significantly outperforming existing benchmarks on industry queries.

View →
cs.CRcs.AIMay 21, 2026

Measuring Security Without Fooling Ourselves: Why Benchmarking Agents Is Hard

Sahar Abdelnabi, Chris Hicks, Konrad Rieck, Ahmad-Reza Sadeghi

This paper identifies three core weaknesses—benchmark vulnerabilities, temporal staleness, and runtime uncertainty—that undermine current AI agent security evaluations and proposes directions for buil…

View →
cs.MAEmpiricalRecentJun 15, 2026

Human-on-the-Bridge: Scalable Evaluation for AI Agents

Fouad Bousetouane

This paper introduces Human-on-the-Bridge (HOB), a scalable evaluation paradigm for agentic AI that amplifies evaluation quality without requiring equally large evaluator models.

View →
cs.SEEmpiricalRecentJul 8, 2026

Rethinking Code Performance Benchmarks for LLMs

Nhat Minh Le, Yisen Xu, Zhijie Wang, Tse-Hsun +1 more

This paper evaluates the performance of large language models on popular benchmarks and finds that only a small percentage of the performant implementations are significantly faster than canonical sol…

View →
cs.CLEmpiricalRecentJun 22, 2026

Evaluation Awareness Is Not One Capability: Evidence from Open Language Models

Nilesh Nayan, Aishwarya Sampath Kumar, Rishiraj Girmal, Shivani Anilkumar +4 more

This paper investigates the gap between benchmark performance and deployment behavior in AI models.

View →