20 results for “Familiarity with evaluation frameworks”
CS papers onlyHybrid search: Keyword + semantic, ranked by combined score.ⓘ
Want pure semantic search? Try claim verification →
Jan Batzner, Sree Harsha Nelaturu, Anastassia Kornilova, Jon Crall +43 more
The paper introduces Every Eval Ever, a shared schema and community-crowdsourced repository for AI evaluation results to standardize and compare AI evaluations.
Aakash Pant, Kavya Shah, Apoorv Agnihotri, Sneha Nikam +2 more
The paper critiques current AI benchmarking practices for low-resource settings, arguing that evaluation must shift focus from isolated model performance to the holistic performance of the deployed sy…
Shreeyans Arora, Utsav Avaiya, Zera Lyngkhoi, Hem Gosalia +3 more
The paper presents AuditKIT, a library for evaluating language models and agents by recording various measurements in one result representation.
The paper demonstrates that models can acquire 'evaluation meta-knowledge' from training data describing evaluation practices, leading to inflated safety benchmark performance that is independent of e…
This paper tests the assumption that evaluation is easier than generation in LLM-as-a-Judge and self-evaluation pipelines using a controlled in-context QA setting and reveals that evaluation attends t…
This paper introduces DRAFT-Bench, a lifecycle-oriented evaluation framework for assessing the capabilities of frontier general-purpose agents in design tasks.
The paper evaluates the inconsistency of using LLMs as automated judges for multi-dimensional safety evaluations, finding that LLMs are unreliable for nuanced safety issues like financial advice but m…
Laurens Samson, Iva Gornishka, Gossa Lô, Yuki M. Asano +1 more
The paper introduces the 'Grip on LLMs' framework, a systematic evaluation suite for Dutch governmental use of large language models, identifying six evaluation dimensions and operationalising them in…
The paper introduces an agentic, framework-based system to transform under-specified academic papers into standardized, comparable, and executable benchmarks for industrial Prognostics and Health Mana…
This paper evaluates Centaur, a foundation model trained on psychological experiments, for program comprehension tasks and compares its performance to Llama 3.1.
The authors present an evaluation framework for agent skills that assesses their utility and provides a reusable methodology for evaluating individual skills.
Jun Zhang, JianYing Qu, Hanwen Du, Zhongkai Sun +2 more
The paper introduces Code-QA-Bench, a novel framework that rigorously separates genuine code reasoning from mere documentation memorization in repository-level code understanding benchmarks.
Pengyu Zhu, Lijun Li, Yaxing Lyu, Qianxin Luo +7 more
The paper introduces a unified framework to fairly evaluate LLM agentic capabilities by standardizing diverse benchmarks and separating the effects of the LLM model from the surrounding framework and…
The paper introduces CodeGolf Bench, a novel multi-language benchmark using code golf to measure LLMs' ability to generate highly concise and efficient code, showing that reasoning models significantl…
BADGER is a unified, production-grade evaluation framework that integrates text-to-SQL assessment with agentic behavior evaluation, significantly outperforming existing benchmarks on industry queries.
This paper identifies three core weaknesses—benchmark vulnerabilities, temporal staleness, and runtime uncertainty—that undermine current AI agent security evaluations and proposes directions for buil…
This paper introduces Human-on-the-Bridge (HOB), a scalable evaluation paradigm for agentic AI that amplifies evaluation quality without requiring equally large evaluator models.
Nhat Minh Le, Yisen Xu, Zhijie Wang, Tse-Hsun +1 more
This paper evaluates the performance of large language models on popular benchmarks and finds that only a small percentage of the performant implementations are significantly faster than canonical sol…
This paper investigates the gap between benchmark performance and deployment behavior in AI models.