20 results for “evaluation framework”
CS papers onlyHybrid search: Keyword + semantic, ranked by combined score.ⓘ
Want pure semantic search? Try claim verification →
Pengyu Zhu, Lijun Li, Yaxing Lyu, Qianxin Luo +7 more
The paper introduces a unified framework to fairly evaluate LLM agentic capabilities by standardizing diverse benchmarks and separating the effects of the LLM model from the surrounding framework and…
BADGER is a unified, production-grade evaluation framework that integrates text-to-SQL assessment with agentic behavior evaluation, significantly outperforming existing benchmarks on industry queries.
This paper introduces a novel evaluation framework for Large Language Models using pairwise comparisons and Elo rating system.
This paper examines the validity of performance measurement in machine learning under different label-authority regimes.
LLM-FACETS introduces an open-source, privacy-preserving framework designed to enable non-technical domain experts and compliance officers to audit and evaluate the transparency and accountability of…
Jan Batzner, Sree Harsha Nelaturu, Anastassia Kornilova, Jon Crall +43 more
The paper introduces Every Eval Ever, a shared schema and community-crowdsourced repository for AI evaluation results to standardize and compare AI evaluations.
Laurens Samson, Iva Gornishka, Gossa Lô, Yuki M. Asano +1 more
The paper introduces the 'Grip on LLMs' framework, a systematic evaluation suite for Dutch governmental use of large language models, identifying six evaluation dimensions and operationalising them in…
Ryan Burnell, Yumeya Yamamori, Orhan Firat, Kate Olszewska +9 more
The paper introduces a Cognitive Taxonomy and a rigorous evaluation protocol to provide an objective, multi-faceted framework for measuring system capabilities and tracking progress toward Artificial…
Aakash Pant, Kavya Shah, Apoorv Agnihotri, Sneha Nikam +2 more
The paper critiques current AI benchmarking practices for low-resource settings, arguing that evaluation must shift focus from isolated model performance to the holistic performance of the deployed sy…
This paper introduces Human-on-the-Bridge (HOB), a scalable evaluation paradigm for agentic AI that amplifies evaluation quality without requiring equally large evaluator models.
The paper proposes a TEE-based architecture that enables external, auditable verification of AI-assisted grant evaluations without exposing the proprietary model, scoring logic, or intermediate reason…
The Internet Quality Barometer (IQB) framework maps Internet measurement data to actionable insights about Internet quality.
The authors present an evaluation framework for agent skills that assesses their utility and provides a reusable methodology for evaluating individual skills.
This paper studies AI development frameworks for software engineering and proposes a six-dimension process taxonomy.
Aaron Chan, Tengfei Li, Tianyi Xiao, Angela Chen +2 more
The paper introduces LATTICE, a novel benchmark for evaluating how well crypto agents assist user decision-making, finding that different agents excel in different specific areas rather than having a…
This paper introduces DRAFT-Bench, a lifecycle-oriented evaluation framework for assessing the capabilities of frontier general-purpose agents in design tasks.
Weizhi Zhang, Zechen Li, Hamid Palangi, Ben Graef +15 more
The authors introduce RubricsTree, a scalable evaluation framework for personal health agents that uses a hierarchical taxonomy of clinically-verifiable rubrics to improve the evaluation of large-scal…
This paper proposes a comprehensive, risk-based auditing framework designed to help internal and external auditors assess the cybersecurity risks posed by diverse IoT devices within corporate and indu…
The BEAMS initiative establishes comprehensive benchmarks and evaluates AI tools for modeling and simulation, finding that current AI tools excel at qualitative discussion tasks but struggle with comp…
Weiyi Chen, Shuaixiong Wang, Ziyun Gao, Kaichun Hu +4 more
The paper introduces TravelEval, a comprehensive, six-dimensional benchmarking framework that evaluates LLM-powered travel plans using realistic spatio-temporal simulation, revealing that current LLMs…