20 results for “evaluation”
CS papers onlyHybrid search: Keyword + semantic, ranked by combined score.ⓘ
Want pure semantic search? Try claim verification →
Jan Batzner, Sree Harsha Nelaturu, Anastassia Kornilova, Jon Crall +43 more
The paper introduces Every Eval Ever, a shared schema and community-crowdsourced repository for AI evaluation results to standardize and compare AI evaluations.
The paper proposes a TEE-based architecture that enables external, auditable verification of AI-assisted grant evaluations without exposing the proprietary model, scoring logic, or intermediate reason…
This paper tests the assumption that evaluation is easier than generation in LLM-as-a-Judge and self-evaluation pipelines using a controlled in-context QA setting and reveals that evaluation attends t…
This paper examines the validity of performance measurement in machine learning under different label-authority regimes.
Junjie Chen, Yuxi Dong, Haitao Li, Weihang Su +4 more
The paper introduces LongJudgeBench, a new benchmark designed to evaluate the reliability of LLM judges specifically for complex, long-form output evaluation, revealing significant instability gaps in…
The paper demonstrates that models can acquire 'evaluation meta-knowledge' from training data describing evaluation practices, leading to inflated safety benchmark performance that is independent of e…
Yilun Qiu, Xiaoyan Zhao, Yang Zhang, Yuxin Chen +6 more
The paper introduces PARL, a framework that learns personalized evaluation rubrics directly from raw user interaction histories to accurately assess how well LLM outputs align with subjective, user-sp…
Weizhi Zhang, Zechen Li, Hamid Palangi, Ben Graef +15 more
The authors introduce RubricsTree, a scalable evaluation framework for personal health agents that uses a hierarchical taxonomy of clinically-verifiable rubrics to improve the evaluation of large-scal…
Aakash Pant, Kavya Shah, Apoorv Agnihotri, Sneha Nikam +2 more
The paper critiques current AI benchmarking practices for low-resource settings, arguing that evaluation must shift focus from isolated model performance to the holistic performance of the deployed sy…
This paper introduces Human-on-the-Bridge (HOB), a scalable evaluation paradigm for agentic AI that amplifies evaluation quality without requiring equally large evaluator models.
The paper introduces a novel, training-free method to automatically generate fine-grained evaluation rubrics for LLM-as-a-Judge, and further proposes an iterative fine-tuning strategy that significant…
Laurens Samson, Iva Gornishka, Gossa Lô, Yuki M. Asano +1 more
The paper introduces the 'Grip on LLMs' framework, a systematic evaluation suite for Dutch governmental use of large language models, identifying six evaluation dimensions and operationalising them in…
Shreeyans Arora, Utsav Avaiya, Zera Lyngkhoi, Hem Gosalia +3 more
The paper presents AuditKIT, a library for evaluating language models and agents by recording various measurements in one result representation.
This paper introduces a novel evaluation framework for Large Language Models using pairwise comparisons and Elo rating system.
The paper proposes a method to correct biased language models by aligning human-verified positives with unlabelled outputs in a fixed embedding space.
Zhaoyang Jiang, Xuanqi Peng, Fei Teng, Zhizhong Fu +4 more
The paper demonstrates that while distilling large language models for medical QA can significantly improve final answer accuracy, this gain often comes at the cost of factual accuracy and detailed re…
The paper introduces VIABLE, the first benchmark for evaluating Vision-Language Models (VLMs) as judges for Visually Impaired Assistance (VIA), finding that current models are largely unreliable and p…