20 results for “evaluation protocol”
CS papers onlyHybrid search: Keyword + semantic, ranked by combined score.ⓘ
Want pure semantic search? Try claim verification →
The paper introduces a novel, practical evaluation protocol that shifts the assessment of AI pentesting agents from simple task completion to validated, open-ended vulnerability discovery in complex,…
The paper introduces FormInv, a measurement protocol that reveals significant semantic inconsistencies in existing mathematical reasoning benchmarks, showing that standard accuracy metrics fail to cap…
The paper introduces a comprehensive security framework, AgentRFC, to systematically analyze and test the security conformance of various AI agent protocols, identifying critical design gaps, especial…
The paper introduces Acceptance Cards, a rigorous four-diagnostic standard, to provide a comprehensive and reliable evaluation protocol for claims of safe fine-tuning defenses.
Jan Batzner, Sree Harsha Nelaturu, Anastassia Kornilova, Jon Crall +43 more
The paper introduces Every Eval Ever, a shared schema and community-crowdsourced repository for AI evaluation results to standardize and compare AI evaluations.
The paper proposes a TEE-based architecture that enables external, auditable verification of AI-assisted grant evaluations without exposing the proprietary model, scoring logic, or intermediate reason…
This paper introduces Human-on-the-Bridge (HOB), a scalable evaluation paradigm for agentic AI that amplifies evaluation quality without requiring equally large evaluator models.
The paper introduces mcp-attested, a security extension to the Model Context Protocol (MCP) that allows hosts to safely admit and restrict the tools used by external, third-party tool servers.
Aakash Pant, Kavya Shah, Apoorv Agnihotri, Sneha Nikam +2 more
The paper critiques current AI benchmarking practices for low-resource settings, arguing that evaluation must shift focus from isolated model performance to the holistic performance of the deployed sy…
This paper examines the validity of performance measurement in machine learning under different label-authority regimes.
Shreeyans Arora, Utsav Avaiya, Zera Lyngkhoi, Hem Gosalia +3 more
The paper presents AuditKIT, a library for evaluating language models and agents by recording various measurements in one result representation.
The paper introduces WebSP-Eval, a new framework to evaluate web agents on complex website security and privacy tasks, finding that current state-of-the-art models struggle significantly with stateful…
The paper introduces the Human Delegation Provenance (HDP) protocol, a lightweight, token-based cryptographic scheme designed to verify the full, multi-hop chain of human authorization for actions exe…
This paper proposes HCP, a Handle-Capability Protocol runtime for MCP-style agent systems, which implements eight execution-control invariants to enhance security while preserving MCP-like workflows.
Swastik Roy, Rajkumar Pujari, Tharindu Kumarage, Charith Peris +4 more
PReMISE introduces a framework to audit and improve the quality of rubrics used to guide LLM judges, demonstrating that it can significantly increase judge accuracy and reduce the exploitability of re…
The paper introduces a 'replication-first' paradigm for LLM behavioral benchmarking, demonstrating that this rigorous approach uncovers significant, non-obvious performance drops between successive mo…
Guoxiang, Guo, Kla Tantithamthavorn, Neelofar Neelofar +2 more
This paper introduces CodeChat-Eval, an evaluation framework for assessing the functional correctness of Large Language Models in multi-turn code refinement dialogues.