20 results for “Crowdsourced tests”
CS papers onlyHybrid search: Keyword + semantic, ranked by combined score.ⓘ
Want pure semantic search? Try claim verification →
This paper compares the effectiveness of classical and neural speech codecs using P.808 and P.800 DCR tests in crowdsourced and conventional evaluations. It proposes suitable screening methods for imp…
A prototype annotation tool is presented that allows humans to identify important information and LLMs to match it to system outputs, creating a division of labor for reliable AI evaluation.
Aakash Pant, Kavya Shah, Apoorv Agnihotri, Sneha Nikam +2 more
The paper critiques current AI benchmarking practices for low-resource settings, arguing that evaluation must shift focus from isolated model performance to the holistic performance of the deployed sy…
Jan Batzner, Sree Harsha Nelaturu, Anastassia Kornilova, Jon Crall +43 more
The paper introduces Every Eval Ever, a shared schema and community-crowdsourced repository for AI evaluation results to standardize and compare AI evaluations.
Yuanfan Li, Qi Zhou, Chengzhengxu Li, Zhaohan Zhang +4 more
The paper introduces MGTEVAL, a comprehensive and extensible platform designed to systematically evaluate the performance, robustness, and efficiency of machine-generated text detectors.
The paper introduces Croissant Tasks, a declarative metadata format designed to achieve conceptual reproducibility in machine learning by abstracting problem specifications from brittle implementation…
BRQGen is a framework that generates targeted follow-up questions for incomplete bug reports.
The authors report on a study of a production rental-search assistant and present a four-seam framework for identifying defects in complex systems.
Han Jiang, Sunbeom Kwon, Jinwen Luo, Ziang Xiao +1 more
This paper evaluates the reliability of using item response theory (IRT) models for AI benchmarking, comparing four estimation tools under various simulation conditions.
The paper introduces Test-Driven Forensics, an approach that treats forensic expectations as executable tests to detect and measure the degradation of repeatability and confidence in digital forensic…
The paper introduces NumLeak, a framework demonstrating that top-tier LLMs often exhibit high fidelity recall of specific public numeric benchmarks (like financial factors) due to memorization, which…
The paper introduces NumLeak, a framework demonstrating that top-tier LLMs often exhibit high fidelity recall of specific public numeric benchmarks, suggesting that their apparent skill may be due to…
Chase M. Fensore, Kaustubh Dhole, Jason Fan, Eugene Agichtein +1 more
This paper presents a hierarchical framework for studying granularity in retrieval-augmented generation benchmark construction.
The paper introduces FormInv, a measurement protocol that reveals significant semantic inconsistencies in existing mathematical reasoning benchmarks, showing that standard accuracy metrics fail to cap…
Rufeng Chen, Hengcheng Zhu, Wuqi Zhang, Zixu Zhou +1 more
This paper evaluates the effectiveness of test cases in IoT software and identifies challenges in testing IoT software.
This paper introduces SPLIT, a method to optimize the allocation of compute resources for language models and agents to achieve accurate results.
This paper tests the assumption that evaluation is easier than generation in LLM-as-a-Judge and self-evaluation pipelines using a controlled in-context QA setting and reveals that evaluation attends t…
This paper proposes the Computational Assessment Stack (CAS), a four-layer architecture for AI-native measurement in test development.