ArXivCSExplorer
☆☆Bookmarks🏆RSSHow to UseFAQ
Built with and by Teycir Ben Soltane•
How to Use•FAQ•GitHub•arXiv.org•
Share:

20 results for “Crowdsourced tests”

CS papers only

Hybrid search: Keyword + semantic, ranked by combined score.ⓘ

Want pure semantic search? Try claim verification →

eess.ASEmpiricalRecentJun 26, 2026

Screening Matters: A Comparative Study of Conventional and Crowdsourced Listening Tests

Anika Treffehn, Andrea Eichenseer, Emily Kratsch, Nicola Pia

This paper compares the effectiveness of classical and neural speech codecs using P.808 and P.800 DCR tests in crowdsourced and conventional evaluations. It proposes suitable screening methods for imp…

View →
cs.IREmpiricalRecentJun 27, 2026

Human-in-the-Loop Nugget Annotation for Accountable LLM-as-a-Judge Evaluations

Laura Dietz

A prototype annotation tool is presented that allows humans to identify important information and LLMs to match it to system outputs, creating a division of labor for reliable AI evaluation.

View →
cs.AIRecentMay 27, 2026

Benchmarking AI for low-resource contexts: Thinking beyond leaderboards

Aakash Pant, Kavya Shah, Apoorv Agnihotri, Sneha Nikam +2 more

The paper critiques current AI benchmarking practices for low-resource settings, arguing that evaluation must shift focus from isolated model performance to the holistic performance of the deployed sy…

View →
cs.AIcs.CLcs.CYEmpiricalRecentJun 12, 2026

Every Eval Ever: A Unifying Schema and Community Repository for AI Evaluation Results

Jan Batzner, Sree Harsha Nelaturu, Anastassia Kornilova, Jon Crall +43 more

The paper introduces Every Eval Ever, a shared schema and community-crowdsourced repository for AI evaluation results to standardize and compare AI evaluations.

View →
cs.CRcs.CLApr 28, 2026

MGTEVAL: An Interactive Platform for Systemtic Evaluation of Machine-Generated Text Detectors

Yuanfan Li, Qi Zhou, Chengzhengxu Li, Zhaohan Zhang +4 more

The paper introduces MGTEVAL, a comprehensive and extensible platform designed to systematically evaluate the performance, robustness, and efficiency of machine-generated text detectors.

View →
cs.AIRecentMay 28, 2026

Croissant Tasks: A Metadata Format for Reproducible Machine Learning Evaluations

Omar Benjelloun, Leonardo Martins Bianco, Isabelle Guyon, Thanh Gia Hieu Khuong +7 more

The paper introduces Croissant Tasks, a declarative metadata format designed to achieve conceptual reproducibility in machine learning by abstracting problem specifications from brittle implementation…

View →
cs.SEEmpiricalRecentOct 11, 2026

BRQGen: Towards Adaptive Completeness Assessment and Follow-Up Question Generation for Bug Reports

Tevfik Emre Sungur, Akif Emre Köşüş, Uğurcan Doğruçam, Eray Tüzün

BRQGen is a framework that generates targeted follow-up questions for incomplete bug reports.

View →
cs.SEcs.AIcs.LGEmpiricalRecentJun 21, 2026

All Green, Still Broken: Real-Flow Verification Lessons from an LLM-Integrated, Multi-Market Web Application

Muhammad Bilal, Ali Hassaan Mughal

The authors report on a study of a production rental-search assistant and present a four-seam framework for identifying defects in complex systems.

View →
cs.AIEmpiricalRecentJul 16, 2026

Can We Trust Item Response Theory for AI Evaluation?

Han Jiang, Sunbeom Kwon, Jinwen Luo, Ziang Xiao +1 more

This paper evaluates the reliability of using item response theory (IRT) models for AI benchmarking, comparing four estimation tools under various simulation conditions.

View →
cs.CRRecentMay 27, 2026

Do you dare to try Test-Driven Forensics? Increasing Trust in Desktop Forensics with ADARE

Michael Külper, Martin Lambertz, Mariia Rybalka

The paper introduces Test-Driven Forensics, an approach that treats forensic expectations as executable tests to detect and measure the degradation of repeatability and confidence in digital forensic…

View →
cs.LGcs.AIcs.CRRecentMay 28, 2026

NumLeak: Public Numeric Benchmarks as Latent Labels in Foundation Models

Anany Kotawala

The paper introduces NumLeak, a framework demonstrating that top-tier LLMs often exhibit high fidelity recall of specific public numeric benchmarks (like financial factors) due to memorization, which…

View →
cs.LGcs.AIcs.CRRecentMay 28, 2026

NumLeak: Public Numeric Benchmarks as Latent Labels in Foundation Models

Anany Kotawala

The paper introduces NumLeak, a framework demonstrating that top-tier LLMs often exhibit high fidelity recall of specific public numeric benchmarks, suggesting that their apparent skill may be due to…

View →
cs.CLcs.IREmpiricalRecentJun 11, 2026

How Fine-Grained Should a RAG Benchmark Be? A Hierarchical Framework for Synthetic Question Generation

Chase M. Fensore, Kaustubh Dhole, Jason Fan, Eugene Agichtein +1 more

This paper presents a hierarchical framework for studying granularity in retrieval-augmented generation benchmark construction.

View →
cs.LGcs.AIRecentMay 27, 2026

FormInv: A Measurement Protocol for Semantic Invariance in Mathematical Reasoning Benchmarks

Nishal Thomas, Noel Thomas

The paper introduces FormInv, a measurement protocol that reveals significant semantic inconsistencies in existing mathematical reasoning benchmarks, showing that standard accuracy metrics fail to cap…

View →
cs.SEEmpiricalRecentJun 10, 2026

Characterizing Tests in IoT Software: Practices, Challenges and Opportunities

Rufeng Chen, Hengcheng Zhu, Wuqi Zhang, Zixu Zhou +1 more

This paper evaluates the effectiveness of test cases in IoT software and identifies challenges in testing IoT software.

View →
cs.LGcs.CLEmpiricalRecentOct 16, 2026

Sample, Mix, or Verify? Allocating Test-Time Compute under Imperfect Verifiers

Srinivasan Iyengar

This paper introduces SPLIT, a method to optimize the allocation of compute resources for language models and agents to achieve accurate results.

View →
cs.CLcs.AIEmpiricalRecentJun 26, 2026

Can LLMs Judge Better Than They Generate? Evaluating Task Asymmetry, Mechanistic Interpretability and Transferability for In-Context QA

Sambaran Bandyopadhyay

This paper tests the assumption that evaluation is easier than generation in LLM-as-a-Judge and self-evaluation pipelines using a controlled in-context QA setting and reveals that evaluation attends t…

View →
stat.APcs.SETheoreticalRecentOct 24, 2026

The Computational Assessment Stack: An Architecture for AI-Native Measurement

Alina A. von Davier

This paper proposes the Computational Assessment Stack (CAS), a four-layer architecture for AI-native measurement in test development.

View →