20 results for “benchmark suite”
CS papers onlyHybrid search: Keyword + semantic, ranked by combined score.ⓘ
Want pure semantic search? Try claim verification →
Nhat Minh Le, Yisen Xu, Zhijie Wang, Tse-Hsun +1 more
This paper evaluates the performance of large language models on popular benchmarks and finds that only a small percentage of the performant implementations are significantly faster than canonical sol…
BenchMeUp is a system that automatically creates software-engineering benchmarks from natural-language specifications.
The paper introduces XStack, a benchmark that evaluates agents on migrating ML repositories between PyTorch and JAX ecosystems.
Yikun Fu, Bowen Fu, Zhenyu Wu, Shuang Cheng +9 more
The paper presents MacAgentBench, a comprehensive macOS agent benchmark for evaluating computer use agents.
Weihuang Zheng, Tianyuan Zou, Eileen Ye, Alphet Liu +4 more
The paper introduces E-Bench, a synthetic benchmark for evaluating multi-step tool use in Large Language Models across three product domains.
The paper introduces CodeGolf Bench, a novel multi-language benchmark using code golf to measure LLMs' ability to generate highly concise and efficient code, showing that reasoning models significantl…
Junming Chen, Junyang Jiang, Xu Chen, Zibo Liang +1 more
The paper introduces DBA-Bench, a benchmark for evaluating database agents with production fidelity, outcome-first evaluation, and controlled scenario reproducibility.
Aakash Pant, Kavya Shah, Apoorv Agnihotri, Sneha Nikam +2 more
The paper critiques current AI benchmarking practices for low-resource settings, arguing that evaluation must shift focus from isolated model performance to the holistic performance of the deployed sy…
Jincheng Zhong, Weizhi Wang, Che Jiang, Kai Tian +4 more
The paper introduces EnterpriseClawBench, an enterprise agent benchmark with 852 tasks and evaluation protocol, achieving a best configuration score of 0.663.
Jun Zhang, JianYing Qu, Hanwen Du, Zhongkai Sun +2 more
The paper introduces Code-QA-Bench, a novel framework that rigorously separates genuine code reasoning from mere documentation memorization in repository-level code understanding benchmarks.
The paper introduces FinVerBench, a comprehensive benchmark for financial statement verification, concluding that successful verification requires calibrated judgment under realistic observational con…
This paper investigates code generation for no-resource programming languages using large language models.
Mengyu Zheng, Kai Han, Boxun Li, Haiyang Xu +12 more
The paper introduces Claw-SWE-Bench, a benchmark and adapter protocol for evaluating heterogeneous agent harnesses in software engineering tasks.
Yangzhen Wu, Aaron J. Li, Wenjie Ma, Li Cao +9 more
BenchEvolver introduces a solution-centric evolutionary framework to automatically transform saturated coding benchmarks into significantly harder, high-quality, and diverse evaluation suites.