20 results for “task over-specification”
CS papers onlyHybrid search: Keyword + semantic, ranked by combined score.ⓘ
Want pure semantic search? Try claim verification →
Yuanzhe Liu, Bastian Lipka, Avraham Shinnar, Jatin Ganhotra +2 more
The paper addresses the issue of task under-specification and over-specification in AI benchmarks, proposing techniques for deriving parsimonious hints.
The paper introduces GTA, a scalable framework for generating realistic, multi-hop web-agent tasks with dense, executable trajectories, addressing the current lack of process-level supervision in web…
Tomer Keren, Nitay Calderon, Asaf Yehudai, Yotam Perlitz +2 more
The paper introduces TASTE, an automatic task synthesis method that generates challenging agent benchmarks by evolving tool sequences, demonstrating that existing benchmarks are saturated and that TAS…
Yubin Qu, Yi Liu, Gelei Deng, Yanjun Zhang +3 more
The paper introduces SNARE, a novel adaptive benchmarking pipeline that systematically measures overeager behavior in coding agents, finding that the agent framework accounts for the majority of the v…
Yubin Qu, Yi Liu, Gelei Deng, Yanjun Zhang +3 more
The paper introduces SNARE, a novel adaptive testing pipeline that systematically measures overeager behavior in coding agents, finding that the agent framework accounts for the majority of the variat…
The paper introduces Croissant Tasks, a declarative metadata format designed to achieve conceptual reproducibility in machine learning by abstracting problem specifications from brittle implementation…
Sina Mavali, David Pape, Jonathan Evertz, Samira Abedini +4 more
The paper introduces the Task Alignment Benchmark (TAB) to evaluate terminal agents' ability to selectively follow relevant environmental instructions while ignoring misleading distractors, revealing…
Yubin Qu, Ying Zhang, Yanjun Zhang, Gelei Deng +3 more
The paper introduces OverEager-Gen, a new benchmark that measures 'overeager actions'—where coding agents perform unauthorized tasks beyond a benign request—and finds that removing explicit consent de…
Qiao Zhao, JianYing Qu, Jun Zhang, Yehua Yang +2 more
The paper proposes SWE-Future, a method for forecasting future coding tasks in software repositories and uses the forecasts to synthesize a dataset for coding-agents.
The paper introduces XLGoBench, a synthetic benchmark of algorithmic tasks designed to detect persistent cross-lingual skill gaps in large language models.
Likai Wang, Dezhao Su, Jing Hu, Yunyun Wan +2 more
A framework for measuring human contributions in human-agent joint production is proposed.
Sadra Sabouri, Hosein Hasanbeig, Adam Sage, Shreya Sinha +5 more
The paper presents a method to compile agent runs into reusable neuro-symbolic programs (NSPs) to improve efficiency and consistency.
Di Lu, Yongzhi Liao, Xutong Mu, Lele Zheng +4 more
The paper identifies that the convenience of host-acting agents leads to semantic under-specification in user goals, which forces the agent to generate potentially risky execution plans.
This paper investigates various methods for encoding factored tasks, a compact planning representation, into propositional logic for use with SAT solvers, analyzing the impact of encoding choices and…
SCOPE introduces a data-free self-play framework that co-evolves a task-generating Challenger and a document-answering Solver, significantly improving open-ended performance on language models without…
This paper develops a policy-learning framework to optimally assign prediction tasks to multiple agents, considering individual agent expertise and capacity constraints, achieving systematic performan…