cs.AI
AsyncTool: Evaluating the Asynchronous Function Calling Capability under Multi-Task Scenarios
May 27, 2026(revised May 28, 2026)
The paper introduces AsyncTool, a new benchmark designed to evaluate LLM agents' ability to handle multiple, concurrent tasks with delayed tool feedback, demonstrating that asynchronous coordination is a significant challenge for current models.
Large language model (LLM)-based agents have shown strong capabilities in using external tools to solve complex tasks. However, existing evaluations often overlook the temporal dimension of tool use, especially the impact of tool response latency, and are usually limited to single-task settings. In real-world applications, multiple tasks often need to be executed concurrently, and overall efficiency depends on whether an agent can use idle time while waiting for tool responses. We refer to this capability as asynchronous tool calling. To evaluate it, we propose AsyncTool, a benchmark for assessing LLM-based agents in interactive multi-task tool-use environments with delayed tool feedback. AsyncTool presents multiple heterogeneous tasks simultaneously and simulates realistic tool response latency during execution. Using a hybrid data evolution strategy, we construct a diverse asynchronous multitasking dataset that covers multiple scenarios and tool-use patterns. We evaluate models at the step, sub-task, and task levels, and introduce efficiency-oriented metrics to measure task coordination and completion efficiency. Extensive experiments show that delayed tool feedback poses substantial challenges to current agents and leads to clear performance degradation. Models that better coordinate task switching, dependency tracking, and state maintenance achieve stronger performance on AsyncTool. Our analysis identifies key failure modes of current tool-using agents and provides practical insights for designing future systems with stronger temporal reasoning and coordination capabilities.
On Effectiveness and Efficiency of Agentic Tool-calling and RL Training
This paper analyzes tool-calling in LLM agents, demonstrating that evaluation re…
EgoBench: An Interactive Egocentric Multimodal Benchmark for Tool-Using Agents
The paper introduces EgoBench, the first interactive multimodal benchmark design…
Do Agents Know What They Can't Do? Evaluating Feasibility Awareness in Tool-Using Agents
The paper proposes FeasiGen, a method to automatically create infeasible tasks f…
Tool Forge: A Validation-Carrying Toolchain for Governed Agentic Execution
Tool Forge is a validation-carrying toolchain that converts natural language cap…
Same Payload, Different Channel: Measuring Trust Asymmetry in Tool-Using Language Models
The paper introduces the Safety Asymmetry Score (SAS) to measure how a model's s…
ParaTool: Shifting Tool Representations from Context to Parameters
ParaTool introduces a novel framework that shifts tool representations from bulk…
Momento: Evaluating Persistent Memory and Reasoning with Multi-Session Agentic Conversations
The paper introduces Momento, a new benchmark that evaluates agentic AI's abilit…
MAVEN: Improving Generalization in Agentic Tool Calling
The paper introduces MAVEN, a lightweight symbolic reasoning scaffold that signi…