This paper investigates the performance of frontier agents in protein-structure tasks and evaluates the impact of evaluation configurations on benchmark scores.
This paper introduces a new benchmark suite, ProteinGeoBench, and uses it to evaluate the performance of frontier agents in protein-structure tasks.
Before reading this…
Applications
To understand this paper, make sure you know these concepts first:
Frontier agents retrieve structural metadata well and compute geometry from coordinates less reliably. We ask how much of a benchmark score reflects that gap and how much reflects the evaluation configuration around the model. We study ProteinGeoBench, a suite of 131 protein-structure tasks, using 4,249 rollouts from four frontier models and 4,486 further trials across 26 models. Three of the four frontier models fall within 0.031 of one another (0.785 to 0.816) with no difference between them resolved, while the fourth scores 0.707 and all three gaps to it exclude zero. Three of the four do better on metadata lookup than on numeric calculation. The fourth is slightly better on numeric. Numeric tasks are 28 of the 131 but half of the twelve tasks that all four models fail. The ranking also depends on the evaluation budget: with one run per task, 23.9% of bootstrap replicates put a different model first. On a selected 15-task slice, removing network access lowers scores more than blocking eight structural-database domains, but the two contrasts were graded in different passes and cannot be cleanly decomposed. Swapping in each model's own command-line agent changes scores by -0.071 to +0.086, a comparison too small to rule out moderate harness effects. Across 26 models, numeric performance rises with non-numeric performance rather than staying flat. The experiments use different task subsets, so they cannot rank model, harness, access and task composition against one another. Evaluations should report results by task family, along with the number of runs, the harness and the access conditions.
Flex-RSI: Benchmarking Recursive Self-Improvement of Frontier Agents
The paper introduces Flex-RSI, a benchmark for evaluating the recursive self-imp…
GameLogicBench: Evaluating Coding Agents on Runtime Game Logic with Tick-Level State Assertions
This paper introduces GameLogicBench, a benchmark for evaluating the ability of…
Training-free Task Classification for Multi-Task Model Merging
This paper proposes a method called SiM that merges multiple task-specific exper…
Object-Oriented Harness: Evolving Agent Harnesses through Object-Oriented Design
This paper introduces Object-Oriented Harness (OO-Harness), a framework that opt…
GenOlympiad: Benchmarking Frontier Image Models at Their Capability Boundaries
A benchmark called GenOlympiad is introduced to evaluate the performance of text…
Agent in a Bottle: Can LLM Agents Turn Their Capabilities Into Cheap, Scalable Artifacts?
This paper introduces a benchmark called BOTTLED to evaluate the ability of larg…
InfraDev: Can Coding Agents Develop AI Infrastructure, and How Can We Tell?
The paper introduces InfraDev, a benchmark for evaluating AI infrastructure deve…
Towards Agentic Proof Harness Engineering
The paper proposes an agentic proof-harness engineer that automatically construc…