Annotrace: A Workbench for Step-Level Annotation of Agent Traces
Annotrace is a tool for annotating and labeling traces of LLM agents.
Annotrace introduces a novel approach to annotating and labeling LLM agent traces by allowing joint human-AI annotation and measuring agreement between annotators.
Keywords
Before reading this…
Applications
- →LLM agent failure analysis
To understand this paper, make sure you know these concepts first:
- Understanding of LLM agentsfind papers →
- Knowledge of annotation and labeling techniquesfind papers →
Abstract
More Like ThisEvery study of why LLM agents fail begins by labelling traces by hand, and every such study builds its own throwaway tool. Observability platforms record agent traces but offer no structured labelling; annotation tools label text but have no notion of a step, a delegation or a tool call, leaving it unclear what the smallest unit of execution to label should be. Annotrace reads six trace formats, including OpenTelemetry GenAI, LangSmith and Langfuse, and turns each run into an ordered, nested list of steps. The annotator writes their own labelling codebook; a frontier model proposes a starting set from a sample of their own traces, answers first, and is asked exactly the questions the human is asked, making annotation a joint human-AI process. Annotrace measures agreement between any two annotators, human or AI. On a multi-agent question-answering corpus it reproduces a previous manual effort's 235 annotation units exactly, and two models labelling the same traces agree 91% of the time at $κ= 0.82$.