Comparing Computer Use Behavior Across Multiple Levels of Abstraction
This paper presents MAP, a hierarchical semantic representation and annotation pipeline for comparing interaction traces across tasks, interfaces, and input modalities.
MAP provides a novel hierarchical semantic representation and annotation pipeline for comparing interaction traces, which is not addressed by prior work.
Before reading this…
Applications
- →Usability testing
- →Human-computer interaction analysis
To understand this paper, make sure you know these concepts first:
- Human-computer interactionfind papers →
- Usability analysisfind papers →
Abstract
More Like ThisInteraction traces record how people and agents use computers, and comparing them can reveal usability barriers, differences in expertise, and human--agent similarity. Yet traces are recorded as low-level, interface-specific actions, making comparisons difficult across tasks, interfaces, and input modalities. Existing approaches either compare these actions directly, reduce traces to aggregate outcomes, or summarize them at higher-level goals, obscuring where behavior differs. We present MAP (Multi-level Action Parsing), a hierarchical semantic representation and annotation pipeline that maps traces into a common vocabulary across linked levels of abstraction. MAP compares traces by sequence, recurring patterns, and effort, localizing differences by type and abstraction level. A technical evaluation across three computer use datasets shows high agreement with manual annotations and broad coverage of behavior. Applying MAP shows that sighted and blind users increasingly converge at higher levels despite navigation differences, while agents omit corrective, help-seeking, and recovery behaviors present in human traces.