Waypoint: Analytically Modeling GEMM for CPUs
A new analytical model called Waypoint is presented to automatically identify the optimal tile size for General Matrix Multiplication (GEMM) on a given hardware and input matrix shape.
Waypoint is a novel analytical model that automatically identifies the optimal tile size for GEMM, unlike prior work that relies on manual tuning.
Before reading this…
Applications
- →Accelerating LLM inference on CPUs
To understand this paper, make sure you know these concepts first:
- Knowledge of General Matrix Multiplicationfind papers →
- Understanding of computer architecturefind papers →
Abstract
More Like ThisHigh-performance General Matrix Multiplication (GEMM) implementations rely on breaking down GEMM into carefully selected tiles to achieve peak performance. We present Waypoint, a new analytical model to automatically identify the optimal tile size for a given hardware and input matrix shape. Waypoint estimates the achievable performance for a given tile choice by capturing all the architectural effects including interactions with memory hierarchy, instruction-level parallelism, and multi-threading. Using the estimates as a proxy, we search through the space of possible tile sizes to identify the one that maximizes performance. Additionally, Waypoint helps hardware designers estimate realistic performance on experimental hardware optimizations. We build a new GEMM library OPAM for CPUs that uses Waypoint to automatically break down GEMM and achieve performance that matches state-of-the-art GEMM libraries and helps accelerate LLM inference on CPUs achieving $1.36 - 1.98 \times$ speedups.