The Energy Consumption of Transformer Fine-Tuning: A Roofline-Inspired Scaling Model
The paper presents a framework to model and predict energy consumption of Transformer training on multiple GPUs.
The paper introduces a new framework for predicting energy consumption of Transformer training on multiple GPUs, which is a novel approach compared to prior work.
Before reading this…
Applications
- →Sustainable and cost-aware system design
To understand this paper, make sure you know these concepts first:
- Knowledge of Transformer models and GPU architecturefind papers →
Abstract
More Like ThisTransformer-based models underpin modern natural language processing but incur rapidly growing computational and energy costs. As training scales in both model size and parallelism, accurately predicting energy consumption has become critical for sustainable and cost-aware system design. We present a framework for modeling the energy consumption of Transformer training on multiple GPUs. Using controlled architectural sweeps of BERT models, we relate measured energy to lightweight proxies for compute, memory traffic, and hardware efficiency. Inspired by roofline models, our approach incorporates a speedup-based hardware-efficiency factor that captures the effects of tensor parallelism and fully sharded data parallelism. We derive a scaling law model that accurately predicts training energy across heterogeneous configurations.