A Framework for Accelerating Transformer Inference on RISC-V for Edge AI
This paper presents a framework for accelerating transformer-based language models on resource-constrained IoT devices.
This work introduces a novel framework for accelerating transformer-based language models on resource-constrained IoT devices, which is not addressed in prior work.
Keywords
Before reading this…
Applications
- →Edge inference scenarios
- →IoT devices
To understand this paper, make sure you know these concepts first:
- Knowledge of language models and their limitations on resource-constrained devicesfind papers →
- Understanding of instruction set extensions and compiler-directed loop unrollingfind papers →
Abstract
More Like ThisThis work presents a framework for accelerating transformer-based language models (LMs) on resource-constrained IoT devices. The framework targets compact LMs: BERT-Tiny (B-Ty), MobileBERT (M-Bt), MiniLM (M-Lm), Electra (E-Lt) and DeBERTa (D-Bt) -- selected for their architectural diversity and use in edge inference scenarios. The proposed flow derives lightweight instruction set extensions tailored to the non-obvious computational patterns of these models. In addition, a custom instruction is introduced to accelerate the address generation stage of batch matrix multiplication, achieving a performance improvement of 15.39--21.74% with modest ASIC overheads of 6.79% in area and 2.33% in power. To further enhance performance without incurring additional processor core hardware cost, an optional compiler-directed loop unrolling strategy is employed, trading increased code size for overall reduced execution time. Evaluation on the Synopsys trv32p3f RISC-V core demonstrates inference speedups of up to 2.19x, and FPGA implementation on the AMD Zynq UltraScale+ ZCU102 shows a 32.94% area overhead at 75 MHz, whereas the ASIC implementation using the TSMC 28 nm library incurs a 21.08% area overhead while operating at 250 MHz.