This paper presents FlexServe, a fast and secure LLM inference system for mobile devices that addresses the challenges of protecting LLM inference with ARM TrustZone.
FlexServe addresses the challenges of protecting LLM inference with ARM TrustZone by decoupling access permission from management permission of secure resources.
Before reading this…
Applications
To understand this paper, make sure you know these concepts first:
Device-side Large Language Models (LLMs) have grown explosively, offering stronger privacy and higher availability than their cloud-side counterparts. During LLM inference, both the model weights and the user data are valuable, and attackers may compromise the OS kernel to steal them. ARM TrustZone is the de facto hardware-based isolation technology on mobile devices, used to protect sensitive applications from a compromised OS. However, protecting LLM inference with TrustZone incurs significant overhead to both the secure inference and the normal aplications, due to two challenges: the inflexible resource isolation and the inefficient secure resource management. To address these challenges, this paper presents FlexServe, a fast and secure LLM inference system for mobile devices. The key idea is to decouple the access permission from the management permission of secure resources, so that the normal-world OS cannot access them but can still manage them as usual. First, FlexServe introduces a Recallable Resource Isolation mechanism to construct Recallable Secure Memory (Flex-Mem) and a Recallable Secure NPU (Flex-NPU). They can only be accessed by the secure world, but can be efficiently allocated and reclaimed by the normal-world OS. Based on them, FlexServe further introduces a FlexServe Framework to run secure LLM inference in the secure world. It works together with the normal-world OS to perform cooperative secure memory management. We implement a prototype of FlexServe and compare it with two TrustZone-based strawman designs. The results show that FlexServe achieves average TTFT speedups of 10.05X over the strawman and 2.44X over an optimized strawman.
Constructions of Secure Convertible Codes
This paper constructs two families of access-optimal secure convertible codes.
BitLLM: Efficient Low-Bit, Conversion-Hiding LLM Decoding on Mobile GPUs
The paper presents BitLLM, a mobile inference engine that improves the performan…
Blame-aware Recomposition for Secure Compilation Proofs of Low-Level Compartment Isolation
The paper proposes an extension to a proof technique for verifying compartment i…
Unsupervised Head Network Personalization for Split Computing via Federated Learning
This paper proposes a data-efficient head network training method for Split Comp…
TurboWAM: Learning What Matters for Fast World Action Models
This paper proposes TurboWAM, a fast world action model that efficiently aggrega…
Interflow: Breaking the Offline-Online Barrier for Private AI at the Edge
This paper proposes Interflow, an algorithm-hardware co-design that improves the…
AgentLens: Framework-Aware Characterization and Resource Control for Heterogeneous LLM Agents
Combines cross-framework workload characterization with framework-aware resource…
Agent Memory as Recommendation: Recommender-System Design for Long-Term Memory of LLM Agents
The paper proposes AgentMemRec, a method that adapts recommender systems to sele…