Back to Paper
cs.LGcs.AIcs.CR
Local ID: 2605.06505v2
AI Summary: gemma4:e4b
PACZero: PAC-Private Fine-Tuning of Language Models via Sign Quantization
By Murat Bilgehan Ertan, Xiaochen Zhu, Phuong Ha Nguyen, Marten van Dijk, Srinivas Devadas
Revision History Timeline
v15/7/2026
5/7/2026
No submitter comment provided.
v25/24/2026
5/24/2026
No submitter comment provided.
★ Version indexed in ExplorerComparing v1 vs v2
Green = Added • Red = Removed
Title Comparison
PACZero: PAC-Private Fine-Tuning of Language Models via Sign Quantization
Authors Comparison
No author changes.
v1 Comment
No comment for this version.
v2 Comment
No comment for this version.
Abstract Word Diff
We introduce PACZero, a family of PAC-private zeroth-order mechanisms for fine-tuning large language models that delivers usable utility at $I(S^*; Y_{1:T})=0$. This privacy regime bounds the membership-inference attack (MIA) posterior success rate at the prior, an MIA-resistance level the DP framework matches only at $\varepsilon=0$ and infinite noise. All DP-ZO comparisons below are matched at the MIA posterior level. The key insight is that PAC Privacy charges mutual information only when the release depends on which candidate subset is the secret. Sign-quantizing subset-aggregated zeroth-order gradients creates frequent unanimity, steps at which every candidate subset agrees on the update direction; at these steps the released sign costs zero conditional mutual information. We propose two variants that span the privacy-utility trade-off: PACZero-MI (budgeted MI via exact calibration on the binary release) and PACZero-ZPL ($I=0$ via a uniform coin flip on disagreement steps). We evaluate on SST-2 and SQuAD with OPT-1.3B and OPT-6.7B in both LoRA and full-parameter tracks. On SST-2 OPT-1.3B full fine-tuning at $I=0$, PACZero-ZPL reaches ${88.99\pm0.91}$, within $2.1$pp of the non-private MeZO baseline ($91.1$ FT). No prior method produces usable utility in the high-privacy regime $\varepsilon<1$, and PACZero-ZPL obtains competitive SST-2 accuracy and nontrivial SQuAD F1 across OPT-1.3B and OPT-6.7B at $I=0$.