Distributed Grid Search for Hyperparameter Tuning
The paper proposes two distributed methods for selecting hyperparameters in peer-to-peer networks.
The paper presents novel consensus-based search methods for distributed hyperparameter tuning, which is an improvement over existing methods that require global quantities or are tied to specific algorithms and hyperparameters.
Before reading this…
Applications
- →machine learning
- →distributed optimization
To understand this paper, make sure you know these concepts first:
- machine learningfind papers →
- distributed systemsfind papers →
Abstract
More Like ThisThe performance of many distributed optimization algorithms over peer-to-peer networks strongly depends on the choice of their hyperparameters. However, existing selection rules often require global quantities, yield conservative choices, or are tied to specific algorithms and hyperparameters. We propose Distributed Grid Search (D-GS), a consensus-based search method that evaluates and selects hyperparameter configurations across the network, requiring only the ability to run the tuned algorithm and evaluate local gradients. We also propose a zoom-in variant, Distributed Zoom Grid Search (D-ZGS), which improves empirical performance at a reduced cost. We prove that both methods terminate in finite time with all agents holding the same hyperparameter configuration, and quantify the (sub)optimality of this configuration. Numerical experiments on linear and logistic regression show that D-GS closely matches a centralized reference and that tuning and deployment cost a fraction of deploying algorithms with hyperparameters tuned using theoretical results.