cs.LGcs.AIcs.NETheoretical
Closed-Form Steepest Descent Direction toward Flat Minima: Reducing Upper Bounds on the Loss Hessian Eigenspectrum in Neural Networks
Jun 27, 2026
This paper derives the gradient of the Wolkowicz-Styan upper bound on the maximum eigenvalue of the cross-entropy loss Hessian in three-layer NNs to characterize directions leading to flat minima and proposes Hessian Spectral Range Regularization.
First study to report a closed-form gradient that promotes convergence to flat minima without numerical approximations
Keywords
Before reading this…
Applications
To understand this paper, make sure you know these concepts first:
The flatness hypothesis suggests that flatness of the loss landscape, as measured by the eigenvalues of the loss Hessian, correlates with better neural network generalization. While various algorithms reduce these eigenvalues, most focus on procedural design, leaving it unclear how data distributions and NN parameters structurally determine directions toward flat minima. Characterizing these directions analytically is generally intractable. To overcome this mathematical difficulty, recent studies derived the Wolkowicz-Styan (WS) upper bound on the maximum eigenvalue of the cross-entropy loss Hessian in three-layer NNs. Although this upper bound is differentiable, its gradient was not derived. Therefore, we analytically derive the gradient of the WS upper bound to characterize directions leading to flat minima. Based on this, we propose Hessian Spectral Range (HSR) Regularization, which updates parameters along the steepest descent direction of the WS bound. Experiments demonstrate that HSR Regularization narrows the Hessian eigenvalue spectrum, avoids sharp minima and saddle points, and promotes convergence to flat minima. Although the applicability of this method is currently limited to cross-entropy loss and three-layer architectures, to the best of the authors' knowledge, this is the first study to report a closed-form gradient that promotes convergence to flat minima without numerical approximations. Therefore, the theoretical analysis of this gradient is expected to contribute to the further development of NNs.
How Should Agents Read Demonstrations? Hierarchical Structure Beats Flat Action Logs
This paper proposes grouping recorded actions in Programming by Demonstration (P…
Curvature-Weighted Gradient Diversity: A Noise Measure for Geometry-Adaptive SGD Schedules
This paper introduces Curvature-Weighted Gradient Diversity (CWGD), a geometry-a…
Local Minima in Quadratic-Penalty Relaxations of Binary Linear Programs
The paper establishes conditions for QUBO formulations of combinatorial optimiza…
Gradient boosting with vector-valued leafs
This paper extends gradient boosting to functions of vector inputs using a simpl…
Random Reshuffling Dominates Stochastic Gradient Descent
This paper proves that Random Reshuffling in Shuffling Stochastic Gradient Desce…
Explainable AI for Next-Generation Wireless Physical Layer: Basics, State-of-the-Art, and Open Chall…
This survey examines explainable AI (XAI) in wireless PHY layers, formalizing go…
Avoiding unsafe sets when training with Langevin Dynamics
This paper studies the probability of a trajectory lying in a designated failure…
ARES: Scalable and Practical Gradient Inversion Attack in Federated Learning through Activation Reco…
The paper introduces ARES, a novel and practical gradient inversion attack that…