Conservation Laws from Data Symmetry in Neural Networks
This paper investigates whether intrinsic symmetries in training data lead to conserved quantities during gradient-flow training of neural networks.
The paper introduces the concept of tensorizable networks to describe the phenomenon of extra conserved quantities for mean squared error loss.
Before reading this…
To understand this paper, make sure you know these concepts first:
- Neural networksfind papers →
- Symmetriesfind papers →
Abstract
More Like ThisWe explore whether intrinsic symmetries of the training data lead to conserved quantities during gradient-flow training of neural networks. Under the assumption that the loss function is analytic and non-polynomial, we prove that data symmetries generically do not induce any additional integrals of motion. For mean squared error (MSE) loss, on the other hand, there are situations in which data augmentation yields extra conserved quantities. We build a framework, utilizing \emph{tensorizable networks} to describe this phenomenon. Tensorizable networks are a family of architectures whose dependence on parameters and inputs can be separated using an intermediate representation. They include linear and polynomial networks, as well as Lightning Attention.