Symbolic Deep Learning: An Overview
Symbolic deep learning is a technique that involves converting a neural network into an analytic equation. This general approach allows for a better understanding of the neural network's learned representations and has applications in discovering novel physical principles.
The Technique
The technique used in symbolic deep learning involves three steps:
1. Encourage sparse latent representations
Sparse latent representations refer to the idea that the neural network's learned representations should be simplified, reducing the complexity of the equation. This is important in symbolic regression, which is used in the next step.
2. Apply symbolic regression
In this step, symbolic regression is used to approximate the transformations between the input, latent, and output layers of the neural network. Symbolic regression is a machine learning technique that involves fitting an equation to data using symbolic modeling. This technique helps to identify patterns and relationships between the input and output variables.
3. Compose symbolic expressions
Finally, the symbolic expressions obtained from symbolic regression are composed into a single analytic equation. This equation can be used to represent the learned representation of the neural network and can be interpreted for insights into the neural network's process.
Applications
Symbolic deep learning has various applications in discovering new physical principles, including force laws and Hamiltonians, that can be extracted from the neural network. For example, in a recent paper, researchers used the technique on a detailed dark matter simulation and discovered a new analytic formula that can predict the concentration of dark matter from the mass distribution of nearby cosmic structures. The symbolic expressions also generalized to out-of-distribution data better than the neural network itself.
Symbolic deep learning offers an alternative direction for interpreting neural networks and understanding the representations they learn. The technique allows for the discovery of novel physical principles and insights into the neural network's process, making it an increasingly valuable tool in the field of machine learning.