Filter Response Normalization (FRN) is a technique for normalizing and activating neural networks. It can be used in place of other types of normalization and activation for more effective machine learning. One of the key benefits of FRN is that it operates independently on each activation channel of each batch element, which eliminates dependency on other batch elements.
How FRN Works
When dealing with a feed-forward convolutional neural network, the activation maps produced after a convolution operation are arranged in a 4D tensor with shape [B, W, H, C], where B is the mini-batch size, W and H are the spatial dimensions of the map, and C is the number of filters used in convolution. This is also referred to as output channels.
Each activation channel of each batch element is operated on independently, and the mean squared norm of each vector of filter responses is calculated using the following formula:
$\nu^2 = \sum\_i x_i^2/N$
where N is the number of elements in the vector. Next, filter response normalization is applied by dividing each vector of filter responses by the square root of the mean squared norm plus a small positive constant, epsilon:
$\hat{x} = \frac{x}{\sqrt{\nu^2 + \epsilon}}$
This helps to prevent division by zero and normalize the data in a way that is beneficial for training a neural network.
Improving Activation with FRN
One issue with FRN is that it lacks mean centering, meaning that activations may have a bias away from zero. When combined with ReLU activation, this bias can have a negative impact on learning, leading to poor performance or dead units. To resolve this issue, the authors of the FRN paper propose augmenting ReLU with a learned threshold, tau:
$z = \max(y, \tau)$
This activation function has the same effect as having a shared bias before and after ReLU.
Benefits of FRN
FRN offers several benefits over other normalization and activation techniques used in machine learning. By normalizing the activation channels of each batch element independently, it eliminates dependency on other batch elements, which can improve accuracy and performance. Additionally, by combining normalization and activation into one step, FRN can simplify model architectures and reduce computational overhead.
Another benefit of FRN is its ability to prevent overfitting, which occurs when a model is too complex and learns to fit the training data too closely, resulting in poor performance on new data. By normalizing and activating neural networks using FRN, researchers can minimize the impact of individual batch elements on the model, making it less likely to overfit and more effective for real-world applications.
Filter Response Normalization is a powerful technique for normalizing and activating neural networks in machine learning. By operating independently on each activation channel of each batch element, FRN eliminates dependency and can improve the accuracy and performance of models. Additionally, by combining normalization and activation into one step, FRN can simplify model architectures and reduce computational overhead. With its ability to prevent overfitting and improve real-world effectiveness, FRN is a valuable tool for researchers and engineers working with machine learning.
