Global Contextual Transformer (GCT) is a type of feature normalization method that is applied after each convolutional layer in a Convolutional Neural Network (CNN). This technique has been widely used in many different image recognition applications with a great level of success.
GCT Methodology
In typical normalization methods such as Batch Normalization, each channel is normalized independently, which can cause inconsistencies in the learned levels of node activations. GCT is different in the sense that it first computes the l2-norm of each individual channel to collect global information. Afterwards, it applies a learnable vector to scale the feature, and then uses a competition mechanism with channel normalization to interact between the channels. Normalization via rescaling is common in many normalization methods, but GCT differs by using a tanh activation function to control the attention vector. Finally, GCT multiplies the input by the attention vector and adds an identity connection. The entire process can be expressed as follows:
s = Fgct(X, θ) = tanh(γCN(αNorm(X)) + β)
Here, α, β and γ are trainable parameters, Norm indicates the L2-norm of each channel whereas CN represents channel normalization. The final output is calculated using Y = sX + X.
Key Advantages of GCT
The biggest advantage of GCT over other feature normalization methods lies in its lightweight design. The use of the l2-norm of each individual channel to collect global information reduces the number of parameters without sacrificing accuracy. It is a critical advantage, especially in large datasets since computation time and speed are major factors in real-time application settings.
Another benefit of GCT is its ability to compete between channels. This helps the CNN to learn from the whole image, which is different from normalization methods, which normalize each channel independently. GCT allows the model to look at the image from a holistic perspective to find important information that would have been lost otherwise. This feature can help in improving the accuracy of CNN.
How GCT Can Be Used
GCT can be easily implemented in any CNN architecture as it is lightweight and easy to train due to its fewer learnable parameters. It can be used after each convolutional layer to improve the accuracy of the predictive model as it reduces overfitting by being able to converge to a global minimum of the loss function. For Convolutional networks aimed at object recognition, GCT seems to be a promising normalization method that can improve the accuracy of the neural network.
GCT is a feature normalization method that has been widely used in image recognition applications. It differs from previous normalization methods and provided better accuracy with fewer learnable parameters. It's lightweight design is a critical advantage in large dataset models, and its use of channel interaction helps the CNN to learn from the whole image. It can be easily applied to a range of CNN architectures with promising results for object recognition.
