Spatial Transformer Networks (STN) are a type of neural network that focus on important regions in images by learning invariance to different types of transformations, such as translation, scaling, and rotation. By explicitly predicting and paying attention to these regions, STNs provide a deep neural network with the necessary transformation invariance.
What is an Affine Transformation?
To understand how STNs work, we must first take a look at affine transformations. An affine transformation is a type of transformation that preserves parallel lines, meaning that if two lines are parallel to each other in the original image, they will remain parallel after the transformation. An affine transformation can be represented by a $2\times 3$ matrix known as an affine matrix. This matrix is learnable and can be adjusted during training.
How Do STNs Work?
STNs use a learnable affine matrix to transform an input feature map. The affine matrix is calculated using a differentiable function, such as a lightweight fully-connected network or a convolutional neural network. Once the affine matrix is calculated, the network can use it to find the correspondence between the input feature map and the output feature map.
After obtaining the correspondence, the network uses bilinear sampling to sample the input features. Bilinear sampling is a way to sample input features that is differentiable and allows for updates to be made to the network in an end-to-end manner. By using bilinear sampling, STNs are able to focus on discriminative regions automatically and learn invariance to some geometric transformations.
Why Are STNs Important?
STNs are important because they allow neural networks to learn invariance to geometric transformations. This means that a neural network using an STN can recognize objects in images, even if the objects are translated or rotated. This opens up a world of possibilities for computer vision applications, including object recognition, image retrieval, and scene reconstruction.
STNs have also been shown to be effective in other types of deep learning applications, such as natural language processing, where they have been used to align and transform word embeddings.
Spatial Transformer Networks are a powerful tool in deep learning that allow neural networks to learn invariance to geometric transformations. By focusing on discriminative regions and using bilinear sampling to sample input features, STNs provide a deep neural network with the necessary transformation invariance to recognize objects in images, even if they are translated or rotated. STNs have a wide range of applications in computer vision and other deep learning fields.
