The Bottleneck Attention Module (BAM): A Powerful Tool for Improving Neural Network Performance
The bottleneck attention module (BAM) is a neural network module that is used to enhance the representational capabilities of existing networks. It is designed to efficiently capture both channel and spatial information in a network to improve its performance. The module achieves this by utilizing both a dilated convolution and a bottleneck structure to save computational cost, while still providing an effective method for improving network performance.
How Does BAM Improve Network Performance?
When implemented in a network, BAM first infers channel attention and spatial attention in parallel streams, before summing both attention maps together. This process allows BAM to emphasize or suppress features in both the spatial and channel dimensions, improving the representational power of the network. To achieve this, BAM applies dimensional reduction to both the spatial and channel attention branches, which enables it to be integrated with any convolutional neural network while adding little extra computational cost.
Channel Attention Branch
The channel attention branch infers the channel attention $s_c \in \mathbb{R}^C$. Like an SE block, it applies global average pooling to the feature map, which aggregates global information. The output is then passed through an MLP with channel dimensionality reduction. The channel attention branch is responsible for identifying important channels within the input feature map.
Spatial Attention Branch
The spatial attention branch infers the spatial attention $s_s\in \mathbb{R}^{H\times W}$. It combines a bottleneck structure and dilated convolutions, which help utilize contextual information effectively. The dilated convolutions enlarge the receptive field of the spatial attention sub-module, while the bottleneck structure helps save computational cost. Unlike the channel attention branch, the spatial attention branch captures contextual information in the spatial dimension.
How is BAM Implemented?
Overall, BAM can be written as:
s_c = BN(W_2(W_1GAP(X)+b_1) + b_2)
s_s = BN(Conv_2^{1 \times 1}(DC_2^{3\times 3}(DC_1^{3 \times 3}(Conv_1^{1 \times 1}(X))))
s = $\sigma$ (Expand(s_s) + Expand(s_c))
Y = sX + X
Where $W_i$, $b_i$ signify weights and biases of fully connected layers, $Conv_{1}^{1\times 1}$ and $Conv_{2}^{1\times 1}$ are convolution layers used for channel reduction. $DC_i^{3\times 3}$ indicates a dilated convolution of size $3\times 3$ and $\text{Expand}$ expands the attention maps $s_s$ and $s_c$ to $\mathbb{R}^{C\times H\times W}$. The output of BAM, Y, is the residual of the input feature map X and the amplified version of the input feature map X with attention maps applied to it. Using this residual method, Y provides a more significant impact on the loss gradient to update the network weights.
Limitations of BAM
Although dilated convolutions enlarge the receptive field effectively, they still fail to capture long-range contextual information and cannot encode cross-domain relationships as well as some other models. While BAM is a powerful module for enhancing representational capability, it cannot add features that are not consequently provided in the input vector. In such cases, alternative models that integrate well with BAM can be used.
Conclusion
Overall, the bottleneck attention module (BAM) is a valuable tool for improving the performance of neural networks. BAM's unique design efficiently captures both channel and spatial information in a network. Utilizing a dilated convolution and a bottleneck structure, BAM saves computational cost but provides an effective method for improving network performance. Although not perfect, BAM provides a powerful way to increase network expressiveness, thus learning more meaningful features which leads to better predictions.
