[안생강 논문리뷰 03] ECA-Net: Efficient Channel Attention for Deep Convolutional Neural Networks

( 2021.04.19 updated )

 

Paper Link / GitHub Link (2020)

 

 

1. Introduction

SE-Net : squeeze-and-excitation, learns channel attention

- input > GAP for each channel independently > two FC layers with non-linearity followed by a sigmoid function ( two FC layers are designed to capture non-linear cross-channel interaction, which involve dimensionality reduction for controlling model complexity )

- researches improve SE block by capturing more sophisticated channel-wise dependencies & by combining with additional spatail attention (ex. CBAM) => the higher accuracy, the higher complexity !

 

This paper focuses on, 

- Can one learn effective channel attention in a more efficient way?

 

This paper shows,

- dimensionality reduction brings side effect on channel attention prediction, and it is inefficient and unnecessary to capture dependencies across all channels

 

CONTRIBUTIONS

- given the aggregated features obtained by GAP (Global Average Pooling), ECA generates channel weigths by performing a fast 1D convolution of size k, where k is adaptively determined via a mapping of channel dimension C.

- avoid dimensionality reduction

- model complexity ↓

 

---

2. Proposed Method (ECA-Net)

 

Revisiting Channel Attention in SE block

W1, W2 can reduce model complexity,

but destroys the direct correspondence between channel and its weight

( ∵ first projects channel features into a low-dimensional space, then maps them back )

 

 

Local Cross-Channel Interaction in ECA module

- ECA module aims at appropriately capturing local cross-channel interaction, so the coverage of intraction needs to be determined

- manual tuning via cross-validation will cost a lot of computing resources

 

it is reasonable that the coverage of interaction is proportional to channel dimension C

it is well know taht channel dimension C usually is set to power of 2

 

|t|_odd = the nearest odd number of t

 

 

 

Pytorch Code

PyTorch code of ECA module

 

---

3. Experiments

 

ECA-Net has a very large performance improvement compared to the number of parameters.