# ResNet and Its Variations

[shortcut

activation

deep-learning

resnet

pre-activation

post-activation

]
# ResNet

Since proposed in 2016, ResNet has drawn a lot of interests, especially its capabability of training a very deep nerual network (from 19 of VGG19 to 50 or even 200 layers). The magic comes from the residual functions.

With residual function, the output of layer $l+1$ can be written as:
\(x_{l+1} = f(x_l) + x_l\)
where `f`

represents the function of $l_th$ layer.

# ResNet V2

The difference between the V1 and V2 is (as illustrated below):

- V1: Convolution then batch normalization then ReLU
- V2: Batch normalization then ReLu then convolution

The motivation behind is that V2 is much easier to train and generalizes better than the V1 ResNet

In fact, other variations has been explored as well and obviously not all variations work:

# ResNext

Aggregated Residual Transformations for Deep Neural Networks

ResNext combines ResNet with split-transform-merge strategy. A module in the network performs a set of transformations, each on a low-dimensional embedding, whose outputs are aggregated by summation.

This `Network-in-Neuron`

can be understood as the `boosting`

, which combines a set of simpler classifier into a powerful one. It also shares the same idea as group-wise convolution as MobileNet.

# DenseNet

DenseNet addresses the gradient vanish problem in a different way compared with ResNet: ResNet using `sum`

to combine the output of previous layer and output of current layer as the input of next layer; DenseNet using `concatenation`

to combine the outputs of different layers. In a dense block of DenseNet, the input of layer is the concatnation of outputs of **ALL** previous layers.

It alleviates the vanishing-gradient problem, strengthen feature propagation, encourage fea- ture reuse, and substantially reduce the number of parameters. The reason for less parameters is that, with dense net narrower filters can be used (less output channels).

Since the number of input channels increase quadratically with regards to layer within the dense block, a `bottleneck`

layer is introduced to reduce the number of feature channels, which is essentially convolution layer with $1\times 1$ kernel size and less output channels than input ones.