Modern Convolution Neutral Network

[ alexnet  mobilenet  cnn  googlenet  densenet  efficientnet  xception  deep-learning  resnext  resnet  regnet  vgg  ]

Convolutional Neural Networks (ConvNets or CNNs) are a class of neural networks algorithms that are mostly used in visual recognition tasks such as image classification, object detection, and image segmentation. The use of ConvNets in visual recognition is inarguably one of the biggest inventions of decade 2010s in deep learning community.

This post was from

author: Jean de Dieu Nyandwi
title: Modern Convolutional Neural Network Architectures
year: 2022
publisher: GitHub
url: https://github.com/Nyandwi/ModernConvNets

Render nbviewer Open In Colab

Revision of the designs and implementation of Modern Convolutional Neural Network architectures

cnns_image

Introduction to Convolutional Neural Networks

Convolutional Neural Networks (ConvNets or CNNs) are a class of neural networks algorithms that are mostly used in visual recognition tasks such as image classification, object detection, and image segmentation. The use of ConvNets in visual recognition is inarguably one of the biggest inventions of decade 2010s in deep learning community.

A standard ConvNet architecture is typically made of 3 main layers that are convolution, max-pooling, and fully connected layers. Convolution layers are the main component of ConvNets. They are used for extracting features in images using filters.

Pooling layers are used for downsampling the activation or feature maps produced by convolutional layers. Downsampling can also be achieved by using strides(greater than 1) in a normal convolution layer, but max-pooling layers don’t have any learnable parameters and they introduce translational invariance which improves model generalization on the cost of spatial inductive bias. Fully connected layers are used for classification purpose(matching learned features with their respective labels). In classification settings, the last fully connected layer is typically activated with softmax activation function!

Example of ConvNets architectures that follow the above structure are AlexNet and VGG. Most modern ConvNet architectures go beyond plain stack of convolution, max-pooling and fully connected layers. For example, architectures like ResNet and other alike networks involve residual connections.

As you go through the materials in this repository, you will learn more about those architectures and how they are implemented. For more about ConvNets, check resources section!

ConvNet Architectures

Choosing a ConvNet Architecture

Computer vision community is blessed with having many vision architectures that work great across many platforms or hardwares. But, having many options means it is not easy to choose an architecture that suits a given problem. How can you choose a ConvNet architecture for your problem?

The first rule of thumb is that you should not try to design your own architecture from scratch. If you are working on generic problem, it never hurts to start with ResNet-50. If you are building a mobile-based visual application where there is limited computation resources, try MobileNets(or other mobile friendly architectures like ShuffleNetv2 or ESPNetv2).

For a better trade-off between accuracy and computation efficiency, try EfficientNet or latest ConvNeXt!

That said, choosing architecture(or learning algorithm) is a no free-lunch scenario. There is no universal architecture. There is no single architecture that is guaranteed to work for all datasets and problems. It’s all experimentation. It’s all trying!

If you are a visionary or like to stay on the bleeding edge of the field, take a look at vision transformers! We don’t know yet, but they might be the successor of ConvNets!

Important Notes

The implementations of ConvNets architectures contained in this repository are not optimized for training but rather to understand how those networks were designed, principal components that makes them and how they evolved overtime. LeNet-5(LeCunn, 1998) had 5 layers(with learnable weights, pooling layers excluded). AlexNet(Krizhevsky et al., 2012) had 8 layers. Few years later, Residual Networks(He et al., 2015) made the trends after showing that it’s possible to train networks of over 100 layers. Today, residual networks are still one of the most widely used architecture across a wide range of visual tasks and they impacted the design of language architectures.

Computer vision research community is very vibrant. With the intelligent frameworks and better architectures we have to day, understanding how networks architectures are designed before you can throw them in your dataset is never a neccesity, but it’s one of the best ways to stay on top of this vibrant and fast-ever changing field!

If you want to use ConvNets for solving a visual recognition tasks such as image classification or object detection, you can get up running quickly by getting the models (and their pretrained weights) from tools like Keras, TensorFlow Hub, PyTorch Vision, Timm PyTorch Image Models, GluonCV, and OpenMML Lab.

References Implementations

Further Learning

If you would like to learn more about ConvNets/CNNs, below are some few amazing resources:

Citation

If you use any material in this repository, you are welcome to cite it with:

author: Jean de Dieu Nyandwi
title: Modern Convolutional Neural Network Architectures
year: 2022
publisher: GitHub
url: https://github.com/Nyandwi/ModernConvNets
Written on March 20, 2022