Modularized Morphing Of Neural Networks

About this Publication

• Authors: Tao Wei, Changhu Wang, Chang Wen Chen (Microsoft)
• Conference: ICLR
• Years: 2017
• Paper Link: https://arxiv.org/abs/1701.03281

---

Motivation

1. inherite entire knowledge from a well-trained network to a new one
2. booster performance
3. explore new model and architecture for constitutional neural network

---

Previous works

• not practical use
  • limits the change of network parameters including width, depth and kernel size
  • allow to apply a limited layers in a CNN

---

Goal

• design the architecture allowing to apply hundreds of layers with primitive operations

---

What you would learn from this presentation

1. an idea of the architecture for knowledge inheritance in neural networks

2. some technical terminology about neural networks

---

Proposition - Network Morphism

explore a morphing space and apply changes of the basic modules to modularized networks

advantages
1. simple
2. scalability

---

[for beginners] What is CNN (Convolutional Neural Network)?

---

[for beginners] What is modularized network?

a network that is divided into a set of functional units (modules)

---

Goal of Network Morphism

When a CNN $B_i$ grows to a new CNN $B_j$, $B_i$ applies convolutional layers $G_i$.
To be specific, a convolutional layer $G_i$ is morphed into two convolutional layers $F_l$ and $F_{l+1}$.

---

core algorithm

1. create irreducible networks according to the types of morphable networks
2. create a new convolution layers only if an irreducible network impacts to all fixed layers (if needed)

---

Types of Morphable Networks

1. Simple morphable networks

   • morphed with atomic morphing operations
     • TYPE-I: a convolutional layer is morphed into two convolutional layers
     • TYPE-II: a convolutional layer is morphed into two-way convolutional layers

2. Complex morphable networks
   

---

Quiz

Classify 4 networks into simple and complex morphable networks

---

Answer [1/2]

• Simple morphable network = (A) (B) (C)
• Complex morphable network = (D)

---

Answer [2/2]

• Simple morphable network
  

• Complex morphable module
  

---

How to solve Complex Morphable Networks

1. create a irreducible network from a complex morphable network
2. apply atomic morphing operations and create a new convolutional layer for the network

---

Implementation

The authors extend the proposed morphable algorithm in ResNet

---

[for beginners] ResNet

• a winner @ ILSVRC 2015
• Residual Learning
  • conquer the proportional error rate to the number of convolutional layers

---

List of Evaluated Morphable Network

---

Dataset

• CIFAR10

  • consists of 32×32 color images in 10 categories, with 50,000 training images and 10,000 testing images.

• CIFAR100

  • consists of 500 training images in 100 categories and 100 testing images per category

• ImageNet

  • consists of 1.28 million training images in 1,000 object categories and 50K validation images

---

Experimental Results (CIFAR10)

---

Conclusion

• Network morphism is an effective learning scheme to morph a well-trained neural network to a new one with the network function completely preserved

• In benchmark, CNNs with the network morphism shows smaller error rate compared to CNNs by compromising the higher calculation cost

---

Discussions @ coffee with science meetup

---

Reference

https://adeshpande3.github.io/adeshpande3.github.io/A-Beginner%27s-Guide-To-Understanding-Convolutional-Neural-Networks/
http://cs231n.github.io/convolutional-networks/