Assignment 2

Instructions

The goal of this assignment is threefold: (i) train a CNN model from scratch and learn how to tune the hyperparameters and visualise filters (ii) finetune a pre-trained model just as you would do in many real world applications (iii) use an existing pre-trained model for a cool application.
We strongly recommend that you work on this assignment in a team of size 2. Both the members of the team are expected to work together (in a subsequent viva both members will be expected to answer questions, explain the code, etc).
Collaborations and discussions with other groups are strictly prohibited.
You must use Python (numpy and pandas) for your implementation.
You can use any and all packages from keras, pytorch, tensorflow
You can run the code in a jupyter notebook on colab by enabling GPUs.
You have to generate the report in the same format as shown below using wandb.ai. You can start by cloning this report using the clone option above. Most of the plots that we have asked for below can be (automatically) generated using the apis provided by wandb.ai
You also need to provide a link to your github code as shown below. Follow good software engineering practices and set up a github repo for the project on Day 1. Please do not write all code on your local machine and push everything to github on the last day. The commits in github should reflect how the code has evolved during the course of the assignment.
You have to check moodle regularly for updates regarding the assignment.

Problem Statement

Part A: Training from scratch

Question 1 (5 Marks)

model = Sequential()
model.add(Conv2D(16, (3, 3), input_shape=input_shape))
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))

Answer

Answer

we are taking input size as H x W x 3
Kernel size is k*k and each layer has m kernels
Max pooling size is also k
n neurons in Dense layer
Output layer has 10 neurons

Answer

Question 2 (10 Marks)

number of filters in each layer : 32, 64, ...
filter organisation: same number of filters in all layer, doubling in each subsequent layer, halving in each subsequent layer, etc
data augmentation (easy to do in keras): Yes, No
dropout: 20%, 30% (btw, where will you add dropout? you should read up a bit on this)
batch normalisation: Yes, No

accuracy v/s created plot (I would like to see the number of experiments you ran to get the best configuration).
parallel co-ordinates plot
correlation summary table (to see the correlation of each hyperparameter with the loss/accuracy)

Answer:

Validation Accuracy v/s created plot and Training Accuracy v/s created plots are shown below that plots the validation and training accuracy achieved for each experiment performed according to its creation date. Validation accuracy achieved with the given dataset after splitting the training data into 90-10 train-validation split is around 42% and training accuracy achieved more than 50%
Parallel coordinates plot is attached just below the Accuracy v/s created plot that shows all the hyperparameters corresponding to its accuracy.
Correlation summary table is plotted below parallel coordinates plot that shows a summary of correlation of different hyperparameters with respect to the validation accuracy.
Training loss, Validation Accuracy and Training Accuracy are also plotted below for different experiments.

Different filter sizes :
Different ways of organizing filters across layers:
Data augmentation:
Dropout:
Batch normalisation:
Different number of epochs:
Different pool kernel size:
Different pool strides:
Different dense layer size:
Different learning rates:
Different activation functions:
Different Batch sizes:
Weight decay:
Different optimizers:

#This is the config file for performing sweep in wandb
sweep_config = {
  'name': 'Assignment2_PartA_Q2',
  'method': 'bayes',
  'metric': {
      'name': 'Validation Accuracy',
      'goal': 'maximize'   
    },
  'parameters': {
      'epochs': {
            'values': [5,10,20,30,40,50]
        },
        'conv_attributes_channels': {
            'values': [[32,64,32,64,32],[32,32,32,32,32],[16,32,64,128,256],[32,64,128,256,512],[256,128,64,32,16],[64,64,64,64,64],[64,128,256,512,1024]]
        },
        'conv_attributes_kernel_size': {
            'values': [[3,3,5,7,9],[7,5,5,3,3],[11,7,5,3,3],[3,3,3,5,5],[3,3,3,3,3],[11,7,7,5,3],[11,9,7,5,3],[3,5,7,9,11]]
        },
        'pool_attributes_kernel_size': {
            'values': [[2,2,2,2,2],[2,2,2,1,1],[2,1,3,1,2],[3,3,3,2,2]]
        },
        'pool_attributes_stride': {
            'values': [[2,2,2,2,2],[2,2,2,1,1],[1,1,2,2,2],[1,2,1,2,1],[2,2,2,2,1]]
        },
        'dense_layer_size': {
            'values': [32,64,128,256,512]
        },
        'learning_rate': {
            'values': [0.001,0.002,0.0015,0.0001,0.00015, 0.00001]
        },
        'activation': {
            'values': ['relu','elu','gelu','sigmoid']
        },
        'dropout': {
            'values': [0.0 ,0.2 ,0.3 ,0.4 ,0.5]
        },
        'batch_normalization': {
            'values': [False,True]
        },
        'batch_size': {
            'values': [16, 32, 64,128]
        },
        'weight_decay': {
            'values': [0.0,0.00001,0.0001]
        },
        'dataset_augmentation':{
              'values': [False,True]
        },
        'optimizer_name':{
              'values': ['Adam','SGD']
        }
    }
}

Bayes search strategy: Wandb provides with different search strategies to choose between the hyperparameters such as 1) Random search, 2) Grid Search and 3) Bayes Search. We choose Bayes search strategy which uses gaussian function to model the function and then chooses parameter to optimize probability of improvement. Here we are optimizing (maximizing) over the validation accuracy.
Data Augmentation and Batch Normalization: Initially we observed that with batch_normalization and dataset_augmentation true it is giving better result than with false, so to reduce the number of experiments we set these two parameters to 'True' for later experiments and obtained better accuracy.
Weight Decay : We also used weight decay as a hyperparameter to avoid complex models forming.
Resize Image : We resized the image to 256x256 to reduce the experiment time as the processing speed will decrease for each image.

Question 3 (15 Marks)

adding more filters in the initial layers is better
Using bigger filters in initial layers and smaller filters in latter layers is better
..

Answer:

Some insightful observations based on the above plots:

Comments on how increasing/decreasing the size of the filters within or across layers helps/does not help Different filter sizes : As in the sweep hyperparameter conv_attributes_kernel_size we used different filter size like 'values': [[3,3,5,7,9],[7,5,5,3,3],[11,7,5,3,3],[3,3,3,5,5],[3,3,3,3,3],[11,7,7,5,3],[11,9,7,5,3],[3,5,7,9,11]]. So [3,3,5,7,9] means 3x3 size filter in the 1st layer followed by 3x3 in 2nd layer, 5x5 in the 3rd layer, 7x7 in 4th and 9x9 in the 5th layer. Similarly we tried with increasing the filter size in each layer as in [3,3,5,7,9] or decreasing like in [7,5,5,3,3] or even keeping all filter sizes same like in [3,3,3,3,3]. As in the parallel coordinate plot we observed that the increasing filter size like [3,3,5,7,9] is giving better accuracy than decreasing the filter size in later layers like [11,7,7,5,3]. Even keeping filter sizes same in each layer like [3,3,3,3,3] is giving better accuracy than decreasing the filter sizes in subsequent layers.
Comments on how increasing/decreasing/not changing the number of filters across layers helps/does not help: Different ways of organizing filters across layers: As in the sweep hyperparameter conv_attributes_channels we organized the number of filters in many ways like keeping more number of filters initially and gradually decreasing or keeping less number of filters and increasing it or keeping it same. We sweeped over 'values': [[32,64,32,64,32],[32,32,32,32,32],[16,32,64,128,256],[32,64,128,256,512],[256,128,64,32,16],[64,64,64,64,64],[64,128,256,512,1024]]. We got our best test result when we used more number of filters in the initial layers and gradually decreased it like [256,128,64,32,16], even when we applied same number of filters [32,32,32,32,32] we got good validation accuracy. Whereas applying less filters in initial layers and gradually increasing the number of filters like [64,128,256,512,1024] is giving less accuracy.
Comments on whether data augmentation helped or not: Data augmentation: We ran sweep with both dataset_augmentation as 'True' and 'False'. We got better accuracy for cases where dataset_augmentation was 'True' than when we didn't do dataset_augmentation. So, dataset_augmentation plays a vital contribution in accuracy also as from the correlation plot it is clear than it is positively related to the validation accuracy.
Comments on whether dropout helps or not: Dropout: As found from the articles it is better to use dropouts in the dense layers than in convolution layers. So, we used different dropout values in the dense layer like 'values': [0.0 ,0.2 ,0.3 ,0.4 ,0.5]. So, it is found that the accuracy increases when we use dropouts in dense layers. Our best model is trained with dropout taking 0.3 . From the correlation table it is evident that dropout is positively correlated with the accuracy.
Comments on whether batch normalization helps or not : Batch normalisation: We ran sweep with both batch_normalization as 'True' and 'False'. We got better accuracy for cases where batch_normalization was 'True' than when we didn't do batch_normalization. So, batch_normalization plays a vital contribution in accuracy also as from the correlation plot it is clear than it is positively related to the validation accuracy.
Different number of epochs: We tried with different epochs like 'values': [5,10,20,30,40,50] and found that number of epoch plays an important role in accuracy. If we use small epochs like 5 or 10 then the model is less trained and doesn't give good accuracy and with too much number of epochs its overfitted like in 50 and accuracy starts decreasing. So, epochs like 30, 40 are good for this model. Our best model is trained with 30 epochs.
Different pool kernel size: We used different pool_attributes_kernel_size 'values': [[2,2,2,2,2],[2,2,2,1,1],[2,1,3,1,2],[3,3,3,2,2]] and found best model with kernel sizes [3,3,3,2,2]
Different pool strides: We used different pool_attributes_stride 'values': [[2,2,2,2,2],[2,2,2,1,1],[1,1,2,2,2],[1,2,1,2,1],[2,2,2,2,1]]. We got the best accuracy with pool strides [2,2,2,2,1].
Comments on the size of the dense layer: Different dense layer size: We tried with different dense_layer_size like 'values': [32,64,128,256,512] and found that it plays an important role in accuracy. We got best accuracy with dense layer size as 128.
Comments on the use of different learning rates: Different learning rates: We ran sweep with learning_rate 'values': [0.001,0.002,0.0015,0.0001,0.00015, 0.00001] . It is observed that lower learning rate is giving better accuracy like 0.0001 or 0.00001 is giving better accuracy than that of 0.001. Our best model is trained with learning rate 0.0001.
Different activation functions: As in the sweep we use different activation functions activation : {'values': ['sigmoid','relu','elu','gelu']}. From the parallel coordinate plots, it is evident that the activation functions like 'gelu', 'relu' and 'elu' are giving far better results than 'sigmoid'. In fact we got our highest test accuracy of 42% with 'gelu' activation function. Moreover from the Correlation summary table it is evident that sigmoid activation function is highly negatively correlated with the validation accuracy.
Comments on the use of different batch sizes: Different Batch sizes: With different batch_size the accuracy also varies. So we tried with different batch sizes 'values': [16, 32, 64,128]. Our best model is trained with batch size of 16. As from the correlation table batch size highly important and it is negatively correlated with the accuracy so small sizes gave better accuracy.
Weight decay: We tried with different weight_decay 'values': [0.0,0.00001,0.0001] and we got best accuracy at 0.00001
Comments on the use of different learning algorithms: Different optimizers: We tried with two optimizer_name in the sweep 'values': ['Adam','SGD']. 'Adam' gave better accuracy that 'SGD' and our best model is trained with 'Adam'.

Question 4 (5 Marks)

Answer

wandb: Agent Starting Run: aiu1r4ra with config:
wandb: 	activation: gelu
wandb: 	batch_normalization: True
wandb: 	batch_size: 16
wandb: 	conv_attributes_channels: [256, 128, 64, 32, 16]
wandb: 	conv_attributes_kernel_size: [3, 3, 5, 7, 9]
wandb: 	dataset_augmentation: True
wandb: 	dense_layer_size: 128
wandb: 	dropout: 0.3
wandb: 	epochs: 30
wandb: 	learning_rate: 0.0001
wandb: 	optimizer_name: Adam
wandb: 	pool_attributes_kernel_size: [3, 3, 3, 2, 2]
wandb: 	pool_attributes_stride: [2, 2, 2, 2, 1]
wandb: 	weight_decay: 1e-05

Accuracy of the network: 42.35 %
Accuracy of Fungi: 46.5 %
Accuracy of Insecta: 41.0 %
Accuracy of Aves: 52.0 %
Accuracy of Mammalia: 53.5 %
Accuracy of Mollusca: 27.0 %
Accuracy of Animalia: 35.5 %
Accuracy of Arachnida: 44.5 %
Accuracy of Reptilia: 44.0 %
Accuracy of Plantae: 63.0 %
Accuracy of Amphibia: 16.5 %

Answer:

True vs Pred on 10x3 grid test images

Answer

Visualization of the 1st layer filters

Visualization of the 1st layer filters on a random image

Question 5 (10 Marks)

Answer



Question 6 (10 Marks)

We will check for coding style, clarity in using functions and a README file with clear instructions on training and evaluating the model (the 10 marks will be based on this).
We will also run a plagiarism check to ensure that the code is not copied (0 marks in the assignment if we find that the code is plagiarised).
We will check the number of commits made by the two team members and then give marks accordingly. For example, if we see 70% of the commits were made by one team member then that member will get more marks in the assignment (note that this contribution will decide the marks split for the entire assignment and not just this question).
We will also check if the training and test data has been split properly and randomly. You will get 0 marks on the assignment if we find any cheating (e.g., adding test data to training data) to get higher accuracy.

Answer

Part B : Fine-tuning a pre-trained model

Question 1 (5 Marks)

Answer

Answer

Question 2 (5 Marks)

Answer

Pre_trained model
unfreezed_from_last ( Number of layers, weights can update during backpropagation )
learning_rate
batch_size
epochs
weight_decay

Freezing all the layers in the pretrained Freezing all layers except the last layer in the base model
Unfreezing the last k layers in the pretrained model and freezing the rest.
Unfreezing all the layers in the pretrained model.

Question 3 (15 Marks)

Using a huge pre-trained network works better than training a smaller network from scratch (as you did in Part A)
InceptionV3 works better for this task than ResNet50
Using a pre-trained model, leads to faster convergence as opposed to training a model from scratch
... ....

Answer

Inferences:

We tried following huge pretrained models and they best Validation Accuracies.

Using a huge pre-trained network works is very better than training a smaller network from scratch as in part A. Accuracy increased nearly 2 times.
Accuracy is improved when more layers are open for training. Need to train with more epochs when trainable layers are increased. Infact, number of trainable layers is the most important hyperparameter among others in the sweep. This makes the model computationally cheaper. Unfreezing more layers increases training time and is computationally more expensive.
Using a pre-trained model leads to faster convergence as opposed to training a model from scratch.
The best accuracy for pre-trained model occurs at 7 epochs (some models take 10 epochs), while as for part A we required nearly 30 epochs or more.
Negative correlation with batch size implies that training with smaller batches(8,16) are worked well than larger batch size(64,128).
As epochs increasing val-accuracy is increasing. Increasing the number of epochs do help in certain cases but the model was trying to overfit the data.
Using more number of epochs doesn't necessary reduce the validation loss, sometimes we observed that training accuracy is increasing but validation accuracy is decreasing, this is clear sign of overfitting.
The positive correlation with weight decay implies that adding weight decay to our optimizer(Adam) increases accuracy. It decreases the overfitting.
we tried Adam, SGD optimizers. Adam performs well than SGD.
Using Data Augumentation increases the accuray. Infact, we got best validation accuracy of 82% got with data augumentation.

Question 4 (10 Marks)

Answer

Part C : Using a pre-trained model as it is

Question 1 (15 Marks)



Answer

Fire Detection at Petrol Pump

Fire Detection at Airport

Social relevance :

Self Declaration

Designing and implementing Convolution Neural Network model from scratch.
Analysing the parallel coordinates plot and correlation plot for sweeps on our model trained from scratch.
Writing inferences based on above analysis
Analysing and implementing Guided Backpropagation on Convolution layer 5 neurons.
Implemented YoloV3 Model & Object Detection
Written README file for Part-A

Designing and implementing Convolution Neural Network model from scratch.
Analysing the parallel coordinates plot and correlation plot for sweeps on best pre-trained models.
Writing inferences based on above analysis
Training huge pre trained models on iNaturalist dataset.
Implemented YoloV3 Model & Object Detection
Written README file for Part-B