Assignment 3

Instructions

The goal of this assignment is fourfold: (i) learn how to model sequence to sequence learning problems using Recurrent Neural Networks (ii) compare different cells such as vanilla RNN, LSTM and GRU (iii) understand how attention networks overcome the limitations of vanilla seq2seq models (iv) visualise the interactions between different components in a RNN based model.
We strongly recommend that you work on this assignment in a team of size 2. Both the members of the team are expected to work together (in a subsequent viva both members will be expected to answer questions, explain the code, etc).
Collaborations and discussions with other groups are strictly prohibited.
You must use Python (numpy and pandas) for your implementation.
You can use any and all packages from keras, pytorch, tensorflow
You can run the code in a jupyter notebook on colab by enabling GPUs.
You have to generate the report in the same format as shown below using wandb.ai. You can start by cloning this report using the clone option above. Most of the plots that we have asked for below can be (automatically) generated using the apis provided by wandb.ai. You will upload a link to this report on gradescope.
You also need to provide a link to your github code as shown below. Follow good software engineering practices and set up a github repo for the project on Day 1. Please do not write all code on your local machine and push everything to github on the last day. The commits in github should reflect how the code has evolved during the course of the assignment.
You have to check moodle regularly for updates regarding the assignment.

Problem Statement

Question 1 (15 Marks)

Answer:

PARAMETERS

RNN:

LSTM :

GRU :

COMPUTATIONS

RNN

LSTM :

GRU :

Question 2 (10 Marks)

input embedding size: 16, 32, 64, 256, ...
number of encoder layers: 1, 2, 3
number of decoder layers: 1, 2, 3
hidden layer size: 16, 32, 64, 256, ...
cell type: RNN, GRU, LSTM
dropout: 20%, 30% (btw, where will you add dropout? you should read up a bit on this)
beam search in decoder with different beam sizes:

accuracy v/s created plot (I would like to see the number of experiments you ran to get the best configuration).
parallel co-ordinates plot
correlation summary table (to see the correlation of each hyperparameter with the loss/accuracy)

Answer:

Validation Accuracy v/s created plot and Training Accuracy v/s created plots are shown below that plots the validation and training accuracy achieved for each experiment performed according to its creation date. Validation accuracy is achieved on the dev set in dakshina_dataset_v1.0/hi/lexicons/ folder and the validation accuracy is coming around 60%. Training is done on the train set and training accuracy is more than 90%.
Parallel coordinates plot is attached just below the Accuracy v/s created plot that shows all the hyperparameters corresponding to its accuracy.
Correlation summary table is plotted below parallel coordinates plot that shows a summary of correlation of different hyperparameters with respect to the validation accuracy.
Validation Accuracy and Training Accuracy are also plotted below for different experiments.

Validation Accuracy v/s created plot

Training Accuracy v/s created plots

Parallel coordinates plot

Correlation summary table and Validation Accuracy, Training Accuracy

Input embedding sizes (encoder_embedding_size)
Different number of encoder layers(encoder_decoder_latentDims):
Different number of decoder layers(encoder_decoder_latentDims):
Dropout(dropout):
Beam search decoder(beam_width):
Different number of epochs(epochs):
Different cell(cell_type):
Different learning rates(learning_rate):
Different Batch sizes (batch_size):
Different optimizers(optimiser):
Decoder embedding sizes (decoder_embedding_size)

Explanation of where and why dropout is put:

#Different sweep configurations
sweep_config = {
    "method":"bayes",
}
metric = {
    "name" : "val_accuracy",
    "goal" : "maximize"
}
sweep_config['metric']=metric
parameter_dict = {
        'encoder_embedding_size': {
            'values': [32,64,128,256,512]
        },
        'decoder_embedding_size': {
            'values': [32,64,128,256,512]
        },
        'cell_type': {
            'values': ['lstm', 'gru', 'rnn']
        },
        'encoder_decoder_latentDims': {
            'values': [ [256],[512],[64,128],[256,512], [256,128], [64,128,256], [128,128,128], [512,256,128], [512,512,512], [1024 , 512 , 256], [512,256,128,64]]
        },
        'dropout': {
            'values': [0 , 0.1 ,0.3 , 0.5]
        },
        'optimiser': {
            'values': ['adam', 'rmsprop', 'sgd']
        },
        'learning_rate': {
            'values': [0.0001 , 0.001, 0.002, 0.003, 0.005, 0.0005]
        },
        'epochs': {
            'values': [4,6,8,10,15,20,25,30,35,40,50]
        },
        'beam_width': {
            'values': [1,2,3,5]
        },
        'batch_size':{
              'values': [32, 64, 128, 256, 512]
        }
}

sweep_config['parameters']=parameter_dict

Bayes search strategy: Wandb provides with different search strategies to choose between the hyperparameters such as 1) Random search, 2) Grid Search and 3) Bayes Search. We choose Bayes search strategy which uses gaussian function to model the function and then chooses parameter to optimize probability of improvement. Here we are optimizing (maximizing) over the validation accuracy.
Same Encoder-Decoder Size: To reduce the number of runs we tried to reduce the number of parameters in sweep. So, we kept the same number of encoders and decoders.
Beam Width: We tried to make it computationally less heavy and so we tried beam_width of 1,2,3 and we got our desired results.
Encoder embedding size: Since the number of unique tokens in the languages we used (english -> hindi) is not that large, we restricted our input and output embedding sizes to a maximum of 512.

Question 3 (15 Marks)

RNN based model takes longer time to converge than GRU or LSTM
using smaller sizes for the hidden layer does not give good results
dropout leads to better performance

Answer:

Some comment on how increasing/decreasing/not changing the number of encoder and decoder latent dimensions helps/does not help : As in the sweep hyperparameter encoder_decoder_latentDims we used different filter size like 'values': [[256],[512],[64,128],[256,512], [256,128], [64,128,256], [128,128,128], [512,256,128], [512,512,512], [1024 , 512 , 256], [512,256,128,64]]. So [64,128,256] means 64 is the encoder latent dimension size for the first cell followed by 128 for the second and 256 for the third. Since we are taking encoder and decoder as the same size so decoder sizes will also be the same. Similarly we tried with increasing the size in each layer as in [64,128],[256,512] and [64,128,256] or decreasing like in [512,256,128],[1024 , 512 , 256], [512,256,128,64] or even keeping all sizes same like in [128,128,128], [512,512,512]. As in the parallel coordinate plot we observed that the decreasing size like [256,128] is giving better accuracy than increasing the filter size in later layers like [64,128,256]. Even keeping sizes same in each layer like [128,128,128], [512,512,512] is giving better accuracy than increasing the sizes in subsequent layers. We got the best accuracy with the configuration as [256,128].
Some comment on how increasing/decreasing the size of the input embeddings across layers helps/does not help: As in the sweep hyperparameter encoder_embedding_size we tried various numbers of input size. We sweeped over 'values': [32,64,128,256,512]. We got our best test result when we used higher sizes like 512 or 256, whereas we are getting bad accuracy for values like 32. Moreover we get from the correlation summary table that the parameter encoder_embedding_size is highly positively correlated with accuracy.
Some comment on how increasing/decreasing the size of the LSTM hidden state in the encoder/decoder helps/does not help: As in the sweep hyperparameter encoder_decoder_latentDims we used different filter size like 'values': [[256],[512],[64,128],[256,512], [256,128], [64,128,256], [128,128,128], [512,256,128], [512,512,512], [1024 , 512 , 256], [512,256,128,64]]. We could find that LSTM or GRU are giving better results compared to RNN cells. Now, moreover increasing the size of the LSTM hidden stat in the encoder and decoder is giving better result. As in the parallel coordinate plot we observed that the decreasing size like [256,128] is giving better accuracy than increasing the filter size in later layers like [64,128,256]. We got the best accuracy with the configuration as [256,128]. Moreover in the correlation summary table it is positively correlated so higher the value gives better result.
Some comparison between LSTM, GRU, RNN: GRU and LSTM Models converge faster than Simple RNN models. Overall GRU converged fastest among the three. This is clear from their respective correlations in the correlation table. We are getting our best results from GRU cell. LSTM and GRU are positively correlated in the summary table whereas RNN is negatively correlated.
Comments on whether dropout helps or not: Dropout: It is used in recurrent cells like LSTM, GRU or RNN to check the overfit. In our hyperparameters we used dropout 'values': [0 , 0.1 ,0.3 , 0.5]. So, it is found that the accuracy increases when we use dropouts. Our best model is trained with dropout taking 0.3. From the correlation table it is evident that dropout is positively correlated with the accuracy.
Some comment on whether beam search helped/did not help : The positive correlation with beam width depicts that using larger beam width improves the accuracy as it should be the case because we are not purely relying on the topmost prediction rather on a set of top k predictions where k is the beam width. But using large beam widths comes with a computational overhead.
Different number of epochs: We tried with different epochs like 'values': [4,6,8,10,15,20,25,30,35,40,50] and found that number of epoch plays an important role in accuracy. If we use small epochs like 5 or 10 then the model is less trained and doesn't give good accuracy and with too much number of epochs its overfitted like in 50 and accuracy starts decreasing. So, epochs like 15, 20 are good for this model. Our best model is trained with 25 epochs.
Some comment on the use of different learning rates: Different learning rates: We ran sweep with learning_rate 'values': [0.0001 , 0.001, 0.002, 0.003, 0.005, 0.0005] . It is observed that lower learning rate is giving better accuracy like 0.002 is giving better accuracy than that of 0.0005. Our best model is trained with learning rate 0.002. From the summary table it is evident that it is highly positively correlated.
Some comments on the use of different batch sizes: Different Batch sizes: With different batch_size the accuracy also varies. So we tried with different batch sizes 'values': [32, 64, 128, 256, 512]. Our best model is trained with batch size of 32. As from the correlation table batch size highly important and it is negatively correlated with the accuracy so small sizes gave better accuracy.
Comments on the use of different learning algorithms: Different optimizers: We tried with two optimizer_name in the sweep 'values': ['adam', 'rmsprop', 'sgd']. 'rmsprop' gave better accuracy that 'sgd' and our best model is trained with 'rmsprop'. So, most of the good accuracy came with 'rmsprop'.

Question 4 (10 Marks)

The model makes more errors on consonants than vowels
The model makes more errors on longer sequences
I am thinking confusion matrix but may be it's just me!
...

Answer:

Accuracy

#Best configuration model with attention OFF
#If you want to test model without attention use this configuration
latent_dims = [128 , 128 , 128]
embed_dims = [128 , 256]
attention = False
recurrent_layer = 'gru'
dropout = 0.1
batch_size = 64  # Batch size for training.
epochs = 20  # Number of epochs to train for.
learning_rate = 0.002
optimiser = 'rmsprop'
beam_width = 3
wandb_params = False

Test accuracy : 100.0, Completion = 0.0
Test accuracy : 31.683168316831683, Completion = 0.02221235006663705
Test accuracy : 35.32338308457712, Completion = 0.0444247001332741
Test accuracy : 40.863787375415285, Completion = 0.06663705019991115
Test accuracy : 40.14962593516209, Completion = 0.0888494002665482
Test accuracy : 38.32335329341317, Completion = 0.11106175033318526
Test accuracy : 37.770382695507486, Completion = 0.1332741003998223
Test accuracy : 36.80456490727532, Completion = 0.15548645046645934
Test accuracy : 36.5792759051186, Completion = 0.1776988005330964
Test accuracy : 37.957824639289676, Completion = 0.19991115059973344
Test accuracy : 38.96103896103896, Completion = 0.22212350066637052
Test accuracy : 39.963669391462304, Completion = 0.24433585073300756
Test accuracy : 40.96586178184846, Completion = 0.2665482007996446
Test accuracy : 40.4304381245196, Completion = 0.28876055086628166
Test accuracy : 41.256245538900785, Completion = 0.3109729009329187
Test accuracy : 42.10526315789474, Completion = 0.33318525099955576
Test accuracy : 41.78638351030606, Completion = 0.3553976010661928
Test accuracy : 41.740152851263964, Completion = 0.37760995113282986
Test accuracy : 41.14380899500278, Completion = 0.3998223011994669
Test accuracy : 42.08311415044713, Completion = 0.42203465126610396
Test accuracy : 42.478760619690156, Completion = 0.44424700133274103
Test accuracy : 42.69395525940028, Completion = 0.46645935139937805
Test accuracy : 43.980009086778736, Completion = 0.48867170146601513
Test accuracy : 44.15471534115602, Completion = 0.5108840515326522
Test accuracy : 44.35651811745106, Completion = 0.5330964015992892
Test accuracy : 44.742103158736505, Completion = 0.5553087516659263
Test accuracy : 44.482891195693966, Completion = 0.5775211017325633
Test accuracy : 44.42798963346909, Completion = 0.5997334517992003
Test accuracy : 44.62691895751517, Completion = 0.6219458018658374
Test accuracy : 44.777662874870735, Completion = 0.6441581519324745
Test accuracy : 45.21826057980673, Completion = 0.6663705019991115
Test accuracy : 45.21122218639149, Completion = 0.6885828520657485
Test accuracy : 45.48578569197126, Completion = 0.7107952021323856
Test accuracy : 45.71342017570433, Completion = 0.7330075521990227
Test accuracy : 45.89826521611291, Completion = 0.7552199022656597
Test accuracy : 46.072550699800054, Completion = 0.7774322523322967
Test accuracy : 46.1816162177173, Completion = 0.7996446023989338
Test accuracy : 46.09564982437179, Completion = 0.8218569524655709
Test accuracy : 46.14575111812681, Completion = 0.8440693025322079
Test accuracy : 46.0907459625737, Completion = 0.8662816525988449
Test accuracy : 46.16345913521619, Completion = 0.8884940026654821
Test accuracy : 46.15947329919532, Completion = 0.9107063527321191
Test accuracy : 45.8938348012378, Completion = 0.9329187027987561
Test accuracy : 46.05905603348059, Completion = 0.9551310528653931
Test accuracy : 46.26221313337878, Completion = 0.9773434029320303
Test accuracy : 46.23417018440347, Completion = 0.9995557529986673
Test accuracy : 46.24611283873834

Sample Outputs:

Sample test data, predicted hindi and actual hindi

Predictions_Vanilla

Some comment about errors in vowels v/s consonants:

Some comment about errors on longer sequences v/s shorter sequences:

Some comment about confusion between similar characters:

Other insightful observation:

el is predicted as इल but actual ground truth is ऐल. Now the letter 'e' here pronounced as 'ऐ' instead of 'इ'. Here due to different use of same letter the model is confused between the actual use.
Sometimes it may happen that different devnagari script are use to represent same english character like in रैस it is replaced with रई of रईस.
Another important observation is that the model was unable to detect similar sounding corresponding devanagari घ with ग.
Similar sounded words like टब and तब are not distinguished well by the model.
In tarko the model predicted तर्को in place of तर्कों so the dot ं to represent plural is missed.
The english letter 'g' can be pronounced as 'ज' or 'ग' based on the context. In the word 'drager' our model failed to detect the difference and predicted ड्रेगर in place of ड्रेजर.

Question 5 (20 Marks)

Answer:

Validation Accuracy v/s created plot shown below are the plots showing the validation accuracy achieved for each experiment performed according to its creation date. Validation accuracy is achieved with attention on the dev set in dakshina_dataset_v1.0/hi/lexicons/ folder and the validation accuracy is coming around 60%. Training is done on the train set and training accuracy is more than 90%.
Parallel coordinates plot is attached just below the Accuracy v/s created plot that shows all the hyperparameters corresponding to its accuracy.
Correlation summary table is plotted below parallel coordinates plot that shows a summary of correlation of different hyperparameters with respect to the validation accuracy.
Validation Accuracy and Training Accuracy are also plotted below for different experiments.

Accuracy v/s Created plot

Parallel coordinates plot

Correlation summary table and Validation Accuracy, Training Accuracy

Accuracy improvement corresponding to seq2seq model:

Validation Accuracy with attention 64.8%

Test accuracy with Attention: 100.0, Completion = 0.0
Test accuracy with Attention: 40.59405940594059, Completion = 0.02221235006663705
Test accuracy with Attention: 46.26865671641791, Completion = 0.0444247001332741
Test accuracy with Attention: 51.16279069767442, Completion = 0.06663705019991115
Test accuracy with Attention: 49.12718204488778, Completion = 0.0888494002665482
Test accuracy with Attention: 49.30139720558882, Completion = 0.11106175033318526
Test accuracy with Attention: 47.25457570715474, Completion = 0.1332741003998223
Test accuracy with Attention: 46.3623395149786, Completion = 0.15548645046645934
Test accuracy with Attention: 45.3183520599251, Completion = 0.1776988005330964
Test accuracy with Attention: 45.94894561598224, Completion = 0.19991115059973344
Test accuracy with Attention: 46.353646353646354, Completion = 0.22212350066637052
Test accuracy with Attention: 46.59400544959128, Completion = 0.24433585073300756
Test accuracy with Attention: 46.711074104912576, Completion = 0.2665482007996446
Test accuracy with Attention: 46.11837048424289, Completion = 0.28876055086628166
Test accuracy with Attention: 47.0378301213419, Completion = 0.3109729009329187
Test accuracy with Attention: 47.96802131912059, Completion = 0.33318525099955576
Test accuracy with Attention: 47.28294815740163, Completion = 0.3553976010661928
Test accuracy with Attention: 47.325102880658434, Completion = 0.37760995113282986
Test accuracy with Attention: 46.86285397001666, Completion = 0.3998223011994669
Test accuracy with Attention: 47.44871120462914, Completion = 0.42203465126610396
Test accuracy with Attention: 47.876061969015495, Completion = 0.44424700133274103
Test accuracy with Attention: 48.453117563065206, Completion = 0.46645935139937805
Test accuracy with Attention: 49.795547478418904, Completion = 0.48867170146601513
Test accuracy with Attention: 50.282485875706215, Completion = 0.5108840515326522
Test accuracy with Attention: 50.56226572261558, Completion = 0.5330964015992892
Test accuracy with Attention: 50.53978408636545, Completion = 0.5553087516659263
Test accuracy with Attention: 50.403690888119954, Completion = 0.5775211017325633
Test accuracy with Attention: 50.46279155868197, Completion = 0.5997334517992003
Test accuracy with Attention: 50.66047840057122, Completion = 0.6219458018658374
Test accuracy with Attention: 50.84453636677008, Completion = 0.6441581519324745
Test accuracy with Attention: 51.24958347217594, Completion = 0.6663705019991115
Test accuracy with Attention: 50.983553692357304, Completion = 0.6885828520657485
Test accuracy with Attention: 51.23398937831927, Completion = 0.7107952021323856
Test accuracy with Attention: 51.408664041199636, Completion = 0.7330075521990227
Test accuracy with Attention: 51.51426051161423, Completion = 0.7552199022656597
Test accuracy with Attention: 51.670951156812336, Completion = 0.7774322523322967
Test accuracy with Attention: 51.84670924743127, Completion = 0.7996446023989338
Test accuracy with Attention: 51.769791948122126, Completion = 0.8218569524655709
Test accuracy with Attention: 51.775848460931336, Completion = 0.8440693025322079
Test accuracy with Attention: 51.832863368367086, Completion = 0.8662816525988449
Test accuracy with Attention: 51.962009497625594, Completion = 0.8884940026654821
Test accuracy with Attention: 52.011704462326264, Completion = 0.9107063527321191
Test accuracy with Attention: 51.535348726493694, Completion = 0.9329187027987561
Test accuracy with Attention: 51.77865612648221, Completion = 0.9551310528653931
Test accuracy with Attention: 51.96546239491025, Completion = 0.9773434029320303
Test accuracy with Attention: 52.0328815818707, Completion = 0.9995557529986673
Test accuracy :  52.04353620613061

Analysis of the errors corrected by the attention based model compared to the seq2seq:

Sample Outputs:



As in the Q4 we showed how there is more error for vowels than consonants in without attention model, but here with attention this error with vowels is reduced to some extant as there is specific attention given to the pronunciation of each character. For example: Without attention ank is predicted as आंक where as with attention it is correctly predicted as अंक.
el was predicted as इल in the seq2seq model previously due to the different pronunciation of letter e in two different context, but here with attention it is predicted as एल which is more similar to ऐल the ground truth.
As in the previous model, there was some confusion between the long and short vowels. But for models with attention, it is somewhat solved, like in the case with 'e', it is quite solved.
Here, the problem faced by the seq2seq model of not being able to predict similar sounding characters like ग and घ is slightly reduced.
The english letter 'g' can be pronounced as 'ज' or 'ग' based on the context. In the word 'drager' our previous model failed to detect the difference and predicted ड्रेगर in place of ड्रेजर where as the attention based model predicted it correctly as ड्रेजर

Attention maps plot







Interesting patterns on attention maps:

Question 6 (20 Marks)

Answer:

ank अंक:





















Question 7 (10 Marks)

We will check for coding style, clarity in using functions and a README file with clear instructions on training and evaluating the model (the 10 marks will be based on this).
We will also run a plagiarism check to ensure that the code is not copied (0 marks in the assignment if we find that the code is plagiarised).
We will check the number of commits made by the two team members and then give marks accordingly. For example, if we see 70% of the commits were made by one team member then that member will get more marks in the assignment (note that this contribution will decide the marks split for the entire assignment and not just this question).
We will also check if the training and test splits have been used properly. You will get 0 marks on the assignment if we find any cheating (e.g., adding test data to training data) to get higher accuracy.

Answer:

Question 8 (0 Marks)

Answer:

"I love Deep Learning"

Self Declaration

Code for data loading
Calculating number of Computations
Designing and implementing code for model Encoder
Code for Beam Decoder
Designing and implementing code for Decoder Inference
Ran Sweep for best model without attention
Generating Attention map and analysing error
Visualization of our model

Code for data preprocessing
Calculating number of parameters
Designing and implementing code for model Decoder
Code for testing Accuracy
Designing and implementing code for Encoder Inference
Ran Sweep for best model with attention
Generating hindi output for the test data and comparing and analysing errors by our model
Generating "I love Deep Learning" song