ANN Report: Inception
Architecture training reports in Advanced Neural Network course 2022.
Created on April 19|Last edited on April 19
Comment
There were 6 different configurations with two options respectively, resulting in 64 different runs. Three different seeds were also tested to prevent each configurations' lucky change of reaching higher performance. Tested configurations as follows
- Learning Rate: 1e-2 vs. 1e-3
- Batch Size: 16 vs 32.
- Label Smoothing: 0 vs 0.3
- Augmentation: use vs not.
- Regularizer: L2 vs None.
- Scheduler: Plateau vs None.
Across all configurations, same logic was used such as -
- Binary Cross Entropy was used to compare models
- Adam optimizer with typical hyperparameters
- Early stopping criterion triggered by validation loss with patience of 8 epochs.
Learning Rate (1e-2 vs. 1e-3)
Larger learning rate for sure in most cases failed to optimize. Each learning has total 96 (=32 * 3) runs and for smaller learning rate 83 cases (86.54%) successfully optimizerd while only 31 cases (32.29%) survived in larger learning rate. Their validation accuracy is differs about 5~6%p
Batch Size: 16 vs. 32
Difference in having varying batch sizes made ~2%p difference in validation accuracy. However their validation loss in the later stages clearly shows different.
Label Smoothing: 0 vs. 0.3
*
Augmentation: Transform vs. None
Here I tried to use RandAugment for transforming the input image. However, this technique uses numerous transformation techniques and some are not suitable for this medical images. Therefore, only 3 transformations were used: Flip(left-right, up-down), Rotation and Zoom in a slight degree.
The difference in accruacy was less than 0.1%p, no difference.
Regularizer: L2 norm vs. None
In order to prevent models from being overfitted to the data, we give constraints to the norm of parameters to stay under certain scales. In this work, I compared with L2-norm and without regularization and found in early stages condition without regularization surpass the other. However in the end, the difference was considered trivial, having ~1%p gap between validation accuracy.
Learning Rate Scheduler: Plateau vs None.
It is known to reduce learning rate when the performance starts to saturate. Here I applied learning rate scheduler with reducing by factor of 0.3 with patience of 8 epochs, triggered by validation loss. Learning rate in average started to decrease around 30~40 epochs. However trainings without learning rate scheduler slightly outperformed the scheduler around 1%p in validation accuracy.
Overall
At last, the best performed model award goes to
- Learning rate 1e-3
- Batch size 32
- Augmentation True
- Regularizer None
- Label smoothing 0.3
- Scheduler None
With validation accuracy of 95.32% and AUC 98.32% at 65th epoch. Further analysis will be done with this checkpoint.
Add a comment