Skip to main content

Overfitting... Techniques to prevent

Created on May 7|Last edited on May 9

References:

Overfitting .... perform very well on training data but very poorly in validation or test data.
bias-variance tradeoff .... Bias is the difference between the average prediction of our model and the correct value which we are trying to predict. A model with high bias pays very little attention to the training data and oversimplifies the model. It always leads to high error on training and test data. Alternatively, Variance refers to the variability of model prediction for a given data point or a value which tells us spread of our data. Model with high variance pays a lot of attention to training data and does not generalize on the data which it hasn’t seen before. As a result, such models perform very well on training data but has high error rates on test data.

Techniques:

Train on large clean dataset (well distributed images)

Cross-Validation:

To evaluate dataset, we can use various methods of CV. Some of the most commonly ones are listed below:
  1. Hold-out
  2. k-folds
  3. Leave-one-out
  4. Leave-p-out
  5. Stratified K-folds
  6. Repeated K-folds
  7. Nested K-folds
  8. Time series CV

Data Augmentation

Learning Rate Scheduling

Feature Selection

Regularization

  1. L1 Regularization
  2. L2 Regularization

Remove layers/Number of units per layer

Dropout

Ignoring a subset of units of the neural network


Early Stopping

Ensembling***

  1. Bagging: It uses complex base models and tries to "smooth out" their predictions.
  2. Boosting: It uses simple base models and tries to "boost" their aggregate complexity.
  3. Stacking