Object detection of Blood Cells using IceVision

Introduction



Background



IceVision is the first agnostic computer vision framework to offer a curated collection with hundreds of high-quality pre-trained models from torchvision, MMLabs, and soon Pytorch Image Models. It orchestrates the end-to-end deep learning workflow allowing to train networks with easy-to-use robust high-performance libraries such as Pytorch-Lightning and Fastai (IceVision)



Goal

Data Preparation



Data download

 BCCD
├── Annotations: BloodImage_00XYZ.xml (364 items)
├── ImageSets       # Contain four Main/*.txt which split the dataset
└── JPEGImages: BloodImage_00XYZ.jpg (364 items)

EDA

Raw data check

Distribution check

Finding weird cases

Modeling



Basic setting

Augmentation
As the intrinsic information of images don't change when flipping, the default augmentation setting of IceVision (horizontal flip, shift, scaling, rotating, RGB shift, random brightness contrast, bluring, croping) was applied to the dataset.
Model type
Some object detection models: Faster RCNN, Yolov5, Yolof, Vfnet, Retinanet, Deter, Deformale det, Fsaf, Centripetalent, and Efficientdet were tried. (SSD, Yolox, Retinanet, and Sbal were removed as those caused errors in version 0.12.0 of IceVision.)
Optimization
Three famous optimization algorithms Adam, NAdam and AdamW were tried.
Evaluation metric
As described in the previous section, it is important to check not only overall mAP but also AP of each label as some labels of RBCs are missing. So, so AP of each label was tracked in this report.
COCO mAP, which is a popular one type of mAP in measuring the accuracy of object detection models, was used for this report.
COCO mAP
Average precision (AP) is a metric to measure the accuracy of object detection models. Average precision computes the average precision value for recall value over 0 to 1. mAP is the average value of APs for each class.
In COCO mAP, the average AP over multiple IoU is calculated. AP@[.5:.95] corresponds to the average AP for IoU from 0.5 to 0.95 with a step size of 0.05. AP@[.5:.95] over 10 IoU levels on 80 categories is used for the COCO competition. AP@[.5:.95] was used as an evaluation metric for this report.
Here is an easy-to-understand interpretive article on AP and COCO mAP.
💡
Learning rate
Learning rate is an important hyper-parameter as it controls the rate or speed of model learning. Lesile Smith (2015) proposed an idea regarding learning rate, called the learning rate finder. The idea is to reduce the amount of guesswork on picking a good starting learning rate. Fastai's learning rate finder was used in this report, and tried four methods ("minimum", "steep", "valley", "slide").

Fig2. example of a result of learn.lr_find()

Process of model construction

First, some model types with pre-trained backbones were tried as base models.
Additional learning was applied to each model. The detailed setting of simulation can be seen in W&S Sweep setting page.
Then, hyper parameter tuning was conducted to the best model type.
Finally, the model was retrained with larger images and more epochs.

sweep_config = {    
    'method':'random',
    'name':'model',
    'metric':{
        'name':'COCOMetric',
        'goal':'maximize'
    },
}
parameters_dict = {
    'model_selection':{
        'values':["vfnet","retinanet","faster_rcnn","yolof","detr","deformale_detr","fsaf","centripetalnet","efficientdet","yolov5"]
    },
    'pre_size':{
        'value':(640,480)
    },
    'lr_type':{
        'values':['minimum','steep','valley', 'slide']
    },
    'opttype':{
        'values':['NAdam','Adam','AdamW']
    },
	####snip###
}
sweep_config['parameters'] = parameters_dict
wandb.login(key=params.WANDB_API_KEY) 
sweep_id = wandb.sweep(sweep_config, project=params.WANDB_PROJECT)

def train(config=None):
    run = wandb.init()
    config = wandb.config
    
    # data preparation (create_ds is a custom function for data preparation)
    train_ds,valid_ds,test_ds,parser = create_ds(imageSets_dir, annotations_dir, 
                                          images_dir, config.img_size, config.pre_size, config.augment)
    
    # model definition (model_denifition is a custom function for model type definition)
    model,model_type = model_definition(selection=config.model_selection, 
                                              parser=parser, img_size=config.img_size,backbones=config.backbones)
    # data load to model
    train_dl = model_type.train_dl(train_ds, batch_size=config.batch_size, num_workers=config.num_workers, shuffle=False)
    valid_dl = model_type.valid_dl(valid_ds, batch_size=config.batch_size, num_workers=config.num_workers, shuffle=False)
    # model setting (COCOMetric_WBC(), COCOMetric_RBC(), and COCOMetric_Platelets() are custom class to calculate each class's AP)
    metrics = [COCOMetric(),
               COCOMetric_WBC(),COCOMetric_RBC(),COCOMetric_Platelets()]
    opt = opt_func_setting(config.opttype) # (opt_func_setting is a custom function to set optimizer)
    opt_func = partial(OptimWrapper, opt=opt) #https://docs.fast.ai/optimizer.html#optimizer
    learn = model_type.fastai.learner(dls=[train_dl, valid_dl], model=model, 
                                      metrics=metrics, opt_func=opt_func)
    # learning rate
    suggested_lr  =   learn.lr_find(suggest_funcs=(minimum, steep, valley, slide),show_plot=False)
    suggested_funcs_dic = {'minimum':0,'steep':1,'valley':2, 'slide':3}
    base_lr = suggested_lr[suggested_funcs_dic[config.lr_type]]

    # call backs
    callbacks = [
        SaveModelCallback(monitor="COCOMetric"),
        WandbCallback(log_preds=False, log_model=False)
    ]
    # fine tune
    learn.fine_tune(config.epochs, base_lr= base_lr, freeze_epochs=config.freeze_epochs,
                    cbs=callbacks)
    # log
    log_final_metric(learn) # (log_final_metric is a custom function to log learners' metrics)

wandb.agent(sweep_id,function=train,count=100)

1. Model type comparison

2. Hyper parameter tuning

3. Retraining with larger images and more epochs

Evaluation



Result of validation dataset

Result of holdout test

Thoughts



Future work

Impression of W&B

First things first, I like an experience of the reproducibility. Even if a new method is revolutionary, if it cannot be reproduced, it will not bring a value and not lead to the next innovation. From this perspective, I like the idea of ensuring a high level of transparency. I also like the concept of having a uniform output format of model evaluation and management as it promotes collaboration of works.
If I pick up one my favorite feature, it would be the "Artifact". It enables me to check data interactively, and to track which model used which data in "Liniage". "Liniage" is simple, and makes our analysis more transparent and well-organized one.
I also like the portion of open contents. W&B is useful for model building but also information gathering.

It would be great if it gets easier to find examples of registering various type of data to Artifact in docs.
To arrange artifacts, it would be better to have an option to delete multiple Artifact folders at once. Although there is an API to delete Artifacts, the folders were not erased when I tried.
More flexibility of chart titles would improve report's readability.