Registering Models at RBC!

Instructions and Conventions for Getting Your Models to Production!
Created on September 22|Last edited on September 22
Comment
﻿
﻿
Summary of Models at RBCSteps to Log and Register Your Models1. Create a Registered Model2. Log Model Checkpoints as Artifacts to W&B3. Link Your Best Model Version to the Registered ModelAlias Conventions at RBC and what they meanExample Model Card5. Retrieving Models from the Model Registry6. What happens to my model after I register it?
﻿
Summary of Models at RBCThe RBC MLE team is currently working on three primary projects:
Debt Payoff Prediction
Summary: LSTM model to predict the length of time that we thought a client might take between having an influx of revenue to paying off debt or taking on more debt
Credit Prediction
Summary: Given a client’s credit and debt history, if they apply for a line of credit (home equity, credit card, unsecured loan etc) range of acceptable credit that we could lend them below potential default rate
Candidate demos to build/augment:
Fairness
Attaching weights and biases to explore fairness of algorithms.  Fairness is evaluated via Fairlearn and Weights and Biases is used to track everythging done.  Borrows heavily (lots of copy and paste) from the intro example available at fairlearn.org.  The original notebook can be found. 
 https://github.com/fairlearn/fairlearn/blob/main/notebooks/Binary%20Classification%20with%20the%20UCI%20Credit-card%20Default%20Dataset.ipynb
﻿
Steps to Log and Register Your Models﻿
project("wandb", "Model Registry Quickstart").artifact("my-model")
Error: Could not load
1. Create a Registered ModelGo the Model Registry and click "Create Registered Model". Registered Models will house pointers to your model artifact versions which you log in your projects. Think of this as a bookmark folder for the best or most interesting model versions.  
﻿
﻿
﻿
2. Log Model Checkpoints as Artifacts to W&BAt RBC we use a variety of libraries. W&B has several convenient integrations for logging model artifacts using common ML frameworks and libraries. Checkout these links for how to leverage W&B with common integrations. 
﻿XGBoost﻿
﻿Pytorch Lightning﻿
﻿Scikit-learn﻿
﻿LightGBM﻿
Catboost
You can also log model artifacts outside of these frameworks using a couple lines of W&B code (if for example you have custom code that serializes the model):
with wandb.init(project="my_project_name", entity='my_tea', job_type="model_training") as run:
  # Indicate we are using a dependency on another dataset artifact
  training_imgs_artifact = wandb.use_artifact("training_images:latest")
  training_images_dir = training_imgs_artifact.download()
﻿
  # Do some training....
  model = wandb.Artifact(name='resnet50', 
                         type="model", 
                         description="resenet50 model trained on MNIST",
                         metadata={'eval_accuracy': 0.97,
                                   'train_accuracy': 0.99})
  model.add_file('resnet_checkpoint_51122.hd5')
  wandb.log_artifact(model)
Check out additional documentation on how to use W&B Artifacts to track, version, and document your models and datasets:
﻿
3. Link Your Best Model Version to the Registered ModelAfter training a model, you will see a button to link your best model version to the registry. At this time you will append the staging alias to the model:
﻿
Alias Conventions at RBC and what they meanstaging models are ready for review by EMRO team. They will review all staging models and then direct you about what specific contents may be needed on top of the basics detailed below. Two tasks will need to be completed before the model can be moved to production:
Data Scientist creates Report with detailed sections on:
Methodology
Input Datasets and Summary Statistics
Training and Testing metrics and diagnostics (ROC Curve, AUC, F1, etc.)
Feature importance and explainability
Table of sample predictions and prediction explanations
Robustness and Sensitivity (e.g. testing for bias and fairness)
Data Scientist fills out model card detailing:
Model Inputs and Outputs
Model Deserialization and API details
Model Dependencies
production models are ready to be consumed by the engineering team. TBD specifics. Right now every team is different. Model management would not give a go-ahead, just give a checkbox on a governance document. 
Right now, "model card" gets registered, what it does, etc. handled by Governance team. 
Example Model Card
5. Retrieving Models from the Model Registry
6. What happens to my model after I register it?﻿
﻿
﻿
Add a comment