M-KOPA Transforms Their Model Management Workflow with W&B
"Using the W&B Model Registry for model promotion has been extremely useful. Having the full history of the model available also makes it very easy for other departments — like product management — to help manage the deployment process."
Benedict Eugine
Data Scientist
Model management is important for all machine learning teams, but especially critical for those working in highly regulated industries like fintech, as well as small agile data science teams integrated into technology and business product groups. Having automated, scalable systems and processes is crucial to an efficient workflow.
For the ML team at M-KOPA – who provides digital financing in a socially responsible way to underserved markets in sub-Saharan Africa – there is a need to ensure their ML models and workflows are compliant with both internal business stakeholders and external auditors.
The M-KOPA data science team plays a significant role in driving pricing and strategic decisions, assessing risk on retention products and debt facilities, and improving predictions on credit loss and customer upsell potential. Given the importance of their ML work in managing the company’s portfolio, using the Weights & Biases Model Registry as a centralized hub and single source of truth has been a game-changer for the M-KOPA ML team.
W&B Registry as the Model Management Hub
“Before using [W&B Model Registry], we mostly managed tracking forecasting models in adhoc Jupyter notebooks, or READMEs, or in an Excel spreadsheet, or even just in my head,” said David Clarance, Staff Data Scientist at M-KOPA. “It was not a great place to be.”
The team wanted to improve the tracking and overall workflow management for their production models, while maintaining a strong audit trail and building lineage – the tractability and reproducibility of a model with details on how and why it was developed. They also wanted to reduce the burden of institutional knowledge living among individuals, and improve collaboration across their ML development and production teams, and corresponding functional stakeholders.
The team started out by organizing their existing experiment tracking into three different types of runs: a) A production forecasting task that uses the approved production model to generate the daily forecasts used by the business. b) A scheduled automated retraining task that creates a “challenger” model using the most recently available training data. This model is promoted to “QA” if specific validation criteria are met. c) A QA forecasting task that uses the QA model to generate daily forecasts and inform manual decisions about whether the QA model can be promoted to production.
This diagram below illustrates the workflow. Since all the types of runs already existed – with the team tracking all experiments in Weights & Biases – it was seamless for them to get up and running with the model registry.
Without a centralized hub for model management, the process of tracking and promoting production models was manual and potentially error-prone. Model roll out was delayed, and the overall process between the ML team and their business stakeholders was not as seamless as it should have been.
Today, the entire process is managed through W&B Registry, via Protected Aliases. Protected aliases represent key stages of the M-KOPA model development pipeline – such as staging, QA, Challenger, or different model version numbers. Every day, the model version tagged with “QA” will get promoted to a QA environment for human evaluation, providing the team with a much clearer sense of what’s in production at all times, what model version was previously used, and what’s potentially coming next.
“Using the W&B Model Registry for model promotion has been extremely useful,” said Benedict Eugine, Data Scientist at M-KOPA. “Having the full history of the model available also makes it very easy for other departments – like product management – to help manage the deployment process. It’s very easy to use, even for someone not as technical as a developer.”
With access available to many individuals across different departments, having role-based access controls (RBAC) – in addition to the Protected Aliases – is a necessary level of workflow management. No company, certainly not one working with financial models, can risk the wrong person accidentally deploying the wrong models to production, so only assigned Registry admins have access to change the model’s protected alias to “Production”.
The team also currently maintains a manual check for the final step before production. In the pipeline for training and validation, a trained challenger model will go through validation checks and if it passes, will get promoted to QA. If those validation checks fail, it will get tagged accordingly. The ML team works with stakeholders in the credit department to go through the appropriate finance checks and once they’re happy with it, they will manually tag the QA model as a production model – with their appropriate registry admin permissions – and the cycle continues. Manual checks are required because of the financial nature of the models and associated regulations. For low-impact models, the team hopes to automate the entire process, including passing/failing validation checks and subsequent model promotion via protected aliases.
Building trust to redefine finance and drive inclusivity
M-KOPA has a dedicated mission to ensure fairness for their customers and transform financial access in sub-Saharan Africa, while also facing scrutiny from auditors and financial controllers on models and ML activities. W&B Registry, and especially its Action History feature in tracking all details for all aliases throughout history, has been particularly useful in this area.
“Building trust internally is always a hard thing for data scientists,” said Rosina Norton, Head of Data Science at M-KOPA. “A lot of the things we do are quite obscure, so trust has to be built through concrete demonstration. Having a system like Weights & Biases in place, designing processes to answer their questions, really helps build that trust.”
And with streamlined workflows, improved lineage tracking, and better collaboration with their business counterparts, the ML team at M-KOPA is able to deliver the most sophisticated financial models to provide more affordable and flexible financing options to underserved populations. Weights & Biases, and especially the Model Registry, is the key to tying their ML processes together.
“The goal of the company has always been about enabling financial and digital inclusion,” said Norton. “It’s not about how we can maximize the profit of our portfolio, but instead how we maximize the number of people we can help.”