As ML models become increasingly popular in a variety of industries, the field is reaching a certain maturity stage where businesses are starting to process that with the “great power” of AI and ML technologies comes “great responsibility” and great risk as well. Trusting ML models blindly can cause businesses great losses, and thus methods for actively assessing and minimizing risk are essential.
What Is Model Risk Management?
Model risk management (MRM) is a term that comes from banking regulations. In 2011 the OCC released a supervisory guidance stating:
“All banks should confirm that their practices conform to the principles in this guidance for model development, implementation, and use, as well as model validation.”
When applied to general ML models, the model risk management framework is composed of steps and practices to assess and minimize risk of these models. This requires exploring different worst-case scenarios for each model and assessing the probability for such scenarios alongside the value that is gained from the model when it’s working properly. Furthermore, MRM aims to plan steps and delegate responsibility for each such scenario. When done properly, your business will be more prepared to handle edge cases and minimize potential catastrophes caused by poor ML model performance. Following are some of the top steps we recommend for proper model risk management.
Mapping out potential risks and categorizing them is crucial for proper risk management. High impact together with high probability is the most crucial risk (source)
Perhaps the first step toward proper model risk management is continuous monitoring and evaluation of ML models in production. The longer a model goes without up-to-date evaluation, the higher risk it imposes. ML models are often susceptible to significant decrease in performance due to phenomena like data drift and concept drift, this is also called model staleness. Evaluating your model’s performance regularly on the most recent data will ensure that you are basing your risk assessment on relevant information that is not outdated. Additionally, continuously monitoring incoming data and detecting of drift or data integrity issues early on is essential for risk management.
Independent Model Validation
When it comes to ML model validation, it is almost always the same team that develops the model and then validates and evaluates its performance. This may pose a conflict of interests, since the party that developed the model is incentivized to be optimistic about the performance. Thus, it is good practice to have an independent party with a relevant technical background that is involved in the validation and evaluation process of your ML models.
During model validation, we would like to get a fine-grained assessment of the model’s accuracy across different data slices, approximate the probability for errors like false positives and false negatives while keeping in mind the impact of each type of error. In many cases creating a false positive can have a much more drastic outcome than a false negative or vice versa. Thus, during the validation process we might reveal that the chosen decision threshold for a classification model is not aligned with the business goals, and the benefits and risks that lie in the background.
Alignment Across the Board
Compliance with regulations and model governance and control policies should be a well-known objective to all parties involved in the model’s development, implementation and use. All too often there is a separation between the data science team who are more focused on the “theoretical” performance of the model, and the business stakeholders who are focused more on business value and compliance with regulations.
Setting Triggers For Action
A large part of model risk management is being prepared to intervene when something goes wrong as quickly as possible. Such interventions might include retraining or replacing a model, or temporarily disabling it to avoid additional loss. In order to detect issues early on, it is helpful to set up automatic triggers. Thus, there is a need to translate terms like business KPIs and value estimates to metrics such model accuracy, precision and recall. You may want your model to be disabled automatically when crossing a specific threshold, so that you do not rely on it.
Understand Your Model
A key component in reducing any kind of risk, is deepening your understanding of the object in play. When it comes to ML models, this includes practices for explainable and interpretable predictions, which give insight to the inner workings of your model. Furthermore, a thorough error analysis of your model will give you a good idea as to the model’s strengths and weaknesses. Thus, you will be more prepared for potential issues, and you’ll be more likely to handle and debug these issues efficiently.
To conclude, model risk management for machine learning is a set of practices that will help your company be better prepared for potential worst-case scenarios. Many companies are still in the stage where they are blinded by the success of ML models, but without proper risk management and preventative steps, there is significant risk to lose just as much as you might’ve gained from using these sophisticated models. Third party solutions such as Deepchecks can assist in setting up a model risk management policy seamlessly and efficiently.