How to Test Machine Learning Models

Testing in ML models should be concerned with reliability, robustness, and fairness. The significance of thorough testing is hard to overstate as it helps in:

• Preventing adversarial attacks: Testing models can help detect possible adversarial attacks. Rather than letting this attack happen in a production environment, a model can be tested with adversarial examples to increase its robustness prior to deployment.
• Ensuring data integrity and preventing bias: Data collected from most sources are usually unstructured and might reflect human bias that can be modeled during training. This bias might be against a particular group either by gender, race, religion, or sexuality, with varying consequences in society depending on the scale of use. During the evaluation, bias can be missed because it focuses mostly on performance and not the behavior of the model, given the role of the data in this case.
• Spotting failure modes: Failure modes can occur when deploying ML systems into production. These can be due to performance bias failures, robustness failures, or model input failures. Some of these failures can be missed by evaluation metrics, although they can signal problems. A model with an accuracy of 90% means that the model is finding it difficult to generalize with 10% of the data. That can prompt you to check the data and look for errors, giving you better insights on how to solve it. It is not all-encompassing, so structured tests for possible scenarios that may be encountered need to be established to help detect failure modes.

Photo by ThisIsEngineering

This article demonstrates how testing in machine learning differs from testing “normal” software and why evaluating model performance is not enough. You will learn how to test machine learning models and which principles and best practices you should follow. We will also discuss the ethical problems that must be considered in the testing environment to ensure that the models are unbiased and open and adhere to legal requirements.

Software developers write code to produce deterministic behavior. Testing identifies explicitly which part of the code fails and provides a relatively coherent coverage measure (e.g., lines of code covered). It helps us in two ways:

• Quality assurance: whether the software works according to requirements
• Identify defects and flaws during development and in production.

Data scientists and ML engineers train models by feeding them examples and setting parameters. The model’s training logic produces the behavior. This process poses these challenges when testing ML models:

• Lack of transparency: Many models work like black boxes.
• Indeterminate modeling outcomes: Many models rely on stochastic algorithms and do not produce the same model after (re)training.
• Generalizability: Models need to work consistently in circumstances other than their training environment.
• Unclear idea of coverage: There is no established way to express testing coverage for machine learning models. “Coverage” does not refer to lines of code in machine learning as it does in software development. Instead, it might relate to ideas like input data and model output distribution.
• Resource needs: Continuous testing of ML models requires resources and is time-intensive.

These issues make it difficult to understand the reasons behind a model’s low performance, interpret the results, and ensure that our model will work even when there is a change in the input data distribution (data drift) or in the relationship between our input and output variables (concept drift).

Many practitioners may rely solely on machine learning model performance evaluation metrics. Evaluation, however, is not the same as testing. It is important to know the difference.

Table 2: Comparing model evaluation and testing

ML model evaluation focuses on a model’s overall performance. Such evaluations may consist of performance metrics and curves and, perhaps, examples of incorrect predictions. Model evaluation is a great way to monitor your model’s outcome between different versions. Remember that it does not tell us a lot about the reasons behind the failures and the specific model behaviors.

Model development and evaluation workflow, Author

For example, your model might suffer a performance drop in a critical data subset while its overall performance doesn’t change or even improve. In another case, model retraining based on new data might not produce a performance change but instead could introduce unnoticed social biases towards a specific demographic group.

Machine learning tests, on the other hand, go beyond evaluating the models’ performance on subsets of data. It ensures that the composite parts of the ML system are working effectively to achieve the desired level of quality results. You can say that it helps teams point out flaws in the code, data, and model so they can be fixed.

Testing is not easy, and testing machine learning models is even harder. You need to prepare your workflow for unexpected events while working with dynamic inputs, black-box models, and shifting input/output relationships.

Photo by ThisIsEngineering

For this reason, it is worth following these established best practices in software testing:

• Test after introducing a new component, model, or data and after model retraining.
• Test before deployment and production.
• Write tests to avoid recognized bugs in the future.

Testing ML models has additional requirements. You also need to follow testing principles specific to the ML problem:

• Robustness
• Interpretability
• Reproducibility

Robustness

Robustness requires your model to produce a relatively stable performance, even in the case of radical real-time change of data and relationships.

You can strengthen robustness in the following ways:

• Have a machine learning procedure that your team follows.
• Explicitly test for robustness (e.g., drift, noise, bias).
• Have a monitoring policy for deployed models.

Robustness checklist, Author

Interpretability

Maintaining interpretability makes you understand specific aspects of your model:

• Whether the model predicts outputs as it should (e.g., based on human evaluators).
• How input variables contribute to the output.
• Whether the data/model has underlying biases

Interpretability checklist, Author

Reproducibility

Model changes occur due to parameter adjustments, retraining, or new data, and to scale the model in production, no matter the platform it is used on, you need to ensure that your results are reproducible.

Reproducibility has many aspects. Although this is not an article on the subject, here are some tips to ensure that your model is reproducible:

• Use a fixed random seed by a deterministic random number generator.
• Make sure that the components run in the same order and receive the same random seed.
• Use version control even for preliminary iterations.

Reproducibility checklist, Author

Many existing ML model testing practices follow manual error analysis (e.g., failure mode classification), making them slow, costly, and error-prone. A proper ML model testing framework should systematize these practices.

Typical ML development workflow with tests, jeremyjordan

You can map software development test types to ML models by applying their logic to ML behavior:

• Unit test: Check the correctness of individual model components.
• Regression test: Check whether your model breaks and test for previously encountered bugs.
• Integration test: Check whether the different components work with each other within your machine learning pipeline.

Specific testing tasks can belong to different categories (model evaluation, monitoring, validation), depending on your specific problem case, circumstance, and organization structure. This article focuses on tests specific to the machine learning modeling problem (post-train tests), so we do not cover other test types. Make sure that you integrate your ML model tests into your wider ML model monitoring framework.

For code, you can write manual test cases. This is not a great option for machine learning models as you cannot cover all edge cases in a multi-dimensional input space.

Instead, test model performance by doing monitoring, data slicing, or property-based testing targeted at real-world problems.

You can combine this with these test types that examine specifically the internal behavior of your trained models (post-train tests):

• Invariance test
• Directional expectation test
• Minimum functionality test

We will discuss each type below. If you are interested in an overview of approaches to machine learning model testing, check out this post.

• Invariance Test: The invariance test defines input changes that are expected to leave model outputs unaffected. The common method for testing invariance is related to data augmentation. You pair modified and unmodified input examples and see how much this affects the model output. One example is to check whether a person’s name affects their health. Our default assumption can be that there should be no relationship between the two. Having a test failing based on this assumption might imply a hidden demographic connection between name and height (because our data covers multiple countries with different names and height distributions).

Photo by Khuyen Tran

• Directional Expectation Test: You can run directional expectation tests to define input distribution changes and expected effects on the output. A typical example is testing assumptions about the number of bathrooms or property size when predicting house prices. A higher number of bathrooms should mean a higher price prediction. Seeing a different result might reveal wrong assumptions about the relationship between our input and output or the distribution of our dataset (e.g., small studio apartments are overrepresented in expensive neighborhoods).
• Minimum Functionality Test: The minimum functionality test helps you decide whether individual model components behave as you expect. The reasoning behind these tests is that, overall, output-based performance can conceal critical upcoming issues in your model. Here are ways to test individual components:
  • Create samples that are “very easy” for the model to predict in order to see if they consistently deliver these types of predictions.
  • Test data segments and subsets that meet specific criteria (e.g., run your language model only on short sentences of your data to see its ability to predict short sentences).
  • Test for failure modes you have identified during manual error analysis.

Table 1: Listing different test cases with their expected outcomes and actual model performance.

Software development tests often focus on the project’s code. However, this does not always work with ML workflows, as code is not the only element, and behavior does not map so clearly to pieces of code.

A more ‘behavioral’ way to organize a machine learning test is to focus on the “skills” we expect from the model (as suggested by this paper about testing NLP models). For example, we can check whether our natural language model picks up information about vocabulary, names, and arguments. From a time series model, we should expect to recognize trends, seasonalities, and change points.

Photo by Pavel Danilyuk

You can test these skills programmatically by checking for the above-discussed model properties (i.e., invariance, directional expectation, minimum functionality).

Testing a full model takes a lot of time, especially if you do integration tests. To save on resources and speed up testing, test small components of the model (e.g., check whether a single iteration of gradient descent leads to a decrease in loss) or use just a small amount of data. You can also use simpler models to detect shifts in feature importance and catch concept and data drift in advance.

Photo by Tima Miroshnichenko

For integration tests, have simple tests running continuously with each iteration and keep bigger and slower tests running in the background

To ensure the goals of ML testing, ethical considerations are important for social trust and regulatory compliance.

Bias Detection, Fairness, and Mitigation

Training data bias can contribute to the continuation or even introduce new social disparities. Deepchecks provides a comprehensive suite of tools to ensure the detection and mitigation of problems that might be inherent in the dataset and model, including offering to test a model after training, such as data integrity checks, model evaluation, and data and model validations from research to production. These suites of tools provide mechanisms on how to test a model after training, monitoring throughout the entire lifecycle of the model, and quicker root cause analysis. Other MLOps tools, as well as tools like IBM Fairness 360 and FairLearn, provide functionality to help manage bias and improve model fairness.

Deepchecks ML Testing, Try it out now

Transparency

Transparency is essential for responsible AI development and deployment. As Alexander Amini said, “We need the ability to not only have high-performance models but also to understand when we cannot trust those models.” To know when not to trust them, understanding how they arrive at certain predictions is required; this is where the concept of model interpretability begins. Interpretability yields transparency, and transparency maintains trust and ensures fair outcomes.

Techniques for Improving Transparency, which provide detailed explanations of model predictions

• SHAP (SHapley Additive exPlanations) provides
  

  SHAP analysis for diabetes predictions model, Datacamp

• LIME (local interpretable model-agnostic explanations)
  

  LIME analysis for diabetes predictions model, Datacamp

Privacy, PII Protection, and Regulatory Compliance

Protecting user privacy and personally identifiable information (PII) is paramount in any industry, including AI. Adherence to regulatory compliance is not a “can have” but a “must have,” as failure to do so can result in reputational damage and financial and legal repercussions.

PII Landscape, Data Privacy Manager

Some legal regulations include:

• GDPR and CCPA
• HIPAA for healthcare
• FCRA for financial services

It is worth noting that to ensure continuous compliance, regular audits and documentation should be carried out, and you should create and follow a compliance checklist for regulatory requirements to be followed systematically throughout the entire lifecycle of your ML project.

Accountability and Governance

To monitor and responsibly manage your ML project lifecycle, you need to create a data governance unit within your organization to build accountability and governance frameworks. This unit will be responsible for creating the rules and regulations for cross-functional teams working on your machine learning project at different phases, aiding adherence to regulatory compliance.

Photo by fauxels

Testing is an iterative process and can be difficult when working on ML projects that may require huge amounts of data along with long model training cycles. For small and large teams, time is an important resource that they do not have a monopoly on. This means that teams have to pick the best test frameworks that work for their unique situation. This might mean testing and validating random data samples in small batches or unit tests for testing specific behaviors of the model, among others that can be chosen.

Different teams, no matter their size, should integrate these practices into their overall ML project lifecycle to improve the overall quality of their product. Deepchecks is one of the best packages to start with, if quick data and model validation tests are desired for your project. Great Expectations is mostly for data quality, and both Deepchecks and Great Expectations can be integrated with typical testing packages such as the PyTest framework, which makes it easier to write and scale tests in ML applications.

So start testing today!