Tips for Improving Machine Learning Models

As the application of machine learning (ML) models and technologies spread across disciplines, it becomes imperative to efficiently deduce real problems that can be solved with ML, design solutions, and develop effective models to solve those problems. Since there is a generally accepted framework for developing and deploying models (i.e., the ML project lifecycle), there are some best practices that apply across the board for increasing model performance. However, because ML has many applications, improving machine learning models might vary slightly depending on its use.

This article examines those practices and tools, from problem discovery to the post-deployment phase.

The ML lifecycle is a framework that guides the development and deployment of ML models. For the purpose of this article, it is divided into 4 stages:

1. Identifying the Problem and Goals through Research
2. Data gathering and Preparation
3. Model Development, Training, and Evaluation
4. Deployment and Post-Deployment

Fig 1. Image showing the ML lifecycle for ML solutions : Source

It should be noted that every step in this cycle is crucial to the success of your ML project, meaning that every step needs to be critically examined and executed properly to ensure optimal model performance. The goal of improving a model is to make sure that model performance, which is calculated with machine learning metrics (like accuracy, precision, and F1 score), is optimal with a relatively high confidence in the ability of the model to generalize correctly. The tips and best practices mentioned for each stage essentially  increase the chances of improving the model performance.

• Create a well-defined problem statement with clear objectives for your model.Understand the problem you are trying to solve and how the objectives will be measured. Make sure that you can evaluate its performance effectively. This can be done by contacting industry experts in the use case you want to implement, or reading academic papers if necessary to clarify the problem and the design of the model so that it is well-suited to the task. This stage also involves understanding the data with the help of domain experts if necessary.
• Identify the possible ML techniques that are the most appropriate for your use case and the data it will utilize. Clarify the strengths and weaknesses of the possible algorithms that might be used for your use case. Consider the accuracy and performance requirements along with constraints to the development process to help you identify the best approach for your project based on its requirements and budget.

• Machine learning models are only as good as the data they are trained on, so it is important the dataset you are using is relatively large, diverse, and representative of your use case. Depending on the algorithm utilized, you’ll have to determine the quantity of data needed through research. It will allow the model to generalize well on new data and improve model performance.
• Use data engineering or MLOps tools like Great Expectations and Deepchecks  to detect data integrity and accuracy issues. For example, invalid data, duplicate values, null values, data type mismatches, schema, and data distribution shifts.

Testing. CI/CD. Monitoring.

Because ML systems are more fragile than you think. All based on our open-source core.

Deepchecks HubOpen Source

Checking for Data Integrity using Deepchecks

Note: This code example uses Python programming language.

Installing Deepchecks and loading data:

# If you don't have deepchecks installed yet, run:
import sys
!{sys.executable} -m pip install deepchecks -U --quiet

# or install using pip from your python environment

from deepchecks.tabular import datasets

# load data
data = datasets.regression.avocado.load_data(data_format='DataFrame', as_train_test=False)

Defining a Dataset Object:

from deepchecks.tabular import Dataset

# Categorical features can be heuristically inferred, however we
# recommend to state them explicitly to avoid misclassification.

# Metadata attributes are optional. Some checks will run only if specific attributes are declared.

ds = Dataset(dirty_df, cat_features= ['type'], datetime_name='Date', label= 'AveragePrice')

Running Deepchecks full tabular data test suite:

from deepchecks.tabular.suites import data_integrity

# Run Suite:
integ_suite = data_integrity()
suite_result = integ_suite.run(ds)
# Note: the result can be saved as html using suite_result.save_as_html()
# or exported to json using suite_result.to_json()
suite_result.show()

This produces an output with tests that either “Didn’t Pass”, “Passed”, or “Didn’t Run”. It exhaustively does a data integrity check on your tabular data with a few lines of code. The results are shown in this quickstart guide for data integrity checks.There are other checks for computer vision use case:

• Consider augmenting your data to improve model performance. For a computer vision or  Natural Language Processing (NLP) project, you can add different corruptions or slight changes and perturbation (adversarial) samples to make your model more robust.
• Check the validity of your train-test split by looking out for any data leakages and verifying that there are no significant differences in the distribution of features or labels between the training and testing datasets. Additionally, it is important to compare the integrity and distributions of the data batches that are entering the target system to ensure that they are consistent with the original datasets. This will help to ensure the accuracy and reliability of your model.

• After evaluating the performance of the model on the split datasets. Identify weak segments of data (such as demographics or location depending on use case) and evaluate any correlations. After making necessary adjustments, retest the model to determine if there are any improvements in performance.
• Use cross-validation to evaluate the effectiveness of the model. Cross-validation requires splitting the training data into different sets, training the model on each set, and assessing the model’s performance on them. This can aid in improving your understanding of the model’s performance and allowing you to spot any potential problems.
• Perform and error analysis to determine whether bias and variance in the data may be impacting the model’s performance. This involves breaking down and observing erroneous predictions to give you a better understanding of why the model’s performance is low.
• Adjust the model’s hyperparameters to improve performance. Hyperparameters are configuration options that affect how the model behaves and performs. The model can be optimized and its performance enhanced by experimenting with various options.
• Regularization can be used to avoid overfitting. When a model is overly complicated, it tends to learn the noise in the data rather than the underlying patterns, which is overfitting. Regularization is a method that introduces a penalty for model complexity to minimize the chances of or prevent overfitting.
• Depending on the machine learning problem and the nature of your model, ensemble learning can be used to combine multiple models and help improve model performance and reduce overfitting.

• Use an open source or commercial model management platform based on your project specifications and budget to not only track but monitor the performance of your model. Log your model’s input and output data as well as any major changes made to the model to increase transparency and help you troubleshoot with ease when you discover problems.

Fig 2. – Model development and deployment process: source

• Retrain as much as you need to. Machine learning models need to be periodically evaluated and adjusted as new data becomes available because they are not static. Along with tracking the model’s performance, it’s critical to retrain it as required.
• Set up tests at every stage of the ML lifecycle to detect errors before they become a problem in future. Think about using tests like, Invariance Test (INV), Minimum Functionality Test (MFT) or Directional Expectation Test (DET) among others to test the behavior of your model and tweak if necessary.
• Automate your ML project using Continuous Integration/Continuous Deployment (CI/CD) practice which can allow you to iterate on your model and improve faster, test and evaluate various model versions to find the best performing one to save time. It enhances collaboration by automatically integrating model changes in a shared codebase to develop and improve your model’s performance efficiently.

Data integrity – Checks the quality and accuracy of your data.
Tools: deepchecks (ML specific) or great expectations, pandera, anomalo (not ML-specific)

Experiment trackers – These enable you save and compare results.
Tools: Weights and Biases, MLflow, ClearML

Python testing frameworks – Mainly for building unit tests.
Tools: pytest, hypothesis (property-based tests)

ML-focused testing – Framework and algorithms for testing ML-related issues.
Tools: deepchecks (comprehensive ml tests and framework), Checklist (behavioral tests for NLP), Reclist (behavioral test for recommendation systems)

Workflow management platforms – Used for automating runs.
Tools: Airflow, Prefect

CI / CD – Running automatic checks (on model update, pre-deployment).
Tools: Github Actions, Circle CI, Jenkins.

In summary, improving accuracy of machine learning models is not straightforward since issues can arise from any part of the ML system. Sometimes, the ML problem identified and the approach taken might be at odds or the sourced data may not be clean enough to derive value from it or the ML model training techniques selected might be faulty. These tips serve as a reminder of the things you can do to improve your model when working on developing an ML solution. Remember that the ML lifecycle is a cyclic process so if your model performs poorly at any stage, you can always backtrack to the stage in the process, crosscheck, and modify your model design in order to improve the model’s performance.