How to Improve the Performance of Your ML Model

Machine Learning (ML), a subset of artificial intelligence (AI), is often used in today’s businesses. With the fast development of new ML algorithms, more affordable computing power, and greater data availability, the number of use cases has grown exponentially. ML applications are suitable for every industry, especially in the financial and healthcare industries. In order to have an appropriate and accurate model, ML model performance evaluation and improvement must be done regularly. This article offers suggestions for exactly that.

Three directions for ML model performance improvement are presented here: tuning model parameters,  improving data, and selecting a better algorithm.

Tuning is about selecting the best set of parameters and hyperparameters for a learning algorithm; all optimal Machine Learning algorithms maintain it. A parameter can be regarded as an  internal element of the model, whose value is tuned after the model has learned from the data. Regression coefficients in linear regression, support vectors in support vector machines, and weights in neural networks are examples of parameters that can be tuned. By that logic, we consider a hyperparameter as an external element that is set arbitrarily by the developer (manually). There are procedures used to tune hyperparameters automatically, which will be explained below. Examples of hyperparameters include the k in k-nearest neighbors, the number of trees and maximum number of features in a random forest, the learning rate and momentum in neural networks, and the C and gamma parameters in support vector machines.

Two standard hyperparameter tuning techniques are Grid Search and Random Search. Performing a  grid search involves the composition of a grid of possible hyperparameter values whereby models are iteratively built for all hyperparameter combinations in a brute-force manner. In a random search, not all hyperparameter combinations are used. Instead, each iteration uses a random hyperparameter combination.

Grid and random search (Source)

Besides these two algorithms, a stochastic optimization method could also be used for hyperparameter tuning, which will automatically navigate the hyperparameter space as a function of the loss function (i.e., the performance metrics) in order to monitor the model performance.

Sometimes, you can save time and energy by evaluating pre-trained models instead of training a baseline model from scratch. Pre-trained ML models are ready-to-use models usually trained on a large dataset and can be quickly deployed. There are numerous sources of pre-trained models such as Kaggle, Github, or APIs from companies such as Microsoft Azure or Google Cloud, as well as specialized startups such as Scale AI, Hugging Face, and Primer.ai. You can also look into cutting-edge AutoML technology for developing custom ML models. AutoML is an excellent solution for businesses with limited organizational knowledge and resources that want to deploy ML models at scale to meet their business needs.

Data quality evaluation is necessary for ML. More often than not, improving the training data quality and quantity provides a more robust model performance. Some common techniques for data improvement are data annotation and augmentation, handling imbalanced data, missing values, and outliers, which will be briefly described in this section.

The need for well-annotated training data typically bottlenecks ML model enhancement. The expense of annotation (time, money, and subject matter expertise) hinders us from creating sizable labeled training datasets.

Data augmentation techniques can be used to scale the training dataset and improve it where the type of data influences the choice of a specific technique. For instance, augmenting an image involves changing aspects like its brightness, color, hue, orientation, and cropping.

Image augmentation (Source)

Text can be augmented using techniques such as regex patterns, templates, synonym and antonym substitution, back translation, paraphrase generation, or text generated using a language model.

Another standard scenario in which models underperform is when data across categories of interest are imbalanced. Imbalanced data generally refers to an issue with classification problems where the classes are not expressed equally. Upsampling and downsampling data and techniques like SMOTE help balance the distribution of the dataset in the instance of skewed data distribution. Upsampling involves injecting synthetically produced data points into the dataset. Downsampling lowers the number of training samples.

Handling missing values and outliers improves your data and increases the accuracy of ML models. One way to identify outliers is by using summary statistics like the mean or standard deviation. Your models’ accuracy will likely suffer if your data has missing values or outliers.  To solve this issue, you can eliminate the data points with missing values or outliers from your training dataset, impute the missing values using methods like k-nearest neighbors or linear regression, or use a technique like bootstrapping to reduce the impact of the outlier data.

Testing. CI/CD. Monitoring.

Because ML systems are more fragile than you think. All based on our open-source core.

Deepchecks HubOpen Source

Algorithms are in charge of “converting” data sets into accurate approximations of future states and variables of interest. Choosing the correct algorithm for the specific problem you want to resolve is critical for model performance. The numerous types of ML algorithms make it challenging to determine which are appropriate for your data. In an ideal case, any work related to developing an ML algorithm would be preceded by a detailed examination of the problem at hand, including a clear description of the use case and business goals. We recommend starting by analyzing the problem, determining the type of the problem, accordingly creating a list of algorithms that can be used and, as a final step, running your data through the models and comparing their performance.

ML algorithms (Source)

A helpful procedure for selecting a better ML algorithm is to use cross-validation with multiple algorithms on the same dataset and then compare their accuracy scores. This method involves training several ML models on the available input data subsets and then evaluating them on the complementary subsets. While training data conditions your algorithm to recognize patterns, testing data evaluates its accuracy on unknown data.

An ensemble approach, which combines two or more algorithms into one model, is another helpful method. Ensembles are often more accurate than particular algorithms because they combine their strengths and counteract their weaknesses.

In order to optimize the selected ML algorithm, overfitting and underfitting measures for the model performance monitoring can be very useful.

Underfitting happens when an ML model fails to capture the underlying trend of the data. The training and test errors are high in this case. It usually occurs when we want to use a simple model on complex data, and consequently, the model cannot find complex patterns in data. Some approaches to reduce underfitting are increasing model complexity, increasing the number of features, and removing noise from the data.

An ML model is said to be overfitted when it fails to make accurate predictions on testing data. Here, the training error is low, but the test error is high. Overfitting can be fixed by using more training data, reducing model complexity, early termination of the training phase, and introducing the validation set.

Underfitting and overfitting (Source)

The accuracy of your ML models can be increased in various ways. Using techniques like changing parameter values, experimenting with multiple algorithms, and improving data significantly increase your chances of developing an accurate model. Your models will perform better the more you understand your data and the algorithms you use.