MLOps vs. DevOps

Do you struggle with maintaining the performance of your Machine Learning model in the long term? Are you a DevOps engineer who wants to integrate Machine Learning into the DevOps workflow? Do you wonder about how MLOps differ from DevOps?

MLOps is a set of practices that apply DevOps principles to Machine Learning problems to ensure the creation of Machine Learning products that you can verify, trust, and maintain in the long term. Including MLOps practices into the software development cycle becomes more and more necessary as Machine Learning gains more and more roles in software development.

In this article, you will learn how MLOps solutions can validate your Machine Learning model against common issues like data drift and concept drift and how it extends current DevOps practices. Particularly:

• DevOps and MLOps defined;
• Their similarities and differences;
• The specific aspects where MLOps significantly differ from MLOps and require a different approach.

Let’s start with an overview of DevOps.

Testing. CI/CD. Monitoring.

Because ML systems are more fragile than you think. All based on our open-source core.

Deepchecks HubOpen Source

DevOps is a software development framework to increase the speed and reliability of large-scale software system development and operation. It combines software development, testing, and operations practices into a single central repository and implements quality assurance checks with automatic feedback loops.

Core building blocks of DevOps are automated steps to merge and test code at different stages of the development cycle.

A common approach to group these steps is to separate code integration from delivery:

• Continuous Integration. Integrate code changes and tests via automated processes into a central codebase.
• Continuous Delivery. Automates and combines software testing, staging, and deployment.

Typical DevOps tools are git, Jenkins, Jira, Docker, and Kubernetes.

The benefits of DevOps:

• Rapid development cycles. The frequent updates and built-in checks increase confidence in the code, allow quicker improvements, and increase deployment velocity.
• Agile planning. Quick iteration allows faster response to change requests and testing of ideas.
• Clean responsibilities. The automated cycles free software developers from doing operations and pass around and test code “manually.”
• Cross-team work. The centralized but separated areas and stages allow different teams to contribute and review updates together.

MLOps manages Machine Learning model deployment in production environments by integrating Machine Learning into the DevOps cycle.

There are many shared elements between DevOps and MLOps:

• Code and component testing
• Automation
• Continuous integration of code
• Continuous delivery into production
• Cross-team collaboration
• Integration of feedback loops

Because of these similarities, people sometimes refer to MLOps as “Data Science Ops” or “DevOps for Machine Learning”.

While this view is partially valid, this article shows that MLOps require additional considerations compared to DevOps. These characteristics place MLOps at the intersection of Machine Learning, DevOps, and data engineering.

As two Google engineers expressed in a paper:

(source)

To understand the specific characteristics of MLOps, we need to understand the Machine Learning life cycle.

The Machine Learning life cycle describes the way data scientists and Machine Learning engineers build Machine Learning models and put them into production:

Here are its main elements:

• Data collection
• Exploration and analysis
• Preprocessing
• Model training
• Evaluation
• Service deployment
• Monitoring
• Retraining, reevaluation, and redeployment

A simple representation of the Machine Learning life cycle (source)

The Machine Learning life cycle has many elements a typical software development process does not, like working with models (instead of just code), the exploratory and experimental aspects of data science, and the requirement of frequent training.

The challenge of MLOps is to apply DevOps principles to Machine Learning projects.

(source)

Implementing MLOps can have multiple benefits for your Machine Learning project:

• Faster and more consistent automated deployment cycles;
• Higher reliability and confidence in model results;
• Increased specialization of team members and cooperation within and between teams;
• Automatic integration of MLOps best practices;
• Ability to address regulatory and compliance requirements.

We have overviewed what DevOps is and gained a high-level understanding of how MLOps differs from it. In the next section, we will drill down to examine their main differences.

Because of the specific attributes of the Machine Learning life cycle, MLOps require substantially different practices from what we expect in DevOps. Here is a list of areas where these differences are particularly true:

• Model components
• The model training process
• Exploration and experimentation
• Input validation
• Output validation
• Skills and team composition

We will discuss each topic in detail.

If you need a quick overview, this table lists those DevOps attributes where MLOps raises additional requirements:

A Machine Learning model consists of more than just code as it also relies on data, metadata, configuration, parameters, and logs. This composition has several consequences:

• You have to maintain the model and the data simultaneously.
• The performance of a Machine Learning model depends on both data and code.
• With Machine Learning projects, the build trigger can be both code change or rerunning the same code with new data.
• Training a model and exposing it as a service are two separate stages of their life cycle.

Model outcomes rely on incoming data. Real-world events can change the input data in unpredictable ways and reduce model performance.

(source)

Validating data is critical in Machine Learning, as your model’s performance depends on the quality of data you put into it.

Data validation is different from testing. It requires constant monitoring for common cleaning issues (e.g., missing values, types, and value ranges). For modeling purposes, it also needs to check for data drift, the change of attributes important for modeling (distribution, central tendency, and variation metrics, correlations, etc.).

Here are some common data validation areas:

• Data consistency;
• Data validation (missing values, types, value ranges);
• Data shape (distribution, measures of central tendency, skew, and variation);
• Changing data profiles, data, and schema validation.

Let’s take an example!

Assume that we want to work with the sample flight sample dataset that tells the monthly number of passengers of an airline over 12 years.

import seaborn as sns

df = sns.load_dataset("flights")

df.head()

If we want to train a model on this dataset and then apply the model for new batches, we need to ensure that the data is consistent and has no errors.

The specific attributes we want to validate depend on our use case. Below are a few examples:

Checking data columns:

assert df.columns.tolist() == ['year', 'month', 'passengers']

Checking whether all the years have 12 records for each month:

assert all(df.groupby('year').apply(len) == 12)

Checking whether there are any missing values in the data:

assert not df.isna().any().any()

These are simple examples, and MLOps software solutions and platforms offer a greater range of data validity checks that you can apply to your Machine Learning project.

Training models is a different type of activity than writing and deploying code. It requires special hardware and longer time cycles. It can happen both in real-time and “offline,” requiring different deployment and monitoring approaches.

Because of the frequent need for retraining, models are not finished products, making it necessary to be intentional about it and introduce Continuous Training components into the DevOps cycle.

Data exploration and model experiments are core parts of the Machine Learning life cycle. These elements introduce a further level of uncertainty in terms of resource needs and outcomes.

Lots of this type of work happens in jupyter notebooks, which introduce its specific software development challenges.

While software works with explicit rules and relatively clear input-output patterns, Machine Learning models treat problems numerically and produce less clear outcomes. Validating their results requires a different approach.

For this reason, model validation is a core element of MLOps. We can validate our models to ensure they do what we intend to do and their performance does not degrade in time with new data (Concept Drift).

For example, we can monitor whether the performance of our new models (or of our old model with new data) meets our original model performance within a threshold:

assert new_accuracy >= old_accuracy * (1 - performance_threshold_value

Overfitting is another common issue for which we want to look out. We can introduce a check to alert us when our model test substantially performs behind training.

assert test_accuracy <= train_accuracy * (1 - overfit_threshold_value

Performance monitoring is different from code testing because it is more business-specific and requires more context-based error levels. Because of the frequent need to retrain models, it introduces the requirements of Continuous Validation.

MLOps problems require new skill sets. Depending on your organization’s size, this may require skilling up your data scientist with code modularization, testing, and versioning or building a hybrid team of data scientists, data engineers, and DevOps engineers.

(source)

Let’s conclude this discussion on their differences by reviewing your options to implement MLOPs in your Machine Learning project.

Without dedicated MLOPs in place, maintaining the life cycle requires extra effort. This work either manifests in the overburdening of data scientists with operational and engineering tasks or the manual handover of models between data scientists and ML engineers.

Imagine you want to cover every element of your Machine Learning life cycle in scale with a continuous inflow of new and changing data, new regulations, and a shifting technological landscape.

Instead of overstretching your current team’s capabilities, a practical option is to use MLOps platforms like Deepchecks. Such services can spare you from inventing everything from scratch and provide you ready-proven, reliable MLOps solutions that follow the best practices.

With Deepchecks, you can assure Continuous Validation of the whole Machine Learning life cycle. This includes:

• Data monitoring and alerting in case of data drift;
• Checking against data leakage by maintaining best practices in sampling and protecting the training process from label information;
• Validation of Machine Learning performance measures, protection against concept drift;
• Data integrity alerts to ensure a consistent data schema and correspondence between training and production data sets.

Do you want to learn more? Be sure to check out our use cases.