How to Build an Effective Machine Learning Infrastructure

An ML infrastructure is a sum of all the processes, tools, and resources required to develop, train, and maintain ML models at scale. It covers the full length of the Machine Learning workflow, enabling teams to access and manage all the processes and resources for an ML project.

Choosing to buy or build a Machine Learning infrastructure is motivated by boosting the efficiency of your data team. The right ML infrastructure will free up time for your team to focus more on understanding the data, creating models, monitoring their performance in production, and managing them. In essence, teams want to easily:

• Experiment and iterate on models quicker.
• Automate data quality management and version control processes.
• Integrate with tools to automate routine tasks.
• Support emerging ML tools and technologies.
• Scale effectively.

Organizations need ML infrastructures to scale products faster by increasing the efficiency of ML workflows and reducing the probability of human error as much as possible. Building an ML infrastructure is no trivial task and involves varying competencies and collaborative efforts. It also takes time and resources.

This article covers:

• Basics of ML infrastructure
• Building blocks of an ML infrastructure with reliable tools to utilize
• Additional considerations
• ML infrastructure challenges

Let’s go!

Source

Figure 1. ML building blocks

Now that we know what an ML infrastructure is, this section will give an overview of the different parts of an ML infrastructure and the tools that can be used at each point.

The major building blocks of an ML infrastructure are:

• Model Selection
• Data Ingestion
• ML Pipeline Automation
• Visualization and Monitoring
• Model Testing
• Deployment
• Inference

ML model selection is the process of choosing a final model that gives optimal performance for the problem your team set out to solve. A selection process goes beyond just looking for the model with the best fit. It is more nuanced and important, bearing significant impacts on your project if handled carelessly. Be vigilant with the model’s maintainability and complexity and deem models to be the best fit based on:

• Its performance; given the resources (memory, compute utilization) and inference time acceptable for the project.
• Performance compared to other models tested in the process.
• Requirements of the project stakeholders and the constraints surrounding the project.

Other considerations may include comparing performance and model complexity of various models to identify tradeoffs in relation to available resources. Note that the three above are necessities.

The process of developing and selecting models needs sufficient historical data to enable models considered to accurately generalize when new data is encountered. Utilizing the standard splits into train, validation, and test set, different models are chosen based on performance on the test set but it is a bit different in the real world, especially when a problem has insufficient data. Two classes of techniques can be employed in this case to give more insight into the model selection process:

• Resampling Methods: Checking the model’s performance on data samples it hasn’t been exposed to in the past.
  • K-Fold Cross Validation
  • Stratified K-Fold
• Probabilistic Measure: Checks model performance and its complexity.
  • Akaike Information Criterion (AIC)
  • Bayesian Information Criterion (BIC)
  • Minimum Description Length (MDL)

Figure 2. Techniques for Model Selection

Additional information regarding model selection can be found in this article.

Modern model file formats have been developed to allow practitioners to import and export models used with various libraries as a result of the interoperability of several frameworks.

Interoperability Tools:

These tools followi the model selection process and are not immediately apparent to the data science team, but they aim to ensure ease of usage across many platforms and frameworks.

• Model File Format:

File formats specify the model’s encoding and structure. These formats are employed when serving models.

Examples:
Open Neural Network eXchange

• Machine Learning Compilers:

To enable deployment of your model, ML compilers create a common representation before generating hardware code to run models on a certain platform. It serves as a bridge between frameworks and platforms. They can improve memory usage and speed.

Examples:
TensorFlow XLA, Glow, nGraph, and TVM

You can never overemphasize the importance of quality data. Businesses understand this and very often, extract, load, transform (ETL) pipelines are utilized. It processes the data from data sources to target locations, like “data lakes” or “data warehouses” for training models and improving model performance. Data can be ingested in real-time, batches, or by a hybrid approach of both. When choosing tools to utilize, consider the format of the data and features to be used, the size of the data, the frequency of ingestion, and the privacy of data.

Data can originate from various sources and be stored in cloud or on-premises warehouses. Tools used at this stage can enable teams to collaborate – one team member can share access to other team mates to contribute to the process. They can both see and work on all aspects of the data ingestion process.

Testing. CI/CD. Monitoring.

Because ML systems are more fragile than you think. All based on our open-source core.

Install Open SourceBook a Demo

Before starting a project, you should think of how data will be effectively managed throughout the project’s life cycle. Think through:

• The methods you’ll employ in gathering and storing your data;
• Practical applications of the data gathered;
• The goal of the data collected;
• Tools you’ll use for data quality management;
• Ways data will be shared between teams during development; and
• Security of the data.

Once you have, planning how each step of the ETL pipeline will work comes next.

Data Ingestion Tools:

• Storage Solutions:

With the help of these technologies, teams can quickly store and retrieve massive amounts of organized data. This helps the business run more smoothly and efficiently. You need this to scale your projects and do data versioning with ease.

Examples:
Amazon S3, Cassandra, and Azure Blob

• Database Management Systems (DBMS):

A database is a collection of organized business data, while a DBMS enables you to create, manage, and operate databases that contain the data. You’ll need a DBMS to run databases if you want to develop data-driven solutions.

Examples:
PostgreSQL, MongoDB and DynamoDB

• Scripting Languages for Processing Data:

Scripting languages help you process and derive analytics from data from a small scale to a large one. You can discover insightful information about the data you have using these tools.

Examples:
Python, R SAS, and Hadoop

• Data Warehouses:

They are used to store, process, and analyze large datasets to provide business intelligence. This is distinct from storage solutions since they allow teams to query and analyze data in addition to simply storing data. These are excellent for gathering streaming data and real-time analysis.

Examples:
Google Bigquery, Snowflakes, Amazon Redshift, etc.

Figure 3. Steps of Data Ingestion

The idea of creating an ML pipeline generally stems from the difficulty of scaling production-level applications within environments that do not support a continuous re-execution of all the processes that make the product functional. ETL, feature engineering, model training, evaluation, deployment, and monitoring pipelines are important to any ML project, and there are tools to automate all these processes.

Thinking of how interlinked each component is to the next is essential to understanding which parts to automate. Automation of the entire pipeline will enable you and your team to focus on high-level tasks. Tools mostly used (and liked) by your data team can be integrated into the ML infrastructure to automate the pipeline.

Figure 4. An ML pipeline

Pipeline Automation Tools:

The tools in this category manage and automate pipelines and workflows to support the model development process. It increases the efficiency of the model development process, allowing you to concentrate on other crucial project components.

• Open Source: Apache Airflow (Orchestration), TensorFlow Extended, and Kubeflow Pipelines
• Commercial: AWS Sagemaker, Azure ML Studio, Google Cloud, and IBM Watson Studio

In general, a lot of effort is put into ML projects. Who would want all that hard work to go to waste?

You have to visualize and monitor every possible process in order to ensure that your resources are not being wasted and the model is performing as expected. Machine Learning Infrastructure Monitoring enables practitioners to derive insights at a functional and operational level. You can monitor the health of your model and resource usage for the infrastructure. Visualization tools can be integrated at any point in the pipeline, depending on the processes important to you.

It is especially recommended to monitor the infrastructure usage at the training and deployment stages, among others you might want to watch (like the data ETL stages for data integrity). Your team should be clear on the stages to monitor and establish metrics to measure like GPU usage, server counts, accuracy, or ROC-AUC score, based on the different monitoring levels (functional or operational).

Monitoring Tools:

• Standalone tools: Deepchecks, WhyLabs, Neptune, Grafana, and Prometheus

We all agree that models should be tested after training. If you don’t agree, we would love to see your model performance reports for real-world data 🙂

CI/CD pipelines are used in scalable solutions, and models are tested along with the dataset and code that define the pipeline. Creating tests for the code, data, and models reduces the chances of overall failures. To track and evaluate the performance of the model during model testing, you might need to add monitoring, visualization, or, in certain circumstances, data analysis tools to your infrastructure.

Model Testing Tools: Deepchecks, Kolena

By now, you’d have trained and tested your model. Time to deploy!

Here, scrutinization of the Machine Learning production architecture happens. Teams may opt-in for model Application Programming Interface (API) calls, embedded model deployments, streaming model deployments, or an offline/batch deployment. This depends on things like the production requirements of the project and the resources available to the team. Libraries can also be beneficial for the successful deployment of models at scale. For instance, Flask, Django, and other Python libraries can aid with the packaging and deployment of web applications.

Figure 5. ML model deployment pipeline

This process generates predictions from input data provided by the client. Be mindful of your model architecture here since performance requirements and computer resources might differ. For example, a deep learning model might use more GPU resources compared to a simple linear regression model. In the case of speed, projects that require inference in split seconds might prioritize optimizing the hardware resources (e.g., optimizing the model to reduce latency speed on a single machine).

Consider tools based on the source of the data, host system, and destination of the data. If you are streaming data, tools for optimal processing and storage should be considered to meet the demands. The same applies with the type of tool to host your ML automation pipeline and libraries or software to serve the data to users. Considering these informs you about the speed, latency, efficiency, and cost of each tool to meet project requirements. This enables teams to manage resources.

Give thought on the tools for the infrastructure, operational requirements of the project, and security when creating an ML infrastructure.

Tools

It would help to consider the accessibility, flexibility, and collaboration each tool possesses for the data scientists on your team at each stage of the project. These can be open source projects or independent software vendors, either cloud-based, on-premise (on-prem), or edge locations.

Your tool of choice should empower your team by abstracting the complexities in the process of developing and maintaining deployed models as much as possible, helping data scientists do their jobs without needing a very high level of expertise for a specific tool. They should be scalable so users can use the projects to full capacity.

Operational Requirements

The operational requirements for supporting your projects are a major determinant of how your project is served to the user. Typical ML models might not use as many hardware resources as compared to a deep learning model that requires GPUs to run. In scenarios where the ML workflow requires a large amount of data for developing the model (classic or deep learning), GPUs might be employed to accelerate the process.

Ensure you consider the different trade-offs for your project. It is advised that you operate at an optimal level defined by your team to manage costs. For example, GPU resources can cost a lot for training models depending on the ML model architecture, so teams will need to prioritize. You can automate important aspects of the projects and reduce costs in other areas. This is totally dependent on the requirements and constraints of the project.

Source

Security

Data can be sensitive. You have to consider how you will make sure it is safe from the very beginning of your project. For example, a health organization might use sensitive health surveillance data to track disease emergencies. If this ends up in the wrong hands, it can create a dangerous situation for the client and users of the product. Additionally, adversarial attacks used to intentionally generate deceptive input data (so models make incorrect predictions) are becoming popular, so make sure to seriously consider security.

Ascertain there are access controls set up at the weak points of the ML workflow. Do not give all team members administrative access to the database so as to reduce the likelihood of attacks coming from within the team. You can use encryption tools (like a password manager and suggestion tool) and monitoring tools to monitor systems and patterns to stop attacks before they emanate into a big problem.

Smooth workflows are important for any ML project because they encourage efficient collaboration between teams, an increased chance of scalability for models in production, and reliable processes that produce desired results and meet project metrics or Key Performance Indicators (KPIs). Investing the time and cost in building a well-designed ML infrastructure guarantees a high level of compliance with the trust that the services and tools will be available when needed.

Source

Testing. CI/CD. Monitoring.

Because ML systems are more fragile than you think. All based on our open-source core.

Install Open SourceBook a Demo

There are many challenges specific to each team when building an ML infrastructure. Generally, teams might find some typical difficulties in maintaining the ML workflow.

• Data from multiple sources can get mismatched, creating data discrepancies. This might require manual efforts by teams to understand and fix the problem.
• Inadequate data versioning for the projects can lead to model decay over time since real-world data is dynamic. Ensure you modify and update the data utilized in developing your model while storing the metadata for later use.
• Experimentation is an inevitable part of developing ML solutions. The tools used for experimentation can be inefficient, as a consequence of project budget constraints. Moreover, less experienced teams might use notebooks to run experiments, which to a large extent is less effective for production ML models. This can further lead to additional requests for an increased budget to purchase efficient collaborative tools with improved computing strength.
• Validating datasets and selected models is a required task when managing ML projects. Teams often overlook metadata collected through the ML workflow, focusing only on high-level success metrics. This might not be on purpose, but it becomes harmful to the overall project in the long term causing unforeseen problems that will cost the organization.
• Communication among teammates and stakeholders can make or break a project. Friction between the operations and data science teams can lead to time wasted on conflict resolution and extra costs.
• Not having an experienced team is costly. The tool is only as good as the hands that use it.

Always remember that an ML infrastructure implementation can vary depending on several factors, including the model choice for your project. The infrastructure might be tweaked for optimal performance depending on the ML model architecture.

That being said, it might be a herculean task to build your in-house Machine Learning infrastructure from scratch; only a few organizations can do this, as Uber did with its Michelangelo Machine Learning Platform. There are plenty of valuable commercial and open source tools that will allow your team to scale your ML projects.

Deepchecks is one of those open source tools that you can incorporate into your Machine Learning infrastructure to create test suites for your data and models. It can reduce the stress on your team by allowing them to focus more on other aspects of your projects.

To explore all the checks and validations in Deepchecks, go try it yourself! Don’t forget to ⭐ their Github repo – it’s a big deal for open-source-led companies.