If you like what we're working on, please  star us on GitHub. This enables us to continue to give back to the community.

How does model performance management fit into the MLOps lifecycle?

Anton Knight
Anton KnightAnswered

Through the use of model performance management, it is easy to establish a monitoring routine that will aid in model optimization at every stage of the MLOps lifecycle.

  • Avoid launching a high-performing but incorrect model;  your business requires vigilance against bias during the model development process.

Explainable AI can help identify skewed model behavior and data-driven biases in training. Model performance management provides insight into whether a trained model is unduly dependent on its input data, aiding in the selection of the most suitable model for achieving a given set of objectives.

It is recommended that after a model has been deployed to production, it keeps track of extra metadata such as the model version number, in addition to predictions. Such a model monitoring log can stop data drift from creating model drift via misaligned projections by documenting the circumstances around any unanticipated system behavior.

  • Model performance management allows your team to look back over the model’s predictions and metadata after deployment to identify the problems better and fix them.

It is also much simpler for your team to try out newer model versions thanks to the performance management cycle model. In this type of testing (or live experimentation), models can be compared to one another and to real-world data in real time. Live experimentation with just two models (the “champion” and the “challenger”) is called A/B Testing or Champion/Challenger Testing. To evaluate which model performs best, it is common practice to run numerous versions of it simultaneously, a process known as multivariate testing, with the use of a model performance management system.

Open source package for ml validation

Build Test Suites for ML Models & Data with Deepchecks

Get StartedOur GithubOur Github

Subscribe to Our Newsletter

Do you want to stay informed? Keep up-to-date with industry news, the latest trends in MLOps, and observability of ML systems.