Everything You Need for Continuous Evaluation of LLMs & AI

Testing. CI/CD. Monitoring

A comprehensive evaluation solution for every phase in the lifecycle - because AI systems are more fragile than you think. All based on our open-source core

Reduce the risk during the deployment — both the probability of problematic events and your exposure should there be events of this sort.
Gain direct visibility into the functioning of your large language model-based application.
Simplify compliance with AI-related policies, regulations, and soft laws.

Simply and clearly assess "how good is your LLM application" and deploy into production with confidence.
Thoroughly understand what is/isn’t working well! Combine high-level metrics with examples to methodically debug your LLM application.
Track and Compare different combinations of prompts, models, and code.Reduce the risk during the deployment

Try LLM Evaluation

How Does It Work?

Deepchecks Open Source

Install Deepchecks Open Source to Take Care of Both Testing and Monitoring of Your ML

In the past, Deepchecks Open Source was focused only on testing ML during the research phase. As of June 2023, Deepchecks expanded this to a combination of testing and monitoring, so that data scientists and ML engineers can take multiple steps in their evaluation journey while remaining in the open source realm.

Testing

Deepchecks Testing is all about
users iteratively running test
suites on their data and models,
from within a notebook or an IDE.

Install Open Source

CI/CD

Deepchecks CI is all about running
the test suites you know (and
love) as part of your CI/CD, using
tools like GitHub Actions or
Airflow.

Get Started

Monitoring

Deepchecks Monitoring is all
about tracking data and models in
production to make sure your ML
system is behaving as expected.

Install Open Source


pip install -U deepchecks
from deepchecks.tabular.suites import train_test_validation
suite_result = train_test_validation().run(train_dataset, test_dataset)

on:
push: ['main']
job:
run_suites:
train_dataset: 'load.py:get_train_dataset'

test_dataset: 'load.py:get_test_dataset'

model: 'load.py:get_model'

suites: ['custom_suites.py:my_model_evaluation_suite']

dc_client = deepchecks_client.DeepchecksClient(host, API_TOKEN)
model_version = dc_client.create_tabular_model_version(model_name, version_name, schema_file, reference_dataset)
model_version.log_batch(sample_ids, data, timestamps, predictions)

Deepchecks Hub

Combining the Continuous Evaluation of Multiple Models, in a Managed and Secure Setting, from Research to Production

Deepchecks Hub expands the functionality of Deepchecks Open Source to include everything your team will need in a commercial setting. This consists of a variety of scalable deployment options, a unified experience of validating multiple models in parallel, security and access management features, and support.

automatically detect model and data issues

Book a Demo

Deepchecks Supported Data Types

Deepchecks supports: Tabular data, Computer Vision and NLP throughout your model and
data lifecycle from training to production

Vision

Tabular

NLP

Talk to an Expert

Explainer Video

Easy to Install

Using Deepchecks is simple: For example, setting up
Deepchecks Open Source for tabular data requires
only two lines of code.

Please refer to Deepchecks Open Source “Getting
Started” here or Deepchecks Hub Getting Started
here for more information.

Install Open Source Book a Demo


pip install -U deepchecks
from deepchecks.tabular.suites 
import train_test_validation

Run Test Suites for CI/CD

Deepchecks Open Source can be used for CI/CD by
integrating just a few lines of code to your CI/CD
scripts. This will help you to ensure that your re-trained
model will not cause issues when deployed to
production.

Book a Demo


on:
push: ['main']
job:
run_suites:
train_dataset: 'load.py:get_train_dataset'

test_dataset: 'load.py:get_test_dataset'

model: 'load.py:get_model'

suites: ['custom_suites.py:my_model_evaluation_suite']