Everything You Need for Continuous Validation of LLMs & AI

Testing. CI/CD. Monitoring

A comprehensive validation solution for every phase in the lifecycle - because AI systems are more fragile than you think. All based on our open-source core

We're now in closed beta with a brand new module:

Deepchecks For LLM Evaluation

Reduce Risk From Your LLM-Based Application

Reduces both the probability of problematic events, and your exposure should there be events of this sort.

Gain Visibility Into Your LLMs Actual Performance

This assists not only in day-to-day operations but also in forward planning in cases where the product strategy is affected by LLMs.

Simplify Compliance

Simplifies complying with AI-related policies, regulations and soft laws.

Finally: A Clear Definition of Done for Your LLM-Based App

Provides you with a simple and clear assessment of "how good is my LLM application" and whether it's good enough to deploy (this tends to be relatively difficult).

An Intuitive Framework for Evaluating Performance

Helps you understand methodically where LLM application is under-performing (using a combination of high level metrics and examples).

Version Comparison

Helps you compare versions of your LLM-Based Application based on a variety of metrics, annotations and tests.

Try LLM Evaluation

How Does It Work?

Deepchecks Open Source

Install Deepchecks Open Source to Take Care of Both Testing and Monitoring of Your ML

In the past, Deepchecks Open Source was focused only on testing ML during the research phase. As of June 2023, Deepchecks expanded this to a combination of testing and monitoring, so that data scientists and ML engineers can take multiple steps in their validation journey while remaining in the open source realm.

Testing

Deepchecks Testing is all about
users iteratively running test
suites on their data and models,
from within a notebook or an IDE.

Install Open Source

CI/CD

Deepchecks CI is all about running
the test suites you know (and
love) as part of your CI/CD, using
tools like GitHub Actions or
Airflow.

Get Started

Monitoring

Deepchecks Monitoring is all
about tracking data and models in
production to make sure your ML
system is behaving as expected.

Install Open Source


pip install -U deepchecks
from deepchecks.tabular.suites import train_test_validation
suite_result = train_test_validation().run(train_dataset, test_dataset)

on:
push: ['main']
job:
run_suites:
train_dataset: 'load.py:get_train_dataset'

test_dataset: 'load.py:get_test_dataset'

model: 'load.py:get_model'

suites: ['custom_suites.py:my_model_evaluation_suite']

dc_client = deepchecks_client.DeepchecksClient(host, API_TOKEN)
model_version = dc_client.create_tabular_model_version(model_name, version_name, schema_file, reference_dataset)
model_version.log_batch(sample_ids, data, timestamps, predictions)

Deepchecks Hub

Combining the Continuous Validation of Multiple Models, in a Managed and Secure Setting, from Research to Production

Deepchecks Hub expands the functionality of Deepchecks Open Source to include everything your team will need in a commercial setting. This consists of a variety of scalable deployment options, a unified experience of validating multiple models in parallel, security and access management features, and support.

automatically detect model and data issues

Book a Demo

Deepchecks Supported Data Types

Deepchecks supports: Tabular data, Computer Vision and NLP throughout your model and
data lifecycle from training to production

Vision

Tabular

NLP

Talk to an Expert

Explainer Video

Easy to Install

Using Deepchecks is simple: For example, setting up
Deepchecks Open Source for tabular data requires
only two lines of code.

Please refer to Deepchecks Open Source “Getting
Started” here or Deepchecks Hub Getting Started
here for more information.

Install Open Source Book a Demo


pip install -U deepchecks
from deepchecks.tabular.suites 
import train_test_validation

Run Test Suites for CI/CD

Deepchecks Open Source can be used for CI/CD by
integrating just a few lines of code to your CI/CD
scripts. This will help you to ensure that your re-trained
model will not cause issues when deployed to
production.

Book a Demo


on:
push: ['main']
job:
run_suites:
train_dataset: 'load.py:get_train_dataset'

test_dataset: 'load.py:get_test_dataset'

model: 'load.py:get_model'

suites: ['custom_suites.py:my_model_evaluation_suite']