Evaluating LLM performance often comes with an unavoidable challenge – randomness. Even when using the same model and prompt, results can vary slightly from one run to the next. Deepchecks’ Number of Judges feature is designed to bring stability and confidence to this process. By letting you control how many independent model calls – or “judges” – evaluate each property, Deepchecks aggregates multiple perspectives into a single, balanced result. And with the orchestrator option, a stronger model reviews both the judges’ outputs and the data itself, returning a refined decision with clear reasoning.
This feature seamlessly integrates into the property creation flow, giving you complete flexibility to configure and test your setup before saving. Combined with Deepchecks’ research-backed properties and few-shot mechanism, Number of Judges offers a new level of reliability in LLM evaluation – so you can trust your metrics, understand your models better, and make performance improvements with confidence.
Yaron Friedman
