Evaluating Agentic AI Systems in Production

If you would like to contribute your own blog post, feel free to reach out to us via blog@deepchecks.com. We typically pay a symbolic fee for content that’s accepted by our reviewers.

Introduction

Agentic AI systems are independent systems that make decisions to pursue goals with minimal or no human interaction. Unlike traditional systems that operate within a predefined program structure, these systems can identify issues before they develop and adapt and learn from the process as it unfolds.

Imagine you are a delivery person. You must select the best route, avoid any road obstacles, adapt when the road is congested with traffic, and deliver the package without requiring any follow-ups from the company.

Evaluating Agentic AI Systems

Image source: by the author

How would this work if Agentic AI is used? It will first identify the destination and check the traffic. Then, it will gather information from external tools, such as the Maps API and scheduling services. Meanwhile, the process will establish decision points to check if things are on track, and if they are not, it will explore fallback options and re-plan until the delivery is completed.

Evaluating Agentic AI Systems

Image source: by the author

In this way, Agentic AI demonstrates how decisions are continuously adjusted, rather than being governed by a fixed program. This dynamic nature means the programs are not static; they evolve as they are tested in live settings, because a single misalignment can affect the result. This is how Agentic AI works, and these systems are becoming increasingly ubiquitous and essential in various fields, including customer support, cybersecurity, business intelligence, and software development. That is why evaluating these systems is crucial.

Characteristics of Agentic AI

Three characteristics define agentic AI systems: autonomy, goal-directed behavior, and adaptability.

Autonomy

Agentic AI systems are independent, meaning they don’t pause for explicit human instructions. Instead, they determine the required steps autonomously and execute them in the proper order. This is the autonomy within Agentic AI. These systems have goals with multiple steps involved, allowing for the tracking of progress over time with minimal human intervention.

Goal-directed behavior

Once the Agentic AI system is given a task with an objective, it breaks it down into mini tasks and then works step by step, accomplishing each task. As a result, no micro-managing is necessary.

Adaptability

In the Agentic AI system, if the plan hits an obstacle, it adjusts the process and learns from it, so that next time, it will do things differently.

Adaptability

Image source: by the author

Challenges in Evaluating Agentic AI

In real-world settings, evaluating Agentic AI systems is considerably more challenging than assessing a simple AI because of several interrelated factors:

Complex Input and Action Space

Assume you need to move from City A to City B, and you ask a person who is new to the town how to get to City B. They are able to answer your question, but their limited knowledge constrains their answer, and they are only able to tell you one possible route. In contrast, if you were to ask a native resident of city B who is also well-versed in the city, they could provide you with multiple paths, road conditions, traffic times, alternative routes, and more. This scenario applies to all Agentic AI systems, which is why they are challenging to evaluate, as there are multiple paths to test and cover.

Multi-Step Decision Process

An accurate and reliable outcome is not only about getting the final answer correct, but about making every step correct along the way to the final goal. A single test on the final output hides the process. Therefore, it is essential to evaluate the entire process, from planning to execution.

Component Interdependencies

Agentic AI systems are composed of components that enable planning, memory, tool usage, and other functions. Each part should perform its role effectively while interacting with the other parts without disrupting the overall process. Therefore, evaluating sub-components individually does not make sense; instead, you need to test how all parts work together.

Need for Continuous Adaptation

Agentic AI system solutions are not static, so the evaluation shouldn’t be either. Ongoing evaluations must be conducted to monitor the evolving system, as a solution that worked yesterday may not be effective today.

Safety, Alignment, and Emergent Behavior

Agentic AI systems operate independently, which comes with risks. The process might fall off track, misinterpret the rules, or do something unexpected or irrelevant. Effective and adaptable evaluation methodologies are essential for ensuring that the Agentic system adheres to safety rules and acts ethically throughout its multi-step workflows.

Measurement Imbalance

In the real world, the success of agentic AI systems increases when human trust, ease of use, and safety are higher. Their adoption is also greater when the cost is lower. Most evaluations (80%) focus on technical metrics, such as speed and accuracy, while also considering human-centered assessment, including safety and economic impact.

Measurement Imbalance

Image source: by the author

Deepchecks For LLM EVALUATION

Evaluating Agentic AI Systems in Production

  • Version Comparison
  • AI-Assisted Annotations
  • CI/CD for LLMs
  • LLM Monitoring
TRY LLM EVALUATION

Evaluation Frameworks and Methodologies

Serving as a practical embodiment of these principles, Raga AI’s Holistic 8-Step Framework translates high-level evaluation methodologies into a straightforward, iterative workflow that teams can apply end-to-end.

Raga AI’s Holistic 8-Step Framework

This framework tackles the shortcomings of the RAG (Retrieval Augmented Generation) evaluation approaches, as it follows a structured methodology tailored for the complexity of Agentic AI.

The following are the key elements.

Raga AI's Holistic 8-Step Framework

Image source: by the author

  • Synthetic Data Generation for Agent Trajectories: To ensure general coverage, Raga recommends creating a variety of synthetic interactions, such as competing user intentions or tool failures, as real-world logs may overlook edge cases.
  • Comprehensive Logging and Trace Analysis: Each user query involves planning, memory lookup, tool calls, and subtasks. Extensive logs enable keeping a detailed trace of each of these steps, which ultimately helps in diagnosing failures, inefficiencies, biases, and other issues.
  • Automated Evaluation at Scale: Manual testing is no longer a solution. That’s where large-scale continuous testing, such as Agent-as-Judge techniques, comes into play. Here, a secondary evaluator scores each decision and the choice of tools.
  • Component-Level and End-to-End Evaluation: As discussed earlier, Agentic AI comprises planning, memory, and tool usage; evaluating each module separately and together is crucial.
  • Human Feedback Integration: Domain experts’ knowledge is utilized for subtle bias, empathy, and context checks, including A/B tests, feedback loops, and escalations.
  • Experimental Comparison and Impact Analysis: Run controlled comparisons when an update or change is made in memory, model, prompt, etc. Comparing the cost, safety, and performance is essential after updates are made.
  • Enhancement Through Finetuning and Prompt Engineering: Correcting weaknesses with fine-tuning and prompt engineering tweaks while retesting each after every improvement.
  • Real-Time Guardrails: This is like the last safety net in production. Enforcing policies at runtime, monitoring cost, latency, etc., and escalating to humans when needed.

Deepchecks Real-Time Monitoring & Agentic Evaluation

Deepchecks provides an open-source enterprise suite for continuous evaluation and observability of agentic workflows in production, enabling teams to identify and resolve issues quickly.

Deepchecks Real-Time Monitoring & Agentic Evaluation

  • Session-level tracing & alerts: Deepchecks ingests full traces and aggregates latency, token usage, and custom cost metadata, then surfaces alerts whenever metrics drift or failures spike.
  • Interactive trace visualizations: A point-and-click UI shows the agent’s plan, each tool call, and every intermediate output, so you can immediately spot wrong tool selections, token burn, or hesitation without digging through raw logs.
  • LLM-as-a-Judge metrics: For every step, Deepchecks evaluates planning efficiency, tool-calling quality, and tool-response quality with benchmarked LLM “judges,” then aggregates the scores to a single session grade.
  • Observability beyond the output: Tracing captures operational KPIs (per-step latency, token usage, cost) alongside quality metrics, enabling cost governance, regression detection, and performance tuning in one place.
  • Continuous evaluation & version comparison: Session-level scores feed dashboards that let you A/B-test prompts, models, or agent strategies and automatically stop a bad release before it reaches users.

Key Metrics for Assessment

A well-rounded monitoring plan tracks what the agent does, how safely it does it, and at what cost.

  • Technical performance metrics: The ultimate goal is to verify whether the agent has completed the task correctly and coherently. Standard measures, such as precision, recall, and F-score, continue to play a crucial role in determining the accuracy of models.
  • Safety and alignment: Ensuring the policy adherence and safety validation in testing is emphasized for the autonomous behavior of Agentic systems.
  • Adaptability and stability: Agent behavior adapts to changes in data, prompts, and tools. Therefore, establishing mechanisms such as drift tracking and context-shift resilience is encouraged.
  • Observability & tooling: Diagnosing the process by analyzing tool usage, failure rate, per-step latency, error hotspots, retry rate, etc., is crucial in production governance.
  • Human-centered and economic metrics: Leaders are most concerned about user trust and cost. They need a clear view of whether it helps users reliably and transparently, as well as the cost per successful task, the cost of rollout, and other relevant factors. A good balance of these two traits will lead to a higher ROI.

Real-World Use Cases

There are many instances where evaluation frameworks are used for Agentic AI systems. Agentic RAG is explored in healthcare, finance, and other diversified domains where agents retrieve, plan, and act across regulated workflows. Teams validate the hypothesis using domain-aware, multi-step tests, agent-as-a-judge scoring, and production observability through call tracing, among other methods. Two examples are discussed below.

Breast Cancer Treatment Planning

Agentic RAG and Graph RAG were compared in a research project aimed at developing an AI tool for personalized breast cancer treatment plans based on the NCCN guidelines. The system received the patient’s clinical descriptions as input. The evaluations are conducted by comparing the percentage of adherence to policies and counting errors, hallucinations, and other similar incidents.

Breast Cancer Treatment Planning

Image: Performance comparison of systems in adherence to treatment guidelines

Agentic-RAG achieved 100% guideline adherence with zero errors, outperforming ChatGPT-4 (94%) and LLM V1 (92%). Therefore, it is informative to evaluate by multi-step testing and domain-aware scoring to detect loops and emphasize safety. Agentic-RAG works in careful steps and checks itself against the NCCN rules at each step.

Credit Risk/Financial Monitoring

Credit Risk/Financial Monitoring

Source: From Article

Numerical experiments are conducted to assess the robustness of Agentic AI systems designed to perform model development and model risk management tasks in financial services. Agentic AI can handle both building credit-risk models and verifying their quality, while monitoring each step in real time. This early visibility helps catch mistakes faster, cut costs and cycle time, and create audit-ready records.

Summary of Key Points

Agentic AI systems plan each step, choose tools, and act toward the final goal with minimal oversight. Evaluating them in production means going beyond final answers to trace how they think and behave. Key hurdles in approaching the goal include branching inputs, multi-step decisions, tightly coupled components, continual adaptation, safety concerns, and over-reliance on narrow technical metrics.

This article discusses Raga’s Holistic 8-Step Framework and describes how DeepChecks provides powerful tools for real-time monitoring and evaluation, enabling the design of comprehensive tests using synthetic trajectories, end-to-end component checks, guardrails, and emergent-behavior detection, among other methods.

Deepchecks For LLM EVALUATION

Evaluating Agentic AI Systems in Production

  • Version Comparison
  • AI-Assisted Annotations
  • CI/CD for LLMs
  • LLM Monitoring
TRY LLM EVALUATION
×
Deepchecks is joining forces with Check Point Strengthening AI security – together.