Most teams working with agents have encountered the simple version first, with a clean prompt, a tool that responds properly, and a simple input. In this setup, the agent usually looks capable.
The real test starts later. Once the agent is used in an actual system, it must handle broken inputs, missing context, and cases where it may produce something that sounds right but isn’t. That is where Know Your Agent, or KYA, comes in. KYA provides a way to test how an AI agent performs under real-world conditions, not just in a smooth demo.
Start with the job, not the model
KYA does not begin by asking whether the model is smart. Instead, it begins by asking what the agent is allowed to do, what it usually does, and what failure would look like in a real workflow.
That framing matters for AI agent evaluation. If your agent files support tickets, the test scenarios should look like ticket traffic. If it reviews security alerts, the scenarios should include noisy logs, incomplete evidence, and conflicting signals. If it updates CRM records, the test set should include duplicates, stale fields, and permission boundaries.
Many weak agent tests fail here. They measure answer quality in isolation, but the actual system fails on sequencing, tool choice, or state handling.
Build scenarios from real traces
The most valuable source of test scenarios is production history. KYA-style evaluation usually starts with traces from earlier runs, support transcripts, failed automations, and tool logs. Engineers pull out cases where the agent hesitated, looped, skipped a step, or returned something plausible but wrong. Those cases become scenario seeds.
Add failure-oriented mutations
Once the base scenario is established, KYA can mutate it to probe weak spots. This is where hallucination detection becomes more useful. You are not only checking whether the final answer is wrong-you’re checking whether the agent invented facts, claimed a tool ran when it did not, cited fields that were never present, or silently filled gaps with confident language.
Common mutations are simple:
- Remove one source document.
- Corrupt one API field.
- Delay one tool response.
- Rename an enum value.
- Insert an irrelevant but tempting instruction.
- Provide stale memory that conflicts with fresh input.
These are not exotic attacks. They are normal software conditions. But they are exactly where agents start improvising.
A good evaluator predicts the expected behavior for each mutation: e.g., seek clarification when needed; decline action if necessary; retry the tool as required; use fallback logic; provide partial output with indications of uncertainty. The goal is not to trap the model but to see if the system stays within its boundaries.
Score behavior, not just answers
A single pass/fail score is often too blunt. For evaluating AI agents, KYA scenarios tend to be more effective when each run is scored using several small metrics.
- Task completion: Did the agent reach the correct end state?
- Tool correctness: Did it call the right tool with the right arguments?
- Policy adherence: Did it stay within permissions and instructions?
- Recovery behavior: Did it handle missing data or tool failures cleanly?
- Hallucination rate: Did it invent facts, outputs, or execution steps?
Keep the scenarios close to operations
As agents move into more sensitive workflows, KYA is increasingly tied to identity, authorization, and accountability, not just output quality. That means scenarios should also reflect who the agent is acting for, what scope it has, and whether a risky action should require stronger verification or a human checkpoint.
That also keeps the test suite honest. A support agent should not be evaluated like a research bot. A procurement agent should not be tested like a summarizer: same model, different failure surface.
The best teams keep feeding fresh incidents back into the scenario pool. Every bad run buys a new test case. Every weird edge case becomes reusable. Over time, the suite starts looking less like a benchmark and more like a record of operational scars.
Final Thoughts
KYA-generated scenarios are useful because they closely reflect actual agent behavior. They test decisions, tool usage, recovery paths, and whether the system starts guessing when the ground disappears.
That makes Know Your Agent a practical way to improve reliability, not by asking whether the model sounds smart, but by checking how it behaves when the workflow is no longer neat.
Kayley Marshall 
Anton Knight 