LLM Judge Metric

LLM Judge Metrics allow you to evaluate your AI voice agent calls using natural language descriptions. Instead of writing code, you simply describe what constitutes success in plain English, and the system automatically evaluates each call against your criteria. This makes it easy to create custom evaluations without programming knowledge.

For the canonical reference of {{...}} variables you can use in your metric prompts — and which are available in Simulation vs. Observability — see Metric Variables.

What You Can Evaluate

LLM Judge Metrics are ideal for evaluating qualitative aspects of conversations that require understanding context and nuance:

Workflow Compliance: Check if agents followed specific steps or procedures
Communication Quality: Assess tone, clarity, professionalism, or empathy
Information Accuracy: Verify agents provided correct information or asked required questions
Customer Handling: Evaluate objection handling, de-escalation, or problem resolution
Policy Adherence: Ensure agents stayed within company policies and guidelines
Call Outcomes: Determine if desired outcomes were achieved (bookings, resolutions, etc.)

Benefits

No Coding Required: Write evaluations in natural language, no programming skills needed
Flexible & Adaptable: Easily modify criteria by updating your metric description
Context-Aware: Understands conversational context, not just keyword matching
Dynamic Variables: Use call-specific data (customer info, metadata) in your evaluations

Creating LLM Judge Metrics

Navigate to the Metrics section and select Create Metric.

Name & Type: Give your metric a descriptive name (e.g., Correct End Call by Main Agent).
Description (The Prompt): Write a natural language description of what constitutes success. This is what the LLM Judge will use to evaluate calls.

Use context variables to make the metric dynamic. For example, use {{metadata.instructions}} to reference specific scenario steps the agent was supposed to follow.You will see a list of context variables in the dashboard when creating a metric. See Metric Variables for a complete list.

Example Description:

Check if the Main Agent ended the call only after all steps in
{{metadata.instructions}} were completed by the Testing Agent.

Use Voice Recording

Off by default — the judge reads the transcript. Turn it on only for criteria the transcript cannot answer, such as tone, pace, silence, or voicemail detection. It costs extra; see Audio Evaluation.

Set Triggers

Define when the metric should run under the Evaluation Trigger section.

Always: Runs on every call (default).
Custom: Use logic to run metrics only in specific scenarios. You can write a trigger prompt in natural language, or write Python code that decides when the metric should run. Trigger code receives the same data dictionary as a Python metric and sets _result (a bool — True to run, False to skip) and _explanation. See Evaluation Trigger (Custom Code) for the contract and an example.

Testing Your Metrics

Before saving, validate your logic immediately within the builder.

Click Test Metric

Navigate to the test section within the metric builder.

Select Call IDs

Select a few past Call IDs from the list to test against.

Run the Test

Run the test to see if the metric passes/fails as expected on historical data.

Create Metric

If satisfied with the results, click Create Metric to save.

Audio Evaluation

Some things cannot be judged from transcript text at all — tone, pace, empathy, talking over the caller, dead air, hold music, voicemail and answering-machine detection, pronunciation, or who is actually speaking. For these the judge has to listen to the recording, not read the transcript.

Enable voice recording on the metric

Turn on Use voice recording — the toggle sits just above the Description on the metric form — and the judge analyzes the call’s audio alongside the transcript. Write the description in terms of what to listen for:

Did the Main Agent speak at a steady, unhurried pace, without talking
over the Testing Agent?

Audio evaluation is billed per minute of audio on top of the standard per-call LLM-judge charge, so it costs meaningfully more than a transcript-only metric. Enable it only for criteria the transcript genuinely cannot answer. Your current rates are on the Billing page.

A metric with voice recording enabled analyzes the whole call. To judge only part of a call, you need a Python metric — see below. Build with AI will also turn this setting on by itself when your description names an audio-only phenomenon, and tell you it did.

From Python code

When calling LLM Judge metrics from Python code, set audio=True to have the judge analyze the actual voice recording instead of (or in addition to) the transcript text. Python metrics can additionally window the audio to a time range:

response = evaluate_llm_judge_metric(
    data,
    api_key,
    description="Did the agent speak clearly and at an appropriate pace?",
    eval_type="binary",
    audio=True,
    audio_start_time=5.2,   # optional: clip start in seconds
    audio_end_time=12.8,    # optional: clip end in seconds
)

A time window changes what the judge listens to, not the charge — the full call duration is billed either way. See Python Metric — Audio-Based Analysis for the full pattern.

Build a Metric with AI - Refine this description with an AI agent, quickly or against your real calls and runs
Metric Variables - Variables you can use in metric descriptions
Creating Good Metrics - Complete guide for building high-fidelity metrics
Python Metric - Write custom evaluation logic in Python, including audio-based evaluation
Pre-defined Metrics - Generic metrics provided by Cekura

Get Started

Key Concepts

Guides

Integrations

Advanced

LLM Judge Metric

What You Can Evaluate

Benefits

Creating LLM Judge Metrics

Use Voice Recording

Set Triggers

Testing Your Metrics

Audio Evaluation

Enable voice recording on the metric

From Python code

​What You Can Evaluate

​Benefits

​Creating LLM Judge Metrics

​Use Voice Recording

​Set Triggers

​Testing Your Metrics

​Audio Evaluation

​Enable voice recording on the metric

​From Python code

​Related Documentation

What You Can Evaluate

Benefits

Creating LLM Judge Metrics

Use Voice Recording

Set Triggers

Testing Your Metrics

Audio Evaluation

Enable voice recording on the metric

From Python code

Related Documentation