Skip to main content

Python Metric

Python Metric allows you to write custom evaluation logic in Python to evaluate your AI agent’s performance. This gives you complete control over the evaluation process and enables complex analysis that goes beyond simple prompt-based metrics.
The set of variables available in Python metrics (the data dict keys) is the same set used by LLM Judge metrics. For the canonical reference table — including which variables are available in Simulation vs. Observability — see Metric Variables. The detailed Python-specific accessors are documented below.

Overview

Custom code metrics are executed in a secure Python environment with access to call data including transcripts, metadata, and dynamic variables. Your code must set specific output variables to provide the evaluation result and explanation.
Once you’ve written a first pass, the Improve button runs your code on real calls and runs and makes surgical fixes for the cases it gets wrong — including repairing code that errors out. See Build a Metric with AI.
Cost: Python metrics do not consume credits. Unlike LLM judge or predefined metrics (which cost 0.2 credits per evaluation), Python metrics run free of charge regardless of evaluation volume.

Available Data Variables

When writing your custom code, you have access to different variables depending on the evaluation context.

Quick Reference

Detailed Field Documentation

Available in Both Simulation & Observability
Availability: ✅ Simulation | ✅ ObservabilityFull conversation transcript as a formatted string with timestamps
Availability: ✅ Simulation | ✅ ObservabilityTranscript as a structured list with detailed timing and speaker information
Availability: ✅ Simulation | ✅ ObservabilityCall duration in seconds as a float
Availability: ✅ Simulation | ✅ ObservabilityReason why the call ended
Availability: ✅ Simulation | ✅ ObservabilityURL to the voice recording file
Availability: ✅ Simulation | ✅ ObservabilityDescription of the AI agent used in the call
Availability: ✅ Simulation | ✅ ObservabilityAdditional context metadata as a dictionary
Availability: ✅ Simulation | ✅ ObservabilityCurrent date in YYYY-MM-DD format
Availability: ✅ Simulation | ✅ ObservabilityISO 8601 formatted timestamp of when the call/run occurred
Availability: ✅ Simulation | ✅ ObservabilityAudio metadata and analysis results as a dictionary containing:
  • has_audio_data (boolean) - Whether audio data is available for analysis
  • sample_rate (integer) - Audio sample rate in Hz (e.g., 8000, 16000)
  • shape (list) - Audio dimensions as [total_samples, channels]
  • separable_channels (boolean) - Whether stereo channels can be separated into distinct speaker channels
  • total_duration (float) - Total audio duration in seconds
  • main_speaking (list) - Speaking segments for the main/agent channel as [[start, end], …] in seconds
  • testing_speaking (list) - Speaking segments for the testing/user channel as [[start, end], …] in seconds
Availability: ✅ Simulation | ✅ ObservabilityProvider execution logs as a list of entry dicts. Only populated when execution logs were captured for the call or run.Each entry has:
  • ts — timestamp (ISO 8601, unix epoch, or seconds into the call)
  • level — severity (info, warn, error, debug)
  • message — what happened
  • logger (optional) — the component that emitted the line
  • raw (optional) — extra provider-specific detail
Observability Only
Availability: ❌ Simulation | ✅ ObservabilityDynamic variables configured for the agent as a dictionary
Availability: ❌ Simulation | ✅ ObservabilityCallLog ID for observability calls
Availability: ❌ Simulation | ✅ ObservabilityCall topic/subject
Simulation Only
Availability: ✅ Simulation | ❌ ObservabilityRecording-aligned listening-cue evidence with version: 2, total_cues, omitted_cues, and a cues list. Each cue includes start_s, end_s, text, status and reason. An inconclusive cue stays counted in coverage. text_alignment_verified indicates whether recording-aligned text can be used.Use .get() for older runs. An unknown total is insufficient to report a failure percentage. The legacy backchannel_cues input uses the session clock; backchannel_cues_recording uses the recording clock and retains its all-or-nothing alignment behavior.
Availability: ✅ Simulation | ❌ ObservabilityTags associated with the scenario
Availability: ✅ Simulation | ❌ ObservabilityProvider-specific call identifier
Availability: ✅ Simulation | ❌ ObservabilityProvider-specific call details as a dictionary
Availability: ✅ Simulation | ❌ ObservabilityCekura-specific transcript format
Availability: ✅ Simulation | ❌ ObservabilityTest scenario data configured for simulation runs
Availability: ✅ Simulation | ❌ ObservabilityRun ID for simulation runs
Availability: ✅ Simulation | ❌ ObservabilityExpected outcome value for the test scenario
Availability: ✅ Simulation | ❌ ObservabilityList of explanation strings for expected outcome

Metric Results Access

These results are available in both Simulation and Observability contexts:
Availability: ✅ Simulation | ✅ ObservabilityAccess any evaluated metric result directly by name
Availability: ✅ Simulation | ✅ ObservabilityList of explanation strings for each metric
Availability: ✅ Simulation | ✅ ObservabilityLatency metrics for performance analysisThe latency_data list contains detailed information about each turn’s latency:
  • latency: The duration of the latency in milliseconds.
  • speaker: The entity associated with the latency (e.g., “Main Agent”).
  • start_time: The timestamp when the turn started, in seconds.

Required Output Variables

Your Python code must set these two variables:
  • _result - The evaluation outcome (can be boolean, numeric, string, etc.)
  • _explanation - A string explaining the reasoning behind the result

Example Code

Here’s a simple example that checks if the agent mentioned a specific product:

Evaluation Trigger (Custom Code)

A metric’s Evaluation Trigger decides whether the metric runs on a given call (distinct from the metric body, which decides the score). When the trigger is set to Custom with Custom Code, you write a short Python snippet that runs in the same secure Python environment as a Python metric: it receives the same data dictionary and must set _result and _explanation.
  • _result — a boolean: True to run the metric on this call, False to skip it.
  • _explanation — a string explaining the decision.
A trigger is not a function — there is no return. Set _result and _explanation as variables, exactly like a Python metric. The transcript is in data["transcript_json"] (there is no messages variable), and each turn’s role is "Main Agent" or "Testing Agent" — not "agent", "assistant", or "user".
Example — only run the metric when the Main Agent spoke more than 5 times:
Example — skip the metric when the customer hung up:

Latency Threshold Example

This example detects if latency is under a threshold in each turn.

Complete Data Reference

Here’s the complete structure of data available to your custom Python code:

Data Flow and Execution Order

Metrics are evaluated in dependency levels. A metric that reads another metric’s result runs after that metric has finished; metrics with no prerequisites run in parallel in the first level, and user-defined Python metrics with no prerequisites run after the LLM-judge and predefined metrics of the same level, so they still receive every earlier result.
  1. Declare prerequisites with configuration.metric_dependencies (a list of metric IDs; names or slugs are also accepted), or leave the key out and the prerequisites are inferred from literal data["Metric Name"] reads in your code.
  2. Each level runs in parallel. The next level starts once every metric in the previous level has a result.
  3. If a prerequisite has no real result (its trigger skipped it, it timed out, or it returned None), the dependent metric is not scored: it is saved with extra.dependency_failed, extra.missing_metric_ids and extra.missing_metric_names, and is not charged.
  4. Cycles, more than 10 prerequisites on one metric, and chains deeper than 5 levels are rejected when the metric is saved.
Prerequisites must be enabled for the same source as the dependent metric: a metric that is observability-only cannot be satisfied on simulation runs.

Using Metric Results

You can access the results of other metrics that were evaluated for the same call directly by metric name using data["Metric Name"]. You can also access their explanations using data["explanation"]["Metric Name"]. Prerequisites are also available under data["metrics"], keyed by metric slug and by name. Each entry has id, name, slug, score, enum, value (the score or enum, whichever is set) and explanation:
Example usage:

Backchannel handling

Backchannel Handling answers whether your agent handled delivered listening cues, such as “mm-hmm” or “uh-huh”, properly:
  • Pass: every delivered cue was confidently handled; the agent continued naturally.
  • Fail: at least one cue caused a supported stop, unnecessary restart or an inappropriate answer that disrupted the explanation.
  • Unscored: no failure was proven, but some cues could not be judged, the cue total is unknown, or no cues were delivered.
The explanation identifies each cue by its recording timestamp and text, then states whether it was handled properly, not handled properly or could not be judged, with the reason and supporting speech. One failed cue, three handled cues and one unclear cue produce Fail, with all five outcomes explained. Four handled cues and one unclear cue remain Unscored, rather than a false pass. Results also include scored/total coverage for assessing the available evidence. The evaluator combines measured speech timing with conversational evidence. It judges at most 25 eligible cues within a 90-second budget; remaining cues are inconclusive. The initial timing threshold is uncalibrated: validate reviewed continuation and failure recordings, including whole-call error rates at different cue counts, before using the result as a release gate. Existing installations are only replaced through an explicit update; numeric metrics and rubric rules are not converted automatically.

Calling LLM Judge Metrics from Python

Function Reference: evaluate_llm_judge_metric

The evaluate_llm_judge_metric function allows you to evaluate LLM Judge metrics directly from your Python code. This function sends your data and evaluation criteria to Cekura’s LLM judge system and returns the evaluation result. Function Signature:
Parameters:
Dict
required
This is the same data object available in your custom Python code with access to transcript, metadata, and other call data.
str|None
default:"None"
Deprecated. This parameter is retained for backward compatibility and is ignored. Cekura authenticates the request automatically, so new code should omit it.
str
required
The evaluation prompt/description that guides the LLM judge on how to evaluate the metric.You can use context variables in the description using {{variable}} syntax (e.g., {{metadata.instructions}}). See LLM Judge Available Variables for a complete list of available variables.
str
default:"None"
The type of evaluation to perform. Supported values:
  • "binary" - Binary evaluation (returns 0 or 5)
  • "numeric" - Numeric evaluation (returns integer or float)
  • "continuous_qualitative" - Continuous scale from 0 to 5
  • "enum" - Enumerated values (requires enum_values parameter)
List[str]|None
default:"None"
List of possible values when using eval_type="enum". Only applicable for ENUM type evaluations.Example: ["Excellent", "Good", "Fair", "Poor"]
bool
default:"False"
When True, the LLM judge analyzes the actual voice recording instead of (or in addition to) the transcript. Use this to evaluate speech delivery, tone, pacing, or other audio properties not captured in the transcript text. Requires data["voice_recording"] to be a valid URL.
float|None
default:"None"
Clip start in seconds from the call start when audio=True. If omitted, the full recording is analyzed. Use transcript_json entry start_time values to locate specific utterances.
float|None
default:"None"
Clip end in seconds from the call start when audio=True. Must be greater than audio_start_time if both are provided.
str|None
default:"None"
Recording channel to analyze when audio=True: "testing" (the testing agent only), "main" (your agent only) or "mixed". If omitted, the mixed recording is analyzed. Isolated channels ("testing", "main") require a recording with separable channels.
float|None
default:"None"
Optional limit in seconds for this helper call’s request and polling wait. Omit to preserve the existing evaluation deadline. An exhausted limit returns an unscored result; it does not cancel an evaluation already started.
Return Value: Returns a dictionary with two keys:
  • result: The evaluated metric value (type depends on eval_type)
    • Binary types: 0 or 5
    • Numeric: int or float
    • Continuous: float between 0 and 5
    • Enum: string from enum_values
  • explanation: List of String explaining the evaluation result or error message
Using Context Variables: You can make your LLM judge evaluations dynamic by using context variables in the description parameter. For example, use {{metadata.instructions}} to reference specific scenario steps the agent was supposed to follow. See LLM Judge Available Variables for the complete list.

Example Usage

Example For Calling Basic Metrics (Deprecated)

Deprecated: evaluate_basic_metric is deprecated in favor of evaluate_llm_judge_metric. Please use evaluate_llm_judge_metric for new implementations.

Example For Calling Advanced Metrics (Deprecated)

Deprecated: evaluate_advance_metric is deprecated in favor of evaluate_llm_judge_metric. Please use evaluate_llm_judge_metric for new implementations.

Audio-Based Analysis

Custom Python metrics have access to the full voice recording and pre-computed audio segment timing, enabling you to evaluate speech delivery properties that go beyond what transcript text alone can capture. This is the right approach when no pre-defined metric covers the specific speech quality requirement you need to validate. Available audio data:
  • data["voice_recording"] — URL to the voice recording file
  • data["recording_data"] — Pre-computed speaking segments: main_speaking and testing_speaking arrays of [start, end] timestamps in seconds
  • data["transcript_json"] — Per-utterance data including start_time and end_time in seconds, useful for locating the audio window around specific spoken content
Pattern: Locate relevant utterances, then evaluate the audio Use transcript_json to find when specific content was spoken, then reference those timestamps to identify the corresponding audio segment for evaluation:
Pattern: Pass audio directly to evaluate_llm_judge_metric Use the audio=True parameter along with audio_start_time / audio_end_time to let the LLM judge listen to the actual recording segment rather than reading the transcript:
For custom signal processing instead of LLM judging, use data["voice_recording"] and data["recording_data"] directly. See Calling LLM Judge Metrics from Python for the full function reference including all audio parameters. Use cases:
  • Verifying that multi-part spoken content is delivered with appropriate pauses between segments
  • Detecting whether specific terms or sequences are spoken with the required cadence
  • Evaluating audio-level speech patterns not captured in the transcript (e.g., pacing, grouping, emphasis)

Advanced Example

Here’s a more complex example that analyzes sentiment and response time:

Example Using Multiple Data Sources

Here’s an example that combines multiple metric results with call metadata and tags: