Runs & Results

Setup steps and authentication are in the Overview. This page covers triggering runs and reading their output.

A run is one execution of a scenario against an agent. A result is the parent batch (one trigger → many runs, one per scenario × personality combination). The CLI and SDK expose both.

Pick a mode

The right command depends on the transport:

Mode	When to use
`text`	Fast functional check — no audio, scenarios run as text exchanges
`voice`	Outbound voice call via your provider (Vapi/Retell/etc.)
`chirp`	Cekura’s hosted voice runner
`livekit_v2` / `pipecat_v2`	Self-hosted LiveKit or Pipecat agents
`sip`	SIP-based provider (e.g. Plivo, Twilio SIP)
`websocket`	Custom WebSocket protocol
`vapi_webrtc`, `retell_webrtc`, `elevenlabs`	Provider WebRTC paths

Trigger a run

# Text mode (fastest for iteration)
cekura run start --agent-id 123 --scenario-ids 1,2,3 --mode text

# Voice call
cekura run start --agent-id 123 --scenario-ids 1,2,3 --mode voice

# Cekura's hosted voice runner takes a JSON request body
cekura scenarios run-chirp --from-file run.json

cekura run start also takes --tags, --folder-path (with --project-id), --frequency, --personality-ids, --test-profile-ids and --concurrency-limit. It prints the result ID; with --format json it prints the result object, whose id is the result ID.For self-hosted LiveKit and Pipecat agents, start runs from the SDK (see the SDK tab) or the API.

from cekura import Cekura

client = Cekura()

# Text mode
result = client.scenarios.run_text(agent_id=123, scenarios=[1, 2, 3])
result_id = result["id"]

Every run method returns the result object; its id is the result ID. Scenarios are passed as scenarios (IDs), or selected with tags or folder_path:

client.scenarios.run(agent_id=123, scenarios=[1, 2, 3])        # voice
client.scenarios.run_sip(agent_id=123, scenarios=[1, 2, 3])
client.scenarios.run_livekit_v2(agent=123, scenarios=[{"scenario": 1}, {"scenario": 2}])
client.scenarios.run_pipecat_v2(agent=123, scenarios=[{"scenario": 1}, {"scenario": 2}])

Wait for the result

A result is finished when its status is completed, failed, timeout or cancelled, and completed alone does not mean the runs passed. See Result statuses and gating for what each status and count means.

# Block until finished; exit 1 unless every run passed
cekura run start --agent-id 123 --scenario-ids 1,2,3 --mode text \
  --wait --fail-on-threshold 1.0

# Or watch a result that is already running
cekura run watch <result-id>
cekura run status <result-id>

--fail-on-threshold takes a pass rate from 0.0 to 1.0. The command exits 1 when the result does not meet it and 2 when the run could not start or the wait timed out.

import time

TERMINAL = {"completed", "failed", "timeout", "cancelled"}

while True:
    result = client.results.get(result_id)
    if result["status"] in TERMINAL:
        break
    time.sleep(15)

total, passed = result["total_runs_count"], result["success_runs_count"]
all_passed = result["status"] == "completed" and total > 0 and passed == total
print(f"{passed}/{total} passed", "✅" if all_passed else "❌")

Count passes with success_runs_count against total_runs_count: failed_runs_count leaves out runs that errored or timed out.

Inspect a run

cekura runs get 5544
cekura runs list --result-id <result-id>
cekura runs list --agent-id 123

run = client.runs.get(5544)
print(run["status"], run["success"], run.get("error_message"))
print(run["transcript"])

Live operations on in-progress runs

# Get a live listen URL while a voice run is in progress
cekura runs listen-url 5544

# End an in-progress call
cekura runs end-call 5544

client.runs.get_listen_url(5544)
client.runs.end_call(5544)

Vote on a metric result

Capture thumbs up/down feedback on a specific metric evaluation for a run, optionally attach the expected value and free-text feedback. The run is marked as reviewed; the metric evaluation is updated. Feeds the labs / metric-review workflow.

cekura runs mark-metric-vote 5544 \
  --metric-id 55 \
  --thumbs-down \
  --expected-value 2 \
  --feedback "Agent missed the reschedule step"

--expected-value is parsed as JSON when possible (5, true, "foo"), so numeric / boolean metrics get the right type.

client.runs.mark_metric_vote(
    5544,
    metric_id=55,
    thumbs_up=False,
    expected_value=2,
    feedback="Agent missed the reschedule step",
)

Expected outcome

If a scenario has an expected_outcome, you can re-score whether the run met it in the background, then thumbs-vote on the outcome verdict.

# Re-score expected outcome for this run (returns a progress ID)
client.runs.run_expected_outcome_bg(5544)

# Thumbs vote on the outcome verdict
client.runs.mark_expected_outcome_vote(5544, thumbs_up=True)

Flag a critical-scenario verdict as wrong

client.runs.mark_critical_scenario_wrong(5544, critical_category=42, feedback="Not a real escalation")
client.runs.unmark_critical_scenario_wrong(5544, critical_category=42)

critical_category is the ID of the critical category the verdict came from.

Re-evaluate without re-running

If you change a metric prompt and want to score existing runs against the new definition:

echo '{"run_ids": [5544, 5545], "metric_ids": [55, 56]}' > evaluate.json
cekura results evaluate-metrics 987 --from-file evaluate.json

client.results.evaluate_metrics(987, run_ids=[5544, 5545], metric_ids=[55, 56])

Promote a run into a test set

Take a passing run and freeze it as a regression dataset:

cekura test-sets create-from-run --run-id 5544 --name "regression-2024-04"

client.test_sets.create_from_run(run_id=5544, name="regression-2024-04")

Evaluators

The scenarios that runs execute.

Metrics

Define how runs are scored.

Calls

Production calls — same scoring engine, different input.

API Reference

Full field reference for runs, results, and run-mode payloads.

CLI & SDK

Runs & Results

Pick a mode

Trigger a run

Wait for the result

Inspect a run

Live operations on in-progress runs

Vote on a metric result

Expected outcome

Flag a critical-scenario verdict as wrong

Re-evaluate without re-running

Promote a run into a test set

See also

Evaluators

Metrics

Calls

API Reference

​Pick a mode

​Trigger a run

​Wait for the result

​Inspect a run

​Live operations on in-progress runs

​Vote on a metric result

​Expected outcome

​Flag a critical-scenario verdict as wrong

​Re-evaluate without re-running

​Promote a run into a test set

​See also

Evaluators

Metrics

Calls

API Reference

Pick a mode

Trigger a run

Wait for the result

Inspect a run

Live operations on in-progress runs

Vote on a metric result

Expected outcome

Flag a critical-scenario verdict as wrong

Re-evaluate without re-running

Promote a run into a test set

See also