Evaluators

Setup steps and authentication are in the Overview. This page covers the inputs your evaluations consume.

An evaluator is the input to a run. The most important is a scenario — a structured test case describing what the simulated caller does and what the agent should achieve. Scenarios are paired with optional test profiles (transport / mode config), personalities (caller behavior), and dynamic variables (per-run overrides).

Scenarios

List, get, create, update, delete

# List scenarios for an agent
cekura scenarios list --agent-id 123

# Get one
cekura scenarios get 9876

# Create from a JSON file
cekura scenarios create --from-file my_scenario.json

# Update one
cekura scenarios update 9876 --from-file patch.json

# Delete
cekura scenarios delete 9876

from cekura import Cekura

client = Cekura()

# List
scenarios = client.scenarios.list(agent_id=123)

# Get one
scenario = client.scenarios.get(scenario_id=9876)

# Create
sc = client.scenarios.create(
    agent=123,
    name="Reschedule appointment",
    description="Caller wants to move their booking to next week",
    scenario_type="conversation",
    tags=["smoke"],
)

# Update
client.scenarios.update(scenario_id=sc["id"], description="Updated wording")

# Delete
client.scenarios.delete(scenario_id=9876)

Bulk update

The bulk-update payload is a single JSON object with a top-level scenarios list, plus any of metric_ids_to_add, metric_ids_to_remove, tool_ids_to_add, etc.

Prepare bulk.json:

{
  "scenarios": [
    { "id": 101, "tags": ["regression"] },
    { "id": 102, "tags": ["smoke"] }
  ],
  "metric_ids_to_add": [55],
  "tool_ids_to_remove": [12]
}

Apply:

cekura scenarios bulk-update --from-file bulk.json

client.scenarios.bulk_update(
    scenarios=[
        {"id": 101, "tags": ["regression"]},
        {"id": 102, "tags": ["smoke"]},
    ],
    metric_ids_to_add=[55],
    tool_ids_to_remove=[12],
)

Generate scenarios

Cekura can generate scenario coverage from a description or an existing call.

cekura scenarios generate --agent-id 123 --description "Bookings flow — happy path and reschedules" --count 10
cekura scenarios generate-progress --progress-id <id>

job = client.scenarios.generate_bg(
    agent_id=123,
    description="Bookings flow — happy path and reschedules",
    count=10,
)
client.scenarios.generate_progress(progress_id=job["progress_id"])

Generate or improve scenario instructions

When you have a scenario shell and need Cekura to author (or rewrite) its instructions — for example, after you’ve added an expected_outcome and want the simulated caller’s script regenerated to match — use the instructions pipeline. It runs in the background and exposes a single progress endpoint.

# AI-author instructions from an outline
job = client.scenarios.generate_instructions(
    agent_id=123,
    extra_instructions="Caller wants to escalate after two failed retries",
)

# Improve the instructions on an existing scenario
client.scenarios.improve_instructions(
    scenario_id=9876,
    feedback="Make the caller more impatient",
)

# Poll either job
client.scenarios.instructions_progress(progress_id=job["progress_id"])

Build a scenario from a transcript

Two flavors. create_from_transcript makes a brand-new scenario; update_with_transcript revises an existing one (useful when a production call exposes a scenario’s gap).

# New scenario from a transcript
client.scenarios.create_from_transcript(
    agent=123,
    transcript=[
        {"role": "agent", "text": "How can I help?"},
        {"role": "user",  "text": "I'd like to reschedule."},
    ],
    name="Reschedule — from call 7788",
)

# Refresh an existing scenario from a newer transcript
client.scenarios.update_with_transcript(
    scenario_id=9876,
    transcript=[...],
)

Scenario agent (interactive AI builder)

The scenario agent is an LLM-driven authoring loop — you describe an intent and it iterates on a scenario for you. Kicks off in the background.

job = client.scenarios.scenario_agent(
    agent_id=123,
    prompt="Build a scenario where the caller cancels then immediately re-books",
)
client.scenarios.scenario_agent_progress(progress_id=job["progress_id"])

Test profiles

A test profile bundles transport + mode settings (voice, text, SIP, livekit_v2, pipecat_v2, etc.) so you can reuse them across runs.

cekura test-profiles list --agent-id 123
cekura test-profiles create --from-file profile.json
cekura test-profiles update 7 --from-file patch.json

client.test_profiles.list(agent_id=123)
client.test_profiles.create(
    agent=123,
    name="Returning customer",
    information={"customer_name": "Alex", "account_id": "A-552"},
)
client.test_profiles.update(7, name="Returning customer (EU)")

Personalities

Personalities define how the simulated caller behaves (interruptive, accented, polite, etc.). The catalog is platform-managed — you enable or disable items per project.

# Browse the catalog
cekura personalities list

# Enable / disable on a project
cekura personalities enable-for-project 742 --personalities 3,29
cekura personalities disable-for-project 742 --personalities 3

client.personalities.list()
client.personalities.enable_for_project(project_id=742, personalities=[3, 29])
client.personalities.disable_for_project(project_id=742, personalities=[3])

Dynamic variables

Dynamic variables let you parameterize a scenario at run time (e.g. customer name, account ID). Their values come from a test profile: attach one to the scenario, or pick profiles when you start the run. Each profile passed multiplies the run count.

cekura run start --agent-id 123 --scenario-ids 101 --mode text --test-profile-ids 7

client.scenarios.run_text(agent_id=123, scenarios=[101], test_profile_ids=[7])

Agents

The agent under test that scenarios run against.

Runs & Results

Trigger evaluations using these scenarios.

Evaluator concepts

Conditional actions, mock tools, and design best practices.

API Reference

Full field reference for scenarios, test profiles, and personalities.

CLI & SDK

Scenarios

List, get, create, update, delete

Bulk update

Generate scenarios

Generate or improve scenario instructions

Build a scenario from a transcript

Scenario agent (interactive AI builder)

Test profiles

Personalities

Dynamic variables

See also

Agents

Runs & Results

Evaluator concepts

API Reference

​Scenarios

​List, get, create, update, delete

​Bulk update

​Generate scenarios

​Generate or improve scenario instructions

​Build a scenario from a transcript

​Scenario agent (interactive AI builder)

​Test profiles

​Personalities

​Dynamic variables

​See also

Agents

Runs & Results

Evaluator concepts

API Reference

Scenarios

List, get, create, update, delete

Bulk update

Generate scenarios

Generate or improve scenario instructions

Build a scenario from a transcript

Scenario agent (interactive AI builder)

Test profiles

Personalities

Dynamic variables

See also