Tests as Code

What is Tests as Code?

Normally you build evaluators in the Cekura dashboard and then run them. Tests as Code lets you skip that: write your test cases in a JSON file, commit it alongside the agent it tests, and run it straight from CI. Nothing in the file becomes an evaluator in your workspace. Your evaluator list stays exactly as you left it.

Why Use It?

Tests live with the code

A change to your agent’s prompt and the tests that cover it land in the same pull request, reviewed together.

Full history

Your version control shows who changed which test case and why — including on branches, where dashboard evaluators cannot follow.

No dashboard setup

New environment, new branch, fork of the repo: the tests come with it. Nothing to recreate.

Same results

Runs appear in Results exactly like any other run, with the same metrics, transcripts, and reports.

Use dashboard evaluators when non-engineers author or review tests, and when you want them reusable across scheduled jobs. Use Tests as Code when the tests belong to a codebase. They work side by side — one agent can have both.

Quickstart

1. Create a spec file

Save this as cekura.tests.json in your repository:

{
  "$schema": "https://docs.cekura.ai/schemas/test-suite/v1.json",
  "version": "1",
  "suite": { "name": "Checkout regression" },
  "scenarios": [
    {
      "name": "Refund happy path",
      "instructions": "You are Maria. Order {{test_profile.order_id}} arrived damaged and you want a refund. Be polite but firm.",
      "expected_outcome": "The agent verifies the order and confirms a refund.",
      "metrics": ["greeting_by_name", "no_hallucination"],
      "personality": 3,
      "test_profile": {
        "agent_variables": { "order_id": "ORD-4471" },
        "caller_variables": { "user_name": "Maria" }
      }
    }
  ]
}

The $schema line is optional but gives you autocomplete and inline validation in most editors.

The file says what to test. Which agent to run it against, and how to reach that agent, are supplied by the request — so the same file works unchanged across agents, projects and environments.

2. Check it

Validate without running or spending anything:

curl --request POST \
  --url 'https://api.cekura.ai/test_framework/v1/scenarios/run_scenarios_json/?dry_run=true' \
  --header 'X-CEKURA-API-KEY: <api-key>' \
  --header 'Content-Type: application/json' \
  --data "{\"agent_id\": 12, \"channel\": \"voice\", \"spec\": $(cat cekura.tests.json)}"

You get back the run that would happen — each test case, the total run count, and the estimated cost:

{
  "dry_run": true,
  "valid": true,
  "plan": {
    "agent_id": 12,
    "project_id": 742,
    "channel": "voice",
    "suite_name": "Checkout regression",
    "scenario_count": 1,
    "total_runs": 1,
    "estimated_cost": 0.35,
    "scenarios": [
      {
        "key": "refund_happy_path",
        "name": "Refund happy path",
        "type": "instruction",
        "frequency": 1,
        "metric_ids": [412, 413],
        "tags": [],
        "personality": { "mode": "existing", "id": 3, "name": "Normal Male" },
        "test_profile": {
          "mode": "inline",
          "sections": ["main_agent_variables", "testing_agent_variables"]
        }
      }
    ]
  }
}

personality.mode tells you which form was resolved — existing (referenced by ID), override (a base plus your changes), inline (defined from scratch), or project_default. Use it to confirm the file did what you meant before spending anything.

3. Run it

Drop ?dry_run=true and the runs start:

curl --request POST \
  --url 'https://api.cekura.ai/test_framework/v1/scenarios/run_scenarios_json/' \
  --header 'X-CEKURA-API-KEY: <api-key>' \
  --header 'Content-Type: application/json' \
  --data "{\"agent_id\": 12, \"channel\": \"voice\", \"spec\": $(cat cekura.tests.json)}"

The response is the same shape as any evaluator run, so poll the returned id against the Results API for progress and success rate.

The request

POST /test_framework/v1/scenarios/run_scenarios_json/ Authenticate with X-CEKURA-API-KEY. The key’s own scope resolves the project, so no project_id is needed.

Field	Required	Description
`agent_id`	yes	The agent to test.
`spec`	yes*	The spec, as a JSON object.
`file`	yes*	A `.json` spec file, as `multipart/form-data`. Use instead of `spec`.
`channel`	no	`voice` (default), `text`, `elevenlabs`, `livekit_v2`, `pipecat_v2`, or `chirp`.
`livekit_data`	no	LiveKit connection overrides for this run. Only valid when `channel` is `livekit_v2`. Accepted keys: `agent_name`, `config`, `url`.
`pipecat_data`	no	Pipecat Cloud connection overrides for this run. Only valid when `channel` is `pipecat_v2`. Accepted keys: `pipecat_agent_name`, `config`, `room_properties`.
`chirp_data`	no	CHIRP connection overrides for this run. Only valid when `channel` is `chirp`. Accepted key: `websocket_url`, which replaces the agent’s saved WebSocket URL for this run only.
`name`	no	Label for this run. Defaults to the spec’s `suite.name`.
`agent_number`	no	Your agent’s phone number for this run, in E.164 format. Overrides the number configured on the agent. `voice` and `text` only.
`outbound_phone_number`	no	For an inbound voice agent, the number Cekura dials, in E.164 format. Defaults to `agent_number`, then the agent’s configured number. Rejected for outbound voice agents — set `phone_number` on the test case instead. For SMS and WhatsApp agents, the contact number for the conversation. `voice` and `text` only.
`mode`	no	`same_number` (default) places every call from one Cekura number; `different_numbers` spreads calls across your pool and can’t be combined with `phone_number` on a test case. Inbound `voice` agents only.
`frequency`	no	Run every test case this many times, on top of each case’s own `frequency`. Default `1`.
`concurrency_limit`	no	Maximum number of runs in parallel. Overrides the spec’s `defaults.concurrency_limit`.
`mock_tool_names`	no	Mock tools to activate for this run. Omit or send `null` to mock every tool configured on the agent; send `[]` to mock none.

* Send exactly one of spec or file. This endpoint is also reachable at POST /test_framework/v2/scenarios/run_scenarios_json/, which behaves identically; existing integrations on either path keep working.

Query parameter	Description
`dry_run=true`	Validate and price without creating or charging anything.

Uploading the file instead of embedding it

Avoids $(cat …) quoting entirely, which is handy in shells where escaping gets awkward:

curl --request POST \
  --url 'https://api.cekura.ai/test_framework/v1/scenarios/run_scenarios_json/' \
  --header 'X-CEKURA-API-KEY: <api-key>' \
  --form 'agent_id=12' \
  --form 'channel=voice' \
  --form 'file=@cekura.tests.json'

Writing a spec file

Structure at a glance

{
  "$schema": "…/schemas/test-suite/v1.json",   optional, for editor autocomplete
  "version": "1",                             required
  "suite":    { name, description },           optional
  "defaults": {                                optional, applied to every case
    frequency, concurrency_limit, language, max_duration,
    metrics, personality, test_profile, tags, phone_number
  },
  "scenarios": [                               required, 1–200
    {
      key, name,                               name required
      type,                                    instruction | conditional_actions
      instructions | conditional_actions,      one, depending on type
      expected_outcome, first_message,
      metrics, personality, test_profile,
      tools, language, frequency, max_duration,
      phone_number
    }
  ]
}

Note what is not in there: no agent_id, no channel, no credentials, no provider ids. Those are request parameters or agent configuration — see Choosing the run target. A file containing agent_id is rejected, so the split stays honest.

The envelope

Field	Required	Description
`version`	yes	Spec format version. Currently `"1"`. Pin it — new formats ship as new versions.
`suite.name`	no	Labels the run in Results.
`defaults`	no	Applied to every test case unless the case overrides it.
`scenarios`	yes	Your test cases. Between 1 and 200.

A test case

Field	Required	Description
`name`	yes	Shown in Results. Up to 80 characters.
`key`	no	Stable identifier for this case. Keep it constant across commits so results stay comparable. Defaults to `name`.
`instructions`	yes*	What the simulated caller does.
`expected_outcome`	no	What success looks like, in plain language.
`first_message`	no	The caller’s opening line. Defaults to `"Hello"`.
`metrics`	no	Existing metrics, by ID or slug.
`personality`	no	An ID, or an inline definition.
`test_profile`	no	An ID, or inline test data.
`tags`	no	Freeform labels.
`tools`	no	`end_call`, `dtmf`, `receive_dtmf`, `end_call_only_on_transfer`, `send_sms`, `call_hold`.
`language`	no	Language code, e.g. `en`.
`frequency`	no	How many times to run this case. Defaults to 1.
`max_duration`	no	Max call length in seconds.
`phone_number`	no	E.164 number on Cekura’s side of the call. See Pinning a phone number.
`type`	no	`instruction` (default) or `conditional_actions`.

* instructions is required for instruction cases; conditional_actions cases need conditional_actions and language instead.

Metrics must already exist

A spec references metrics, it cannot define them. Create them in the dashboard first, then point at them by numeric ID or by slug:

"metrics": [412, "greeting_by_name"]

Prefer slugs — a file full of slugs can be pointed at a different project without editing, while numeric IDs are project-specific.

A referenced metric must be enabled for the agent you are testing. A metric that isn’t would not be evaluated, and the run would pass having checked nothing — so the API rejects it with an error naming the metric instead. Enable it for that agent in the dashboard, or drop it from the spec.

Personalities

Three forms, depending on how much control you want:

Use an existing one
Adjust an existing one
Define one from scratch

"personality": 3

Any personality in your organization, or any Cekura-provided one.

"personality": {
  "base": 3,
  "speed": 1.15,
  "background_noise": "office"
}

Starts from personality 3 and changes only what you list. Anything you leave out is inherited.

"personality": {
  "prompt": "You interrupt often and speak quickly. You are in a hurry.",
  "language": "en",
  "voice_model": "eleven_multilingual_v2",
  "voice_id": "TX3LPaxmHKxFdv7VOQHJ",
  "background_noise": "off",
  "provider": "11labs",
  "speed": 1.2
}

With no base, these five are required: prompt, language, voice_model, voice_id, background_noise.

Settable inline: name, prompt, language, voice_id, voice_model, provider, speed, background_noise, end_call_enabled. accent is not in that list and a spec that sets it is rejected: the caller is synthesised from voice_id alone, so accent is a property of the voice and is reported back as a read-only label. Set voice_id to a voice that has the accent you want.

Personalities defined inline are used for that run only. No personality is created in your workspace — your personality list is untouched. Cekura manages the underlying voice and transport configuration, so a spec cannot reach those internals.

Interruption behaviour and the finer audio controls — interruption_level, generation_config, network_simulation, background_sound_volume, message_plan, start_speaking_plan, stop_speaking_plan — cannot be set inline, and a spec that tries is rejected rather than having them quietly ignored. Configure them on a personality in the dashboard and reference it with base, which carries all of them through.

Test data

Reference a saved test profile, or supply the data inline:

"test_profile": {
  "agent_variables":  { "order_id": "ORD-4471", "account_tier": "gold" },
  "caller_variables": { "user_name": "Maria" }
}

agent_variables reach the agent under test as dynamic variables at call time.
caller_variables are context for the simulated caller only, and never sent to your agent. Reference them in instructions as {{test_profile.<key>}}.

Like inline personalities, inline test data is used for that run only. No test profile is created — your test-profile list is untouched.

Defaults

Anything repeated across cases can move up into defaults:

{
  "version": "1",
  "defaults": {
    "language": "en",
    "metrics": ["greeting_by_name"],
    "personality": 3,
    "max_duration": 300
  },
  "scenarios": [
    { "name": "Refund", "instructions": "..." },
    { "name": "Order status", "instructions": "...", "frequency": 3 }
  ]
}

A test case overrides defaults field by field. That includes metrics: a test case that sets its own metrics list replaces the default list rather than adding to it.

Pinning a phone number

By default a voice run uses the least-used number from your pool, so the Cekura-side number can differ from run to run. Set phone_number when a case needs a fixed one — for example when your agent whitelists caller IDs, or when your agent places the call and must be told which number to dial (a handoff or transfer flow):

{ "name": "Warm transfer", "instructions": "...", "phone_number": "+14155550123" }

When Cekura dials your agent, this is the caller ID it shows. When your agent dials Cekura, this is the number Cekura answers on. The number must be one available to your organization — a Cekura pool number or one you imported — in E.164 format (+ followed by country code and digits). Anything else is rejected at scenarios[i].phone_number before any call is placed. Like the other case fields, phone_number can go in defaults to apply to every case in the file, and it is included when you export existing evaluators that have a number configured.

Choosing the run target

Two request parameters decide where and how a spec runs — neither belongs in the file:

Parameter	Description
`agent_id`	Required. The agent to test.
`channel`	`voice` for a phone call (the default), `text` for chat, `elevenlabs` for an ElevenLabs WebRTC session, `livekit_v2` for a LiveKit WebRTC session, `pipecat_v2` for a Pipecat Cloud WebRTC session, or `chirp` for a raw-PCM WebSocket (CHIRP) endpoint.

--data "{\"agent_id\": 42, \"channel\": \"text\", \"spec\": $(cat cekura.tests.json)}"

Because the target lives in the request, the same committed file runs against staging and production, or across several agents, without edits. Use metric slugs rather than numeric IDs to make that fully portable, since IDs are project-specific. Anything the channel needs beyond that comes from the agent’s own configuration: a phone number for voice, a chat provider for text, ElevenLabs credentials and agent id for elevenlabs, LiveKit credentials for livekit_v2, Pipecat Cloud credentials for pipecat_v2, or a WebSocket URL (and optional Basic Auth) for chirp. If the agent isn’t configured for the channel you pick, the run is rejected with a message naming what’s missing. For livekit_v2, pipecat_v2 and chirp, you can pass connection overrides in the request via livekit_data, pipecat_data or chirp_data respectively. These overrides apply only to the current run and are not part of the portable spec — keeping the committed file environment-agnostic while letting you target a specific agent deployment at request time.

Running against a CHIRP WebSocket agent

A chirp run needs no phone number, so it works for an agent that is reachable only over its WebSocket endpoint. If the endpoint changes between runs — for example a tunnel (Cloudflare Tunnel, ngrok) started fresh in each CI job — pass the current URL in chirp_data.websocket_url instead of updating the agent:

--data "{\"agent_id\": 42, \"channel\": \"chirp\", \"chirp_data\": {\"websocket_url\": \"$TUNNEL_URL\"}, \"spec\": $(cat cekura.tests.json)}"

Without chirp_data, the run uses the WebSocket URL saved on the agent. Basic Auth credentials always come from the agent. If the agent has Basic Auth set, the URL must be wss://. Both the dry run and the validate endpoint reject a missing URL, or Basic Auth over plain ws://, before any call is placed.

Run overrides

Everything that describes a test case lives in the file: instructions, metrics, personality, test data, and the Cekura-side phone_number. Settings that describe this run go on the request instead, so one committed file can run against different numbers or load levels without edits:

--data "{\"agent_id\": 42, \"agent_number\": \"+14155550123\", \"frequency\": 3, \"concurrency_limit\": 5, \"spec\": $(cat cekura.tests.json)}"

These are the same run options the evaluator run endpoints accept, and they apply to every test case in the spec:

agent_number replaces the agent’s configured number for this run — for example to test a staging line with the same agent. It is your agent’s number; the case-level phone_number is Cekura’s side of the call.
outbound_phone_number sets the number Cekura dials when your agent receives calls. If you only pass agent_number, Cekura dials that. For an agent that places calls, choose the Cekura number it dials with phone_number on the test case instead; outbound_phone_number is rejected there.
frequency multiplies each case’s own frequency: a case with "frequency": 2 run with frequency: 3 runs 6 times. The dry run’s total_runs and estimated cost include it.
concurrency_limit wins over defaults.concurrency_limit in the file.
mode applies to agents that receive calls. different_numbers picks the Cekura number for every call, so it can’t be combined with a test case’s phone_number.
mock_tool_names works as it does on evaluator runs. Send mock_tool_names: [] to mock no tools; leave it out to mock every tool configured on the agent.

Personalities and test profiles are chosen per test case in the file (see Personalities and Test data), so personality_ids and test_profile_ids are not accepted here. Neither are scenarios, tags or folder_path — the spec is the list of test cases. Sending any of them with a value returns a 400 that says where the setting belongs, rather than silently ignoring it; empty placeholders such as [] or null are ignored. A dry run runs the same checks as the real run, so a rejected override, missing channel credentials, or a missing LiveKit agent_name or Pipecat pipecat_agent_name fails the dry run too. It echoes the overrides it applied under plan.run_overrides, and the numbers the run will use under plan.agent_number and plan.outbound_phone_number, so you can confirm them before placing any calls.

Reading validation errors

Every problem in a spec is reported at once, keyed by its location in the file, so one round trip tells you everything to fix:

{
  "errors": {
    "scenarios[0].instructions": ["Instructions are required."],
    "scenarios[1].metrics[0]": [
      "No metric with slug 'greetng_by_name' in project 742. Check the slug in the dashboard, or reference the metric by its numeric ID."
    ],
    "scenarios[2].personality.voice_model": [
      "'voice_model' is required when no 'base' personality is given."
    ]
  }
}

Run the dry check on every commit and you catch these before anything is spent.

Using it in CI

Runs are asynchronous: the run call returns once they are queued, before anything is dialled. A CI step has to wait for the result to finish and then fail the job unless the runs passed. cekura run spec does both: --wait polls the result until it is completed, failed, timeout or cancelled, and --fail-on-threshold 1.0 exits non-zero unless every run passed. See Result statuses and gating for what each status and count means if you call the API directly instead.

name: Agent tests
on: [pull_request]

jobs:
  cekura:
    runs-on: ubuntu-latest
    env:
      CEKURA_API_KEY: ${{ secrets.CEKURA_API_KEY }}
    steps:
      - uses: actions/checkout@v4
      - run: pip install "cekura[cli]"

      - name: Check the spec
        run: cekura run spec cekura.tests.json --agent-id ${{ vars.CEKURA_AGENT_ID }} --dry-run

      - name: Run the tests
        run: |
          cekura run spec cekura.tests.json --agent-id ${{ vars.CEKURA_AGENT_ID }} \
            --name "PR #${{ github.event.pull_request.number }}" \
            --wait --fail-on-threshold 1.0

agent-tests:
  image: python:3.12-slim
  before_script:
    - pip install "cekura[cli]"
  script:
    - cekura run spec cekura.tests.json --agent-id "$CEKURA_AGENT_ID" --dry-run
    - cekura run spec cekura.tests.json --agent-id "$CEKURA_AGENT_ID" --wait --fail-on-threshold 1.0

pipelines:
  pull-requests:
    '**':
      - step:
          name: Agent tests
          image: python:3.12-slim
          script:
            - pip install "cekura[cli]"
            - cekura run spec cekura.tests.json --agent-id "$CEKURA_AGENT_ID" --dry-run
            - cekura run spec cekura.tests.json --agent-id "$CEKURA_AGENT_ID" --wait --fail-on-threshold 1.0

--channel picks the target (voice by default, text, or elevenlabs). Lower --fail-on-threshold (e.g. 0.9) to allow some runs to fail; a result that ends failed, timeout or cancelled fails the job either way. The GitHub Actions, GitLab and Bitbucket integrations run dashboard evaluators by ID, tag or folder; for a committed spec, use the CLI as above.

Split it across two jobs: run the dry check on every push (free, fast, catches a broken file), and the real run only on the branches you actually want tested.

Starting from evaluators you already have

If your tests currently live in the dashboard, export them rather than rewriting them by hand: select your evaluators, choose Export, and pick Test suite spec (.json). The file it produces is a valid spec you can commit as-is.

Limits


Test cases per spec	200
Spec file size	1 MB
Total runs per request	Subject to your plan’s run limits

Test cases from a spec are cleaned up automatically once no recent run needs them, and results keep their test-case names regardless. Inline personalities and test data never become rows at all, so nothing accumulates in your personality or test-profile lists.

Getting the schema

The machine-readable schema this endpoint accepts:

curl 'https://api.cekura.ai/test_framework/v1/scenarios/json_schema/'

No API key needed — it is a public contract with no data of yours in it. Link it from your spec file as $schema for editor autocomplete, or feed it to a JSON Schema validator to check specs locally before they reach CI. The same document is published at a stable URL, which is what the $schema line in the examples above points at:

https://docs.cekura.ai/schemas/test-suite/v1.json

Get Started

Key Concepts

Guides

Integrations

Advanced

What is Tests as Code?

Why Use It?

Tests live with the code

Full history

No dashboard setup

Same results

Quickstart

1. Create a spec file

2. Check it

3. Run it

The request

Uploading the file instead of embedding it

Writing a spec file

Structure at a glance

The envelope

A test case

Metrics must already exist

Personalities

Test data

Defaults

Pinning a phone number

Choosing the run target

Running against a CHIRP WebSocket agent

Run overrides

Reading validation errors

Using it in CI

Starting from evaluators you already have

Limits

Getting the schema

​What is Tests as Code?

​Why Use It?

Tests live with the code

Full history

No dashboard setup

Same results

​Quickstart

​1. Create a spec file

​2. Check it

​3. Run it

​The request

​Uploading the file instead of embedding it

​Writing a spec file

​Structure at a glance

​The envelope

​A test case

​Metrics must already exist

​Personalities

​Test data

​Defaults

​Pinning a phone number

​Choosing the run target

​Running against a CHIRP WebSocket agent

​Run overrides

​Reading validation errors

​Using it in CI

​Starting from evaluators you already have

​Limits

​Getting the schema

What is Tests as Code?

Why Use It?

Quickstart

1. Create a spec file

2. Check it

3. Run it

The request

Uploading the file instead of embedding it

Writing a spec file

Structure at a glance

The envelope

A test case

Metrics must already exist

Personalities

Test data

Defaults

Pinning a phone number

Choosing the run target

Running against a CHIRP WebSocket agent

Run overrides

Reading validation errors

Using it in CI

Starting from evaluators you already have

Limits

Getting the schema