> ## Documentation Index > Fetch the complete documentation index at: https://docs.cekura.ai/llms.txt > Use this file to discover all available pages before exploring further. # Run Evaluator V2 > Start evaluator runs against an agent. Pick how the simulated user reaches the agent with `transport`; it must be one of the agent's `enabled_connections` (see `aiagents_retrieve`), and a transport that is not enabled is rejected with the list of enabled ones. Fields that belong to another transport are rejected. ## OpenAPI ````yaml post /test_framework/v2/scenarios/run/ openapi: 3.1.0 info: title: Cekura API version: v1 description: >- Complete API documentation for the Cekura platform. This API provides endpoints for testing, observing, and evaluating AI voice agents — including managing agents, running evaluators, defining metrics, and analyzing call quality. servers: - url: https://api.cekura.ai security: [] paths: /test_framework/v2/scenarios/run/: post: tags: - Evaluators summary: Run evaluators against an agent over any transport description: >- Start evaluator runs against an agent. Pick how the simulated user reaches the agent with `transport`; it must be one of the agent's `enabled_connections` (see `aiagents_retrieve`), and a transport that is not enabled is rejected with the list of enabled ones. Fields that belong to another transport are rejected. **Scenario selection (pick one):** `scenarios` (evaluator IDs), `tags`, or `folder_path` with `project_id`. `websocket` and `pipecat_v1` take their evaluators inside `websocket_urls` / `pipecat_rooms` instead, one entry per evaluator with its own URL. **Run-count math:** total runs = `len(scenarios) × frequency × max(1, len(personality_ids)) × max(1, len(test_profile_ids))`. `text`, `email` and `websocket` runs bill at the text-simulation rate; every other transport places a voice call and bills at the voice rate. **Polling for completion:** the response is the result itself — read its top-level `id`. Poll `GET /test_framework/v1/results/{result_id}/` until `status` is `completed`, `failed`, `timeout` or `cancelled`. Check `failed_reasons` for infrastructure errors and `runs[].error_message` for per-run errors. operationId: scenarios-run requestBody: content: application/json: schema: $ref: '#/components/schemas/RunScenariosRequest' examples: VAPIWebRTCRun: value: transport: vapi_webrtc agent: 12 scenarios: - 101 - 102 frequency: 1 name: Q3 regression concurrency_limit: 5 summary: VAPI WebRTC run application/x-www-form-urlencoded: schema: $ref: '#/components/schemas/RunScenariosRequest' multipart/form-data: schema: $ref: '#/components/schemas/RunScenariosRequest' required: true responses: '200': content: application/json: schema: $ref: '#/components/schemas/RunScenarioResult' description: '' '400': content: application/json: schema: type: object properties: field_name: type: array items: type: string description: '' security: - api_key: [] - oauth2: [] - supabase_session: [] components: schemas: RunScenariosRequest: type: object properties: transport: enum: - voice - sip - text - email - vapi_webrtc - retell_webrtc - elevenlabs - livekit_v2 - pipecat_v2 - chirp - agora - websocket - pipecat_v1 type: string x-spec-enum-id: 971409196dcf842e description: >- How the simulated user reaches the agent. Must be enabled on the agent; `aiagents_retrieve` lists the options under `enabled_connections`. `websocket` and `pipecat_v1` carry their connection details per scenario instead. * `voice` - Voice * `sip` - SIP * `text` - Text * `email` - Email * `vapi_webrtc` - VAPI WebRTC * `retell_webrtc` - Retell WebRTC * `elevenlabs` - ElevenLabs * `livekit_v2` - LiveKit * `pipecat_v2` - Pipecat Cloud * `chirp` - Raw-PCM WebSocket * `agora` - Agora * `websocket` - WebSocket (per-scenario URLs) * `pipecat_v1` - Pipecat (per-scenario room URLs) agent: type: integer description: >- Agent ID to run against. Required unless every selected evaluator already belongs to an agent. scenarios: type: array items: type: integer description: >- Evaluator IDs to run, in order. Provide exactly one of `scenarios`, `tags`, `folder_path`. The dashboard may also send its selection shorthand (`"all"`, a count, or a folder path). tags: type: array items: type: string description: Run every evaluator carrying any of these tags. folder_path: type: string description: >- Dot-separated folder path whose evaluators run. Requires `project_id`. project_id: type: integer description: Project ID. Required with `folder_path`. frequency: type: integer minimum: 1 default: 1 description: >- Times each evaluator runs. Total runs = evaluators × frequency × personalities × test profiles. name: type: string description: Label for the result. maxLength: 255 personality_ids: type: - array - 'null' items: type: integer description: >- Personality IDs to run each evaluator with instead of its own. Only personalities enabled for the project. test_profile_ids: type: - array - 'null' items: type: integer description: Test profile IDs to run each evaluator with instead of its own. concurrency_limit: type: - integer - 'null' minimum: 1 description: 'Cap on runs executing in parallel. Default: unbounded.' calls_per_second: type: - number - 'null' format: double maximum: 4 minimum: 0.1 description: >- Most calls started per second for this run, a number from 0.1 to 4 (e.g. 0.5 starts one call every 2 seconds). Leave unset for no limit. mock_tool_names: type: - array - 'null' items: type: string description: >- Mock tools to activate for this run. Omit to mock every configured tool; send an empty list to mock none. agent_version_id: type: string description: >- Version label or numeric history id of a stored agent version to run against instead of the live agent. outbound_phone_number: type: string description: '`voice`, `text`: override the number dialed, in E.164 format.' agent_number: type: string description: >- `voice`, `text`: override the number the agent under test receives calls on, in E.164 format. mode: enum: - same_number - different_numbers type: string x-spec-enum-id: 9ea0c9743bd9d873 description: >- `voice`: use one phone number for every run (`same_number`) or rotate numbers (`different_numbers`). * `same_number` - same_number * `different_numbers` - different_numbers sip_endpoint: type: string description: '`sip`: override the agent''s SIP endpoint for this run.' websocket_url: type: string description: >- `text`: websocket URL the simulated user connects to instead of the agent's chat gateway. `chirp`: override the agent's raw-PCM websocket endpoint. pattern: ^wss?:// websocket_headers: type: object additionalProperties: {} description: >- `text`: headers sent on the websocket connection, merged over the agent's configured headers. agent_email: type: string format: email description: '`email`: override the agent''s configured email address for this run.' vapi_assistant_id: type: string description: >- `vapi_webrtc`: VAPI assistant ID to test. Defaults to the agent's stored assistant id. retell_agent_id: type: string description: >- `retell_webrtc`: Retell agent ID to test. Defaults to the agent's stored assistant id. elevenlabs_agent_id: type: string description: >- `elevenlabs`: ElevenLabs agent ID to test. Defaults to the agent's stored assistant id. elevenlabs_data: allOf: - $ref: '#/components/schemas/SchemaElevenLabsData' description: >- `elevenlabs`: per-run overrides (`elevenlabs_base_url_override`, `elevenlabs_environment`, `elevenlabs_branch_id`). livekit_data: allOf: - $ref: '#/components/schemas/LivekitRunDataOverride' description: >- `livekit_v2`: per-run overrides (`agent_name`, `url`, `config`). Credentials come from the agent or project. pipecat_data: allOf: - $ref: '#/components/schemas/SchemaPipecatV2Data' description: >- `pipecat_v2`: per-run overrides (`pipecat_agent_name`, `config`, `room_properties`). Credentials come from the agent or project. websocket_urls: type: array items: $ref: '#/components/schemas/WebsocketScenarioEntry' description: >- `websocket`: the evaluators to run, each with its own websocket URL. Replaces `scenarios`. pipecat_rooms: type: array items: $ref: '#/components/schemas/PipecatRoomEntry' description: >- `pipecat_v1`: the evaluators to run, each with its own Daily.co room URL. Replaces `scenarios`. required: - transport RunScenarioResult: type: object properties: id: type: integer readOnly: true description: | Unique identifier for the Result Example: `123` name: type: string description: | Name of the Result Example: `"Test Result 1"` maxLength: 255 agent: type: integer readOnly: true description: | ID of the AI agent that was tested Example: `123` agent_inbound: type: boolean readOnly: true description: > Whether the agent is configured for inbound calls (true) or outbound calls (false) Example: `true` or `false` status: enum: - running - completed - failed - pending - in_progress - evaluating - in_queue - timeout - cancelled - scaling_up type: string x-spec-enum-id: 623e22fc1eb17833 default: pending description: |- Current status of the result * `running` - Running * `completed` - Completed * `failed` - Failed * `pending` - Pending * `in_progress` - In Progress * `evaluating` - Evaluating * `in_queue` - In Queue * `timeout` - Timeout * `cancelled` - Cancelled * `scaling_up` - Scaling Up success_rate: type: number format: double default: 0 description: Percentage of runs that passed, 0 to 100 run_as_text: type: boolean default: false description: | Whether this test was run in text mode instead of voice mode Example: `true` or `false` is_cronjob: type: string readOnly: true description: | Whether this result was created by a scheduled cronjob Example: `true` or `false` runs: type: object additionalProperties: {} description: >- Run summaries for this result. For full run data including transcripts and metric scores, fetch the result detail endpoint — there `runs` is a dict keyed by run ID. readOnly: true created_at: type: string format: date-time readOnly: true description: | Timestamp when this test result was created Example: `2021-01-01 00:00:00` updated_at: type: string format: date-time readOnly: true description: Timestamp when this test result was last updated required: - name SchemaElevenLabsData: type: object description: Per-run ElevenLabs overrides. Credentials are read from the agent. properties: elevenlabs_base_url_override: type: - string - 'null' description: >- Override the ElevenLabs base URL for this run (e.g. a regional or self-hosted endpoint). elevenlabs_environment: type: - string - 'null' description: >- ElevenLabs environment for this run, used to resolve the agent's environment variables (tool URLs, secrets, headers). Defaults to the environment saved on the agent, then to production. elevenlabs_branch_id: type: - string - 'null' description: >- ElevenLabs agent branch for this run. Defaults to the branch saved on the agent, then to the agent's default branch. LivekitRunDataOverride: type: object description: >- Per-run LiveKit overrides. Credentials (api_key/api_secret) are never taken here — they are read from the agent/project ProviderCredential. properties: config: description: >- Room configuration to use for this run (e.g. empty_timeout, max_participants). Fully replaces the agent/project saved config — no merge. agent_name: type: - string - 'null' description: >- LiveKit agent name for explicit dispatch. Overrides the agent/project saved agent_name. url: type: - string - 'null' description: >- LiveKit server URL override. Falls back to the agent/project saved url when omitted. SchemaPipecatV2Data: type: object description: >- Per-run Pipecat Cloud overrides. Credentials are read from the agent/project ProviderCredential. properties: pipecat_agent_name: type: - string - 'null' description: >- Pipecat Cloud agent name. Overrides the agent/project saved pipecat_agent_name. config: description: >- Session configuration for this run. Replaces the agent/project saved config — no merge. room_properties: description: >- Daily.co room properties for this run. Replaces the agent/project saved room_properties — no merge. WebsocketScenarioEntry: type: object properties: scenario: type: integer description: Evaluator ID to run. websocket_url: type: string description: ws:// or wss:// URL this evaluator connects to. pattern: ^wss?:// required: - scenario - websocket_url PipecatRoomEntry: type: object properties: scenario: type: integer description: Evaluator ID to run. pipecat_room_url: type: string description: Daily.co room URL this evaluator joins. pattern: ^https?:// pipecat_token: type: string description: Room token, when the room requires one. call_id: type: string description: Call ID to associate with the Pipecat session. publish_data_message: description: Data message published into the room when the session starts. required: - pipecat_room_url - scenario securitySchemes: api_key: type: apiKey in: header name: X-CEKURA-API-KEY description: >- API Key Authentication. It should be included in the header of each request. oauth2: type: http scheme: bearer bearerFormat: JWT description: OAuth access token issued by Cekura for connected apps. supabase_session: type: http scheme: bearer bearerFormat: JWT description: Cekura dashboard session token. Not a customer API credential. ```` This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.