Skip to main content
Writing a metric that behaves exactly the way you intend usually takes a few rounds of trial and error — write a definition, preview it on some calls, notice a case it gets wrong, tighten the wording, repeat. Build with AI collapses that loop into a single conversation. When you click Improve on a metric’s definition, an AI agent:
  1. Cleans up your definition and gives it a quick check. If the intent is unclear or the definition has an obvious gap, it asks you a short question up front — before spending a run on calls it can’t yet judge meaningfully.
  2. Runs the definition against a sample of the agent’s real calls and runs.
  3. Reviews the predictions and finds edge cases — early hang-ups, partial completions, off-topic calls, caller-vs-agent fault, voicemail, transfers, and so on.
  4. Resolves the clear cases itself by tightening the definition, and asks you a short question only when a verdict depends on a policy it cannot infer (for example, “should a partial completion count as PASS?”).
  5. Proposes a refined definition for you to review and apply.
Build with AI works for both LLM Judge metrics (it refines the natural-language description) and Python metrics (it makes surgical edits to your code — see below). You commit the final metric yourself; nothing is saved until you click Use this metric and then Create/Update.

​
Starting a build

1

Write a starting point

For an LLM Judge metric, write a first-pass description of what success looks like. For a Python metric, write your initial evaluation code. You don’t need to get it perfect — that’s what the builder is for.
2

Click Improve

The Improve button sits next to the Description (LLM Judge) or the Custom Code editor (Python). Pick the agent whose calls and runs the builder should sample from.
3

Chat with the builder

The builder works through your calls and runs and either asks a clarifying question or presents a refined definition. Answer questions by selecting an option (the recommended default is marked) or typing your own guidance. You can also message the builder at any time — for example, “make it stricter about confirming the appointment time.”
4

Review and apply

When it’s done, the builder shows a diff of the proposed definition against your original, plus a short list of the edge cases it handled. Click Use this metric to drop the refined definition into the form, then save.

​
Working in the background

A build runs an AI agent over dozens of real calls and runs, so it typically takes a few minutes (usually 4–6). You don’t have to wait on it:
  • Minimize the chat and it keeps running as a small card in the corner.
  • The card follows you across the platform — open another metric, inspect the example calls the builder linked, or check a run, and the build keeps going.
  • Reopen the card any time to see progress, answer a question, or apply the result.
The card shows the current step and elapsed time so you always know where the build is.

​
Python metrics

For a Python metric the builder works in code. Before it runs anything, it cleans up your starting point — fixing syntax errors and obvious bugs — so a rough first draft doesn’t stall the build. From there it runs your code on the sampled calls and runs and makes the smallest change that fixes each case it gets wrong, keeping it valid, runnable Python in the same style as what you wrote.
You don’t have to start from code. Describe the metric in plain English in the Python editor and the builder turns it into a working Python metric on the first pass — then refines it against your real calls and runs like any other build.
Everything the builder produces is the exact code the metric will run in production, so what you review is what you ship.

​
Credits

A build runs your metric against the sampled calls and runs, and consumes credits for each pass, priced the same way the metric optimizer is:
  • LLM Judge metrics — and Python metrics that call evaluate_llm_judge_metric — are charged at the LLM-judge per-evaluation rate.
  • Python metrics that judge audio (calling evaluate_llm_judge_metric with audio=True) are charged at the audio per-minute rate times each call’s duration. A time window (audio_start_time/audio_end_time) changes what the judge listens to, not the charge.
  • Pure Python metrics are charged at the custom-code per-evaluation rate.
The cost of a pass is roughly the number of sampled items (≈50) times that rate. A pass runs only when the metric definition actually changed, so answering a clarifying question or sending a note that doesn’t alter the metric won’t re-charge. Minimizing the chat doesn’t stop the build; if you don’t want to spend more credits, Stop it.