Skip to main content
Writing a metric that behaves exactly the way you intend usually takes a few rounds of trial and error — write a definition, preview it on some calls, notice a case it gets wrong, tighten the wording, repeat. Build with AI collapses that loop, from two places:

Quick Mode

The default view of the Create Metric page. Pick an agent, describe what to measure, and click Generate Metric — the AI drafts the metric from your description and the agent’s context and saves it for you, usually in 1–2 minutes. No form to fill, no evaluation credits spent.

Improve (detailed build)

The Improve button next to a metric’s definition runs the draft against a sample of your agent’s real calls and runs first, then refines it in a conversation. Slower (5–12 minutes) and it spends evaluation credits, but it is the only path that can find edge cases you haven’t thought of.
Both paths work for LLM Judge metrics (refining the natural-language description) and Python metrics (writing and surgically editing code — see below). Quick Mode saves the metric itself and links you to it when it’s done; an Improve build commits nothing until you click Use this metric and then Create/Update.
Use Improve when a metric matters and you want it grounded in real transcripts. Use Quick Mode to get a clean, working metric fast, then run an Improve build later once the agent has call history worth testing against.

​
What a detailed (Improve) build does

  1. Cleans up your definition and gives it a quick check. If the intent is unclear or the definition has an obvious gap, it asks you a short question up front — before spending a run on calls it can’t yet judge meaningfully.
  2. Runs the definition against a sample of the agent’s real calls and runs.
  3. Reviews the predictions and finds edge cases — early hang-ups, partial completions, off-topic calls, caller-vs-agent fault, voicemail, transfers, and so on.
  4. Resolves the clear cases itself by tightening the definition, and asks you a short question only when a verdict depends on a policy it cannot infer (for example, “should a partial completion count as PASS?”).
  5. Proposes a refined definition for you to review and apply.
A Quick Mode build does steps 4 and 5 only — it reasons about the definition itself rather than about predictions, makes the most reasonable assumption when your description leaves a decision open (and lists what it assumed on the saved metric’s summary), and only stops to ask when the metric genuinely can’t be defined safely without you.
Not enough history? A detailed build needs at least 10 calls or runs to sample. A brand-new agent doesn’t have them, so the build runs as a quick one instead of stalling on an empty sample — it says so in the chat, and no evaluation credits are spent. A later Improve build uses real transcripts once the agent has them.

​
Quick Mode — generate a metric

1

Open Create Metric

Metrics → Create Metric lands on Quick Mode. (The Detail Mode toggle switches to the full form if you’d rather configure everything yourself.)
2

Describe what to measure

Pick the agent and describe the metric in plain language — what should pass, what should fail, anything the verdict depends on. Quick Mode builds it as a Python metric (deterministic and cheap to evaluate, calling the LLM judge inline only where a language judgment is genuinely needed) and infers the verdict shape — pass/fail, rating, number, or categories — from your description.
3

Generate

Click Generate Metric. You’re taken back to the metrics list while the build runs in the bottom-right progress card; when it finishes, the card shows Metric created with a link straight to the saved metric — named, typed, and enabled on the agent you picked.

​
Improve — refine against real calls

1

Write a starting point

For an LLM Judge metric, write a first-pass description of what success looks like. For a Python metric, write your initial evaluation code. You don’t need to get it perfect — that’s what the builder is for.
2

Click Improve

The Improve button sits next to the Description (LLM Judge) or the Custom Code editor (Python). Pick the agent to build against.
3

Chat with the builder

The builder either asks a clarifying question or presents a refined definition. Answer questions by selecting an option (the recommended default is marked) or typing your own guidance. You can also message the builder at any time — for example, “make it stricter about confirming the appointment time.”
4

Review and apply

When it’s done, the builder shows a diff of the proposed definition against your original, plus a short list of the edge cases it handled. Click Use this metric to drop the refined definition into the form, then save.

​
Working in the background

A Quick Mode build usually takes 1–2 minutes and runs in the corner card from the start — there is no chat to watch. A detailed build runs an AI agent over dozens of real calls and runs, so it typically takes 5–12 minutes — longer for a multi-turn Python build. Either way, you don’t have to wait on it:
  • Minimize the builder chat and it keeps running as a small card in the corner.
  • The card follows you across the platform — open another metric, inspect the example calls the builder linked, or check a run, and the build keeps going.
  • Reopen the card any time to see progress, answer a question, or apply the result — and a finished Quick Mode build’s card links straight to the saved metric.
The card shows the current step, the elapsed time, and the typical range for the mode you’re running, so you always know where the build is.

​
Python metrics

For a Python metric the builder works in code. Before it runs anything, it cleans up your starting point — fixing syntax errors and obvious bugs — so a rough first draft doesn’t stall the build. An Improve build then runs your code on the sampled calls and runs and makes the smallest change that fixes each case it gets wrong; a quick build makes the same kind of surgical edits, reasoning from the code and the agent’s description rather than from predictions. Either way it stays valid, runnable Python in the same style as what you wrote.
You don’t have to start from code. Quick Mode always works this way — plain English in, working Python out — and the same applies in the Python editor: describe the metric in plain English and the builder turns it into runnable code on the first pass, then refines it from there.
Everything the builder produces is the exact code the metric will run in production, so what you review is what you ship.

​
Credits

What a build costs depends entirely on the mode. Quick builds (Quick Mode, or an Improve build that fell back to quick) run no evaluation pass, so there is nothing to meter per call. They are charged a small flat amount once per build, however many clarification rounds it takes. A build that fails is not charged. Detailed builds run your metric against the sampled calls and runs and consume credits for each pass, priced the same way the metric optimizer is:
  • LLM Judge metrics — and Python metrics that call evaluate_llm_judge_metric — are charged at the LLM-judge per-evaluation rate.
  • LLM Judge metrics with voice recording enabled, and Python metrics that judge audio (calling evaluate_llm_judge_metric with audio=True), are charged at the audio per-minute rate times each call’s duration. A time window (audio_start_time/audio_end_time) changes what the judge listens to, not the charge.
  • Pure Python metrics are charged at the custom-code per-evaluation rate.
The cost of a pass is roughly the number of sampled items (≈50) times that rate. A pass runs only when the metric definition actually changed, so answering a clarifying question or sending a note that doesn’t alter the metric won’t re-charge. Minimizing the chat doesn’t stop the build; if you don’t want to spend more credits, Stop it. Your current rates are on the Billing page.