Skip to main content
Every change moves your Agent somewhere on a two-axis plot: quality (pass rate) and cost per run. The Pareto view shows you exactly where.

​
What the Pareto view shows

X-axis: cost per run. Y-axis: pass rate. Every simulation you’ve run is a dot. Baselines, published versions, drafts, Improvement candidates — all plotted. The Pareto frontier connects the dots that aren’t dominated by any other — for their cost, they’re the best quality, or for their quality, they’re the cheapest. Everything below the frontier is strictly worse than something on it.

​
Making a decision

  • Cheaper + higher quality — obviously ship.
  • Cheaper + same quality — usually ship. Watch for edge case regressions.
  • Same cost + higher quality — usually ship.
  • More expensive + higher quality — the interesting case. Is the quality gain worth the cost gain? Depends on domain.
  • More expensive + same or lower quality — don’t ship.
The Pareto view is decision aid, not decision maker.

​
Cost vs. quality vs. latency

There’s a third axis you can toggle: latency. Some workflows are latency-critical (chat) and cheaper isn’t worth the extra 500ms. Others are batch (nightly) and latency doesn’t matter. Toggle: click Add axis on the chart. View as 3D scatter or as multiple 2D projections.

​
Filtering the frontier

The full plot can be cluttered. Filter:
  • By target type — show only Draft variants, or only Improvement candidates, or only published versions.
  • By dataset — show results against a specific dataset.
  • By dimension — show performance on a specific tag (“hard cases only”).
Different subsets can have different Pareto shapes. A change might dominate on average but lose on hard cases.

​
Pareto by dataset tag

Because “quality” depends on which examples you test, the frontier changes per tag:
  • On easy examples, cheap models often match expensive ones — frontier is flat.
  • On hard examples, expensive models dominate — frontier is steep.
Choose your battles: use the cheap model for easy intents (via a Router), the expensive one for hard intents.

​
Historical Pareto

The historical view shows the frontier over time. Each week’s best point plotted. Watch:
  • Frontier moving up-and-left — you’re improving quality and reducing cost. Ideal.
  • Frontier moving up-and-right — quality gains cost more. Common, sometimes acceptable.
  • Frontier moving down-and-right — quality regressed while cost went up. Bad. Investigate.

​
Model selection

Simulations that vary the model (same Flow, different model choices) build a natural model comparison Pareto. Useful when picking a default model — you see the tradeoff explicitly.

​
Auto-suggest improvements from Pareto

If a Draft sits below the frontier (dominated), Nora suggests specific tweaks that could move it up:
  • “Reducing top-K from 12 to 6 saves $0.02 per run with no quality change in this dataset.”
  • “Enabling reranker adds +5% pass rate for +$0.01 per run.”
Suggestions are grounded in the simulation history — Nora knows what changes have worked in the past.