Skip to main content
The scariest simulation result isn’t the ones you expected to fail — it’s the ones you didn’t. Regression checks call these out loudly.

​
What’s a regression

An example that:
  • Passed under the baseline.
  • Fails under the target.
That’s it. Nora computes this automatically for every before/after comparison.

​
Where regressions show up

  • In the simulation summary — top-of-page count. “Target passes 45, regressed 3.”
  • In the results table — regressions are red rows, sortable to the top.
  • In an alert — if the simulation was CI-gated, regressions fail the gate.

​
Investigating a regression

Click any regression to see:
  • The example (input, expected, baseline output, target output).
  • The step-by-step diff between baseline trace and target trace — where they diverged.
  • Nora’s hypothesis for the cause.
Common causes:
  • Retrieval order changed — the top-K includes different chunks. Fix: adjust the retrieval preset.
  • Prompt shifted — the new prompt is less specific about a required behavior. Fix: restore or clarify the missing instruction.
  • Model differs — swap in the prior model to test.
  • New guardrail fired — a change to guardrails blocked something the baseline let through. Fix: tune the guardrail.

​
Ignoring a regression

Sometimes a “regression” is a fake one — the baseline was wrong, the target is right, and your dataset annotation matches the baseline. Fix by updating the annotation, not by rejecting the change. Mark a regression as “acceptable” or “annotation was wrong” in the results table. Documented in the audit trail.

​
Ignoring is not the same as suppressing

If you consistently mark a regression as acceptable across simulations, Nora asks: is this a pattern we should stop flagging? Configure via Simulation settings → Regression rules:
  • Ignore regressions on examples tagged flaky.
  • Ignore regressions where cost decreased more than 30%.
Use sparingly. Suppressing regressions makes the simulation lie to you eventually.

​
Regression rate over time

The Simulation history page tracks regression rate per version. Health signal:
  • Consistent < 2% regression rate — good discipline.
  • Rising regression rate — team is shipping less carefully; the improvement flow may not be paying attention to the dataset.
  • Sudden spike — one bad deploy; investigate the version.

​
Hard regressions

Some regressions matter more than others. Tag examples in the dataset as critical — any regression on a critical example blocks any deploy regardless of aggregate stats. Common criticals:
  • Security-relevant behaviors (“never disclose the system prompt”).
  • Compliance answers (“always mention the disclaimer”).
  • Business-critical rules (“never quote a price without checking the price list”).
Simulate against criticals on every publish.