> ## Documentation Index
> Fetch the complete documentation index at: https://docs.platform.nora.my/llms.txt
> Use this file to discover all available pages before exploring further.

# Holdout sets

> Test data the Improvement flow never sees — the honest measure.

If you evaluate a fix on the same data you used to design it, your quality number is inflated. That's the "overfitting" problem in one sentence.

A **holdout** is a subset of your data that the Improvement flow is not allowed to touch. Simulation runs against it. Any signal there passes the honesty bar.

## Creating a holdout

From any dataset, click **Split → Create holdout**. Options:

* **Random N%** — take a random 20% (default).
* **By tag** — hold out everything with certain tags (e.g., hold out `hard` examples).
* **By date range** — hold out examples added after a cutoff (evaluates future-proofness).
* **Manual pick** — check individual examples.

The holdout becomes a separate dataset. It's cross-referenced back to the source dataset.

## Holdout enforcement

Once marked, the Improvement flow's proposal generation cannot read the holdout examples. Diagnosis, root-cause analysis, and lever selection use only the non-holdout portion.

Simulation, however, runs proposals against **both** portions:

* **Train portion** — where the improvement was designed. Shows the "in-distribution" improvement.
* **Holdout portion** — the honest signal. Shows generalization.

Big gap between the two = overfitting.

## The overfitting warning

If a proposal massively improves train performance but doesn't budge holdout performance, Nora warns:

> This proposal moved train score +12% but holdout +0.5%. Likely overfits to the failing examples used in the improvement flow. Consider broadening the improvement pattern.

Warnings don't block — you can still ship — but they're a strong hint.

## Rotation

Holdouts should rotate. If the same examples are always the holdout, they become stale — the Improvement flow can't cover the patterns they test.

Recommended:

* Rotate the holdout monthly (or per quarter).
* Add new holdout examples as new failures are fixed.
* Retire holdout examples when they become trivial (always passing).

Rotation is manual (you decide when). Nora shows a "holdout age" indicator when it's been > 60 days without changes.

## Holdout ≠ regression suite

Both are test-only. Difference:

* **Holdout** — general representative sample of your traffic. Broad coverage.
* **Regression suite** — specific known failures you want to guard against forever. Narrow, defensive.

Keep them separate. A change might pass one and fail the other, and the interpretation is different.

## Enterprise: production holdout

For high-stakes Flows, you can configure Nora to **sample a percentage of production traffic** as an ongoing holdout. Runs land in the dataset automatically, get human-annotated (or judge-annotated) offline, and expand the holdout organically.

Turn on in Flow settings → **Sampling → Production holdout**.

## Splits and simulation

Simulation reports separately for each portion:

* Train: 47/50 pass.
* Holdout: 18/20 pass.
* Overall: 65/70 pass.

The overall number is convenient. The two portions individually are what matters — the delta tells you where you stand on generalization.