Create a dataset

Four ways to build one — from traces, from clusters, from files, from scratch.

From production traces

The most common source. In Traces, filter to the runs you want (see Search & filter), then Add to dataset.

Two flavors:

• Trace-as-example — the full trace becomes the example. Input = original input, expected output = the answer that was given (marked “capture as-is”).
• Corrected-example — you edit the expected output to what the answer should have been.

Bulk: filter to hundreds of traces, click Add all to dataset. Nora asks whether to include corrections (if any exist as feedback) or capture as-is.

From a signal cluster

Every cluster represents failing patterns. Right-click any cluster → Save as dataset:

• Each signal’s trace becomes an example.
• If the signal has a text correction, that becomes the expected output.
• Otherwise, mark as “needs review” — you’ll annotate before running simulations.

This is often the best pattern: the failures are the tests you want, and Nora already grouped them for you.

From a file

Upload a CSV or JSONL:

CSV:

input,expected_output,tags
"How do I reset my password?","Go to Settings → Security → Reset Password","auth,customer"
"When is my invoice due?","Invoices are due on the first of each month","billing,customer"

JSONL (more flexible — supports structured inputs and outputs):

{"input":{"question":"reset password"},"expected":{"answer":"..."},"tags":["auth"]}

Files are the way to import datasets from external QA systems.

From scratch

Datasets → New → Add example. Type or paste input, type expected output, tag. Fastest for a handful of hand-crafted examples to bootstrap testing.

Multi-turn examples

Some Flows are conversational. Multi-turn examples are supported:

• Each example is a conversation (list of turns).
• Each turn has expected inputs and expected outputs.
• Simulation replays the conversation and grades each turn.

Import format:

{"conversation":[
  {"role":"user","content":"..."},
  {"role":"assistant","content":"expected answer"},
  {"role":"user","content":"follow-up"},
  {"role":"assistant","content":"expected answer 2"}
]}

Tags and grouping

Every example can carry tags. Common uses:

• Feature — which product feature this exercises.
• Difficulty — easy / medium / hard.
• Source — where the example came from (prod, synthetic, hand-crafted).
• Language — for multilingual coverage.

Simulation can be run against a tag subset — “run the customer-facing hard cases only”.

Metadata per example

Beyond input/expected/tags, examples can carry:

• Scope — for testing per-tenant/per-user behavior.
• Attached context — session memory, prior turns, feature flags.
• Notes — free-text for reviewers.

Match production runtime as closely as possible so simulation results reflect real behavior.

Editing after import

Every example is editable in place. Bulk edit tags or notes via multi-select. Delete individual examples anytime.