Annotate examples

Add expected outputs, rubrics, or verdicts so simulation can grade objectively.

An example without a target is just a question. To measure quality, each example needs some way to grade the Agent’s answer. Nora supports several annotation styles — pick the one that fits your domain.

Annotation types

Exact match

Grade passes iff the Agent’s answer exactly equals the expected output. Only useful for structured outputs (JSON, enum, IDs).

Fuzzy match

Grade passes iff similarity above threshold (edit distance, semantic similarity, or a mix). Good for short factual answers where wording flexibility is OK.

Rubric

A checklist the Agent’s answer is graded against:

- Mentions the 30-day refund policy: yes/no
- Includes a link to the refund form: yes/no
- Uses a friendly, apologetic tone: yes/no
- Does NOT commit to a refund without approval: must-not-fail

Rubrics are graded by a judge model. Cheap and reliable for text answers.

Reference answer + judge

Provide a “good” example answer. The judge model compares the Agent’s answer to the reference and scores it on faithfulness, completeness, and correctness.

Good middle-ground when exact match is too strict and a full rubric is too much work.

Custom judge (code)

Write a small JS/Python function that takes the Agent’s answer and returns a score + verdict. Full control — useful when grading involves domain logic (e.g., “answer’s SQL must produce the same rows as expected”).

When to use which

• Structured output (function-calling, JSON) — Exact or Fuzzy match on the structured fields.
• Short factual (“what’s the return policy”) — Fuzzy match or reference + judge.
• Long text (explanations, summaries) — Rubric.
• Domain-specific (SQL, code, calculations) — Custom judge.

You can mix within one dataset — different examples with different annotation types.

Multi-annotator agreement

For datasets where annotation is subjective, add multiple annotators. The dataset viewer shows agreement stats — examples with high disagreement are worth revisiting.

Nora doesn’t force consensus. It shows all annotations and lets simulation pick which annotator’s verdict to use.

Adding annotations

Per-example annotation editor. Or bulk-annotate:

• Filter to a subset (by tag, by score, by “unannotated”).
• Apply the same rubric or reference to all.
• Review individually if needed.

Annotation from feedback

If your Signals include text corrections, the corrections auto-populate the expected output when saving to a dataset. Saves the annotation step for anything with existing feedback.

Reviewing annotations

Datasets in review show:

• Unannotated — examples without a target. Won’t be scored in simulation.
• Needs review — auto-populated from feedback, human should verify.
• Ready — human-annotated and confirmed.

Simulation only scores ready examples by default. Toggle to include needs review if you want a broader test.

Editing later

Annotations are versioned. Editing one creates a new version; simulation results are pinned to the annotation version they used. So old simulation results stay valid even as annotations evolve.