nora pipelines

Foundry data pipelines from the terminal — nodes, edges, scheduling, publish, runs, logs.

Pipelines are the Foundry surface: ingest, clean, chunk, enrich, index. The CLI mirrors the visual canvas — every node, every edge, every scheduled run is scriptable.

The pipeline lifecycle is the biggest CLI surface after flows — plan on flows use for your Flow and separately reference pipelines by their IDs.

Command groups

• CRUD: pipelines list/get/create/update/delete
• Nodes: pipelines nodes add/list/update/remove
• Edges: pipelines edges connect/list/update/delete
• Schedule: pipelines schedule set/clear
• Runs: pipelines dry-run, pipelines run file/document/folder
• Blocks: pipelines blocks test/ingest-local
• Publish: pipelines publish/rollback, pipelines publications list/live
• Logs: pipelines logs list/get
• Bulk: pipelines sync
• Cloud ingest: pipelines ingest-cloud

Pipeline CRUD

List / get

nora pipelines list
nora pipelines get p_billing_ingest

list gives one row per pipeline. get prints full metadata plus nodes and edges.

Create

nora pipelines create \
  --name "Billing ingest" \
  --stages source,clean,chunk,enrich,output \
  --vector-store qdrant

Flags:

• --name (required) — display name.
• --stages (required) — comma list of stage names in order.
• --id <id> — optional explicit ID; otherwise server mints.
• --vector-store pgvector|qdrant — index backend for the output stage.

Update

nora pipelines update p_billing_ingest \
  --set-name "Billing ingest (v2)" \
  --set-vector-store qdrant

• --set-name <n>, --set-stages <s,…>, --set-vector-store <backend>.

Delete

nora pipelines delete p_billing_ingest

Nodes and edges cascade. Live pipeline versions remain reachable by ID for a grace period.

Nodes

Nodes are the individual blocks in the pipeline canvas — each palette entry (src_local, chunk_markdown, embed_openai, etc.) becomes a node instance.

List

nora pipelines nodes list p_billing_ingest

Prints all nodes as JSON.

Add

nora pipelines nodes add p_billing_ingest \
  --palette-id chunk_markdown \
  --stage chunk \
  --title "Chunk markdown" \
  --icon scissors \
  --tone default \
  --order 2 \
  --config '{"max_tokens": 400, "overlap": 50}'

Flags:

• --palette-id (required) — the palette entry ID.
• --stage (required) — which pipeline stage.
• --title, --icon, --tone (required) — display.
• --order (required) — visual order within the stage.
• --id <id> — explicit ID; server mints if omitted.
• --sublabel <text> — small caption under title.
• --config <json> — block-specific config.
• --notes <text> — free-text notes.
• --output-ports <json> — override the palette’s default output ports.

Update

nora pipelines nodes update n_chunk \
  --set-config '{"max_tokens": 600, "overlap": 100}' \
  --set-title "Chunk (longer)"

--set-* flags: --set-title, --set-sublabel, --clear-sublabel, --set-stage, --set-order, --set-config, --set-notes, --clear-notes, --set-output-ports.

Remove

nora pipelines nodes remove n_chunk

Edges

List / connect / update / delete

nora pipelines edges list p_billing_ingest
nora pipelines edges connect p_billing_ingest --from-node-id n_source --to-node-id n_chunk
nora pipelines edges update e_abc --set-label "raw markdown"
nora pipelines edges delete e_abc

Flags on connect: --from-node-id, --to-node-id (required), --id <id>, --from-port <port>, --label <text>.

Schedule

Set

nora pipelines schedule set p_billing_ingest \
  --cron "0 2 * * *" \
  --tz Asia/Seoul \
  --target billing \
  --sample 100

• --cron <expr> — 5-field cron.
• --tz <IANA> — timezone.
• --target <tag> — the folder tag to process on each run.
• --sample <n> — cap items per scheduled run.

Only flags passed change. Omit --cron to keep the current one, e.g. change only the target.

Clear

nora pipelines schedule clear p_billing_ingest

Wipes all schedule fields — pipeline becomes manual-only.

Running a pipeline

Dry-run against inline text

nora pipelines dry-run p_billing_ingest --input "This is a test document."
# Or from a file:
nora pipelines dry-run p_billing_ingest --input @sample.md
# Or from stdin:
cat sample.md | nora pipelines dry-run p_billing_ingest --input -

Runs the pipeline end-to-end against the given text without persisting output. Prints results per stage.

Run against a file

nora pipelines run file p_billing_ingest --file ./contract.pdf

Multipart-uploads the file and runs the pipeline.

Run against a stored document

nora pipelines run document p_billing_ingest --document-id doc_abc

Runs against a document already in the library.

Run against a folder

nora pipelines run folder p_billing_ingest --tag billing --max-documents 500

Runs against every document tagged billing (up to --max-documents).

Single-block testing

nora pipelines blocks test \
  --palette-id chunk_markdown \
  --input "# Heading\n\nSome content..." \
  --config '{"max_tokens": 200}'

Test one palette block in isolation — no pipeline required. Useful when iterating on chunker or enricher config.

For src_local nodes on a running pipeline:

nora pipelines blocks ingest-local n_source

Kicks off a local-filesystem scan.

Publishing pipelines

Pipelines version and publish like Flows.

Publish

nora pipelines publish p_billing_ingest \
  --summary "chunker: max_tokens 400 -> 600" \
  --author-label "paul"

Freezes DRAFT into a live version. Scheduled runs and downstream consumers use the live version.

Rollback

nora pipelines rollback p_billing_ingest 5

Flips the live pointer to version seq=5.

Publications

nora pipelines publications list p_billing_ingest
nora pipelines publications live p_billing_ingest

list shows history. live fetches the currently-live payload.

Logs

List runs

nora pipelines logs list p_billing_ingest \
  --status ok \
  --since 2026-07-01 \
  --limit 20

Filters:

• --status ok|error
• --search <text> — full-text over log payloads.
• --since <ISO> / --until <ISO>
• --run-kind <kind> — filter by which entry point.
• --limit <n> — max rows.
• --json — machine-readable.

Get one

nora pipelines logs get p_billing_ingest log_abc

Full per-stage detail for one run.

Bulk sync

nora pipelines sync p_billing_ingest --file ./pipelines/billing.json

Replaces the pipeline’s name, stages, nodes, and edges from a JSON file. Idempotent — safe to run in CI on every deploy.

Cloud ingest

nora pipelines ingest-cloud p_billing_ingest n_gdrive_source

Kicks off a manual bulk ingest for a src_cloud node (Google Drive, S3, etc.). Otherwise these run only on the schedule.

Recipes

Bootstrap a minimal pipeline in one script

P=$(nora pipelines create --name "Docs ingest" --stages source,chunk,enrich,output | jq -r '.id')

nora pipelines nodes add $P --palette-id src_local --stage source --title "Local" --icon folder --tone default --order 1
nora pipelines nodes add $P --palette-id chunk_markdown --stage chunk --title "Chunk MD" --icon scissors --tone default --order 1
nora pipelines nodes add $P --palette-id embed_openai --stage enrich --title "Embed" --icon spark --tone default --order 1
nora pipelines nodes add $P --palette-id out_pgvector --stage output --title "Index" --icon database --tone default --order 1

# Wire (need node IDs — capture from add)
# ...

nora pipelines publish $P --summary "initial"

CI: sync pipeline from git

nora pipelines sync p_docs_ingest --file pipelines/docs.json
nora pipelines publish p_docs_ingest --summary "$(git log -1 --pretty=%s)"

Get the most recent errored run

nora pipelines logs list p_billing_ingest --status error --limit 1 --json | jq '.[0]'

Turn off a schedule for maintenance

nora pipelines schedule clear p_billing_ingest
# Do maintenance...
nora pipelines schedule set p_billing_ingest --cron "0 2 * * *" --tz Asia/Seoul --target billing