Pipeline observability — local Prometheus + Grafana via Caddy¶

This repository builds observability in from the ground up. Every stage emits structured events to a JSONL sidecar from its first commit; an exporter tails the sidecars and serves Prometheus metrics; Grafana renders the operator dashboard; Caddy fronts both behind a single http://localhost:8080 URL.

Status: implemented (see docs/plans/p-059-prometheus-exporter.md). bffi-pipeline serve-metrics tails every runs/*/stage-events.jsonl and serves the metric set below on :9100. The emitter is activated by the run-directory check every stage's --output-dir passes through, so any invocation writing into a canonical run directory produces a sidecar.

No outbound telemetry. The whole stack runs in Docker Compose on the operator's machine; no data leaves the box.

Architecture¶

flowchart TD
    subgraph Stages["Pipeline stages"]
        S2[marc-to-bibframe]
        S3[bibframe-to-bffi]
        S4[bffi-to-marc]
        S5[roundtrip-eval]
    end

    subgraph Sidecars["Per-run sidecars"]
        SJ[stage-events.jsonl]
    end

    subgraph Exporter["metrics-exporter
host :9100"]
        E["/metrics endpoint"]
    end

    subgraph Docker["Docker Compose"]
        PROM[Prometheus
:9090]
        GRAF[Grafana
:3000]
    end

    subgraph Runs["runs/ directory"]
        R[per-record artifacts
diff TSVs, review HTML]
    end

    subgraph Caddy["Caddy reverse proxy
127.0.0.1:8080"]
        CADDY[Caddy]
    end

    S2 & S3 & S4 & S5 --> SJ
    SJ -->|tail| E
    E -->|scrape| PROM
    PROM -->|query| GRAF
    PROM & GRAF --> CADDY
    CADDY -->|/grafana/| GRAF
    CADDY -->|/prometheus/| PROM
    CADDY -->|/files/| R

All local. The exporter runs on the host alongside the conversion pipeline (it shares the data dir directly, no volume mapping); Prometheus, Grafana, and Caddy run as Docker Compose services. Operators reach the stack through a single URL; Caddy routes paths to the right backend and serves the runs/ directory as static files so per-record diff artifacts are clickable from dashboard links.

Event sidecar (the contract)¶

Every stage writes JSON-lines to runs/<run_uuid>/stage-events.jsonl. Events are append-only, schema-versioned, and small. Event types:

Adding a new event type is a code change; the metric vocabulary table below is the spec the stage side, exporter, and dashboard must agree on.

Metric vocabulary¶

The exporter publishes the canonical metric set below. Every metric maps one-to-one to a sidecar event field.

All metrics are Gauges, including the _total-suffixed ones: the sidecar carries absolute cumulative values, so replaying it into a counter would double-count whenever the exporter restarts and re-reads from offset 0. A gauge holding the last observed absolute value is the honest primitive; the _total names are kept because the dashboard queries them.

Label cardinality¶

• stage ∈ {melinda-sync, marc2bibframe, bibframe2bffi, bffi2marc, roundtrip_eval} — this repository's ingestion + bidirectional-conversion stage set. bffi2marc is the reverse direction (BFFI graph → reconstructed MARCXML); roundtrip_eval is the diff harness that compares reconstructed MARC against the source. Extend as new stages land.
• phase ∈ {_, phase1, …}. _ is the sentinel for stages without internal phases.
• outcome is per-stage but bounded: success / failed for every stage, plus closed_namespace_residue and one routing_<name> per discriminator routing for bibframe2bffi.
• error_type is the exception class name the stage caught (XsltprocError, BibframeToBffiError, …) — bounded by the number of failure modes, not by records.
• run_uuid is one value per pipeline invocation.

Cardinality cap (per run_uuid): ~5 stages × ~3 phases + ~30 outcomes + a handful of error types ≈ 60 series. No label carries per-record data, so the cap holds at 800k records as it does at 300. Multiplied by accumulated runs in the current exporter session — well within Prometheus's comfort zone for any realistic dev / production cadence.

Exporter lifecycle¶

The exporter is a small Python program. Five phases:

PID + argv files sit at <runs-root>/.exporter.pid and .exporter.argv — process-global, not per-run, because the exporter is multi-tenant over its sidecars.

Operating modes¶

Mode A — ambient observer (default): one long-lived exporter with --watch-glob 'runs/*/stage-events.jsonl'. Initial walk attaches every existing sidecar; the 30 s rescan picks up new runs as they spawn. The dashboard's active_run dropdown shows new runs within ~30 s — no exporter restart needed.

Mode B — per-run focused bench: single --watch-glob 'runs/<uuid>/stage-events.jsonl'. Useful for an isolated A/B test where you want zero noise from other runs. Operator kills the exporter when done.

Mode A is the durable shape. The exporter never owns the run — it's a passive tail of the run's on-disk JSONL output.

Exporter restarts¶

Restarting the exporter re-attaches every sidecar at offset 0 and replays it. Because every metric is an absolute-value Gauge, the replay lands on exactly the values the previous process had — no reset, no visible discontinuity on the dashboard, and no double-counting. The JSONL on disk is the ground truth; the exporter holds no state the sidecars don't.

The one exception is bffi_stage_errors_total, which accumulates one increment per failed row. A replay reconstructs it from the same rows, so the total is stable across restarts too.

Per-run metric isolation¶

Every metric carries an explicit run_uuid label. Dashboards filter every panel by run_uuid="$active_run", where $active_run is a Grafana templating variable populated by label_values(bffi_stage_started_timestamp, run_uuid). The operator picks the run they want to watch; panels re-render against that single run_uuid. Prior runs' data remains queryable in Prometheus under their own values for forensic comparison.

Bundled Grafana dashboard¶

Auto-loaded from config/grafana/dashboards/bffi-pipeline.json at container start (via config/grafana/provisioning/). Read-only in the UI; operators clone-and-edit if they want a custom view.

Panel set:

Extending¶

Adding a new metric is two edits:

1. Stage side: extend the relevant event's payload at the emit_if_active call.
2. Exporter side: declare the metric in the exporter's metric set and route the event's payload to it.
3. Dashboard (optional): add a panel to the JSON.

The metric vocabulary table above is the spec. Bump the table first; the code follows.

Outstanding¶

• bffi_stage_throughput_per_minute and bffi_stage_eta_seconds derive from a 5-event sliding window, so they stay absent until a stage has emitted two progress events.