ProcureGraph — Observability Design (non-negotiable)

Synthesized from research/05-observability.md (verified packages, snippets, sources). Requirement: track EVERYTHING — traces, logs, LLM calls, prompt quality, money — with numbers, so it can be improved.

1. Stack decisions

Concern	Decision
Tracing core	OpenTelemetry Python SDK; one global TracerProvider configured FIRST
Trace backend + LLM tracing	Self-hosted Langfuse v3 (official compose, vendored via include:) — OTel-native; FastAPI + SQL + httpx + LLM spans land in ONE trace UI; zero internet dependency during the live demo
OpenAI tracing	from langfuse.openai import AsyncOpenAI drop-in (prompts, completions, tokens, cost)
LangGraph tracing	langfuse.langchain.CallbackHandler per invocation + explicit spans (policy.evaluate, scorer.rank) with procure.* attributes
Auto-instrumentation	opentelemetry-instrumentation-{fastapi,httpx,sqlalchemy,psycopg}
Logs	structlog JSON → stdout; processor injects trace_id/span_id; contextvars bind mission_id/task_id/payment_id
Metrics	prometheus-fastapi-instrumentator /metrics + domain counters (payments_blocked_total, budget_spent_usd, prompt_calls_total, prompt_latency_seconds). Grafana/Prometheus containers = stretch only
UI timeline	SSE (sse-starlette), events from the append-only ledger, Last-Event-ID resume. Not WebSockets
NOT building	Jaeger/Tempo, Loki, alerting, Grafana (unless evening 5 has slack)

Init order gotcha (verified): configure our TracerProvider, THEN Langfuse() so its span processor attaches to the existing provider. Handlers constructed per request (asyncio context pitfall).

2. Correlation strategy — one vocabulary, four surfaces

ID	Span attribute	Log field	Langfuse	Ledger column
mission	procure.mission_id	mission_id	session_id	mission_id
task	procure.task_id	task_id	metadata	task_id
payment	procure.payment_id	payment_id	metadata	payment_id
trace	intrinsic	trace_id	trace id	trace_id

IDs minted in the domain layer; bound at the boundary where they enter scope. Every ledger event row carries the current trace_id → the UI renders a "view trace" deep link per timeline item.

3. Prompt quality as numbers (extension research; authority fixed per review A23/A24)

• Canonical store: our DB. prompt_versions.content is what runtime executes — always by exact pinned version id, never by label or "latest". Langfuse Prompt Management mirrors versions for the metrics UI (langfuse_prompt linking → per-version cost/latency/scores tab); labels are promotion metadata only.
• The gate is pytest. The credentialed eval job (make eval, live gpt-4o-mini against the pinned corpus — see design/06 §3) runs deterministic Python evaluators (schema_validity, extraction_accuracy, injection_resistance, hierarchy, no-math, determinism ×5), stores raw outputs + system_fingerprint + scores in the eval_report, and sets prompt_versions.gated_at only if every gate metric is 100% and determinism is 5/5 (extraction accuracy is tracked/trending, not a hard gate — design/05 §4). Langfuse run_experiment mirrors the same run as a dataset run for the UI; it is reporting, never authority.
• Online scores: same metric names scored onto live traces at the moment of truth (e.g. injection_resistance = 1.0 on the blocked $97 payment's trace).
• Prometheus: prompt_calls_total{prompt_name,prompt_version,outcome} + prompt_latency_seconds{...}; rates computed at query time.
• Acceptance test (review B9): an integration test proves a prompt version emits the required numeric scores (Langfuse API query), Prometheus series exist, and the UI trace link resolves.

4. What an evaluator sees (the observability story)

1. Mission control timeline streams live (SSE) through compile → benchmarks → selection → card → purchase.
2. Timeline turns red: payment.blocked ($97 gift cards). Click view trace.
3. Langfuse opens the exact trace: LLM generation shows the malicious provider text; policy.evaluate span shows procure.policy_verdict = BLOCK with the SQL budget-check spans in the same waterfall — visual proof the deterministic engine, not the LLM, blocked it.
4. Session view (session_id = mission_id): every LLM call with cost next to the $20 budget — the mission's own observability bill is reconciled too.
5. Prompt Metrics tab: injection_resistance 0.6 → 1.0 across prompt versions; the live blocked trace carries the same score name.
6. Closer: docker compose logs api | grep payment_id=… → JSON logs with the same trace_id.