Skip to content

ProcureGraph — Architecture Design (rev 3)

Synthesized from docs/research/*.md; revised after codex review (docs/review/). Governing principle: the model proposes, deterministic code scores and authorizes, the payment rail executes.

1. System context

flowchart LR
    U[User / Judge] --> W[React mission control<br/>Vite + TS]
    W -->|REST + SSE| A[FastAPI monolith<br/>Python 3.12]
    A -->|LangGraph 1.2| G[Procurement state machine]
    A --> P[(PostgreSQL 16<br/>ledger + budgets + prompts)]
    G -->|x402 v2 exact| F[Monad facilitator<br/>x402-facilitator.molandak.org]
    F --> M[(Monad testnet 10143<br/>USDC 0x534b…43A3)]
    G -->|RainCardIssuer port| R[Rain adapter<br/>MOCK now → sandbox later]
    G -->|paid HTTP| PR[Provider mocks<br/>ExactFlow · BudgetFlow · LegacyBatch]
    A -->|OTel spans| L[Langfuse v3 self-hosted]

Rain API (rain.xyz / raincards.xyz — see research/01 §0; rain.one is a different company): prod https://api.raincards.xyz/v1/issuing, dev https://api-dev.raincards.xyz/v1/issuing, Api-Key header auth.

Terminology: Mission ≡ Mandate execution (1:1); mission_id = the mandate UUID, used as graph thread_id, ledger key, log field, span attribute, and Langfuse session id.

2. Layering (Clean Architecture, dependency arrows point inward)

backend/src/procuregraph/
├── domain/            # pure Python 3.12; ZERO third-party imports (import-linter enforced)
│   ├── shared/        # Money (int minor units), ids, DomainEvent, Clock protocol
│   ├── mandate/       # Mandate aggregate
│   ├── budget/        # BudgetPool aggregate + journal entities
│   ├── payment/       # PaymentProposal aggregate + state machine
│   ├── evidence/      # EvidenceLedger domain service (append-only, hash-chained)
│   ├── policy/        # Specification rules → verdicts
│   └── agents/        # AgentDefinition aggregate + PromptVersion (ADR-003)
├── application/       # organized BY CAPABILITY (design/07 §2 is normative; no global dto.py)
│   ├── procurement/   # compile_mandate, discover_providers, plan_evidence,
│   │                  #   run_audition, assess_quorum, select_provider
│   ├── payment/       # authorize_and_reserve, prepare_rail, execute_payment,
│   │                  #   confirm_settlement, reconcile_rail_status, compensate
│   ├── verification/  # verify_outcome, remediate_outcome, reconcile
│   ├── agents/        # create_agent, generate_prompt_version,
│   │                  #   activate_prompt_version, freeze_agent
│   ├── ports/         # one consumer-driven protocol per file (design/07 §4)
│   └── common/        # shared command/result base types only
├── infrastructure/
│   ├── db/            # SQLAlchemy 2.0 async models, repositories, SqlAlchemyUoW,
│   │                  #   outbox, ledger-head serialized append
│   ├── rails/x402/    # real Monad testnet adapter (x402==2.17.0, exact scheme)
│   ├── rails/rain/    # MockRainCardIssuer (issuer swap = this file only)
│   ├── llm/           # gpt-4o-mini adapter: strict json_schema, temp 0, fixed seed
│   ├── graph/         # LangGraph wiring — nodes are THIN callers of use cases only;
│   │                  #   business logic in graph nodes is an architecture-test failure
│   └── telemetry/     # OTel + Langfuse bootstrap, structlog, correlation middleware
├── interfaces/
│   ├── api/           # FastAPI routers, Pydantic schemas, SSE endpoint
│   └── cli/           # deterministic demo driver + demo-verify manifest
└── composition.py     # THE composition root (manual DI; no framework — research 06)

Sibling apps: frontend/ (React) and providers-mock/ (two x402-paid provider services + LegacyBatch merchant checkout wired to the mock authorization simulator).

DI decision: manual composition root in the FastAPI lifespan; container dataclass on app.state; Depends(get_container) only at the HTTP edge. Enforced by an architecture-level test (import-linter contracts + a composition-root unit test).

Code craft is normative in design/07-code-standards.md: SOLID mapped to enforced rules — the file-by-file domain tree (many small files, one public concept each, one policy rule per file), consumer-driven segregated ports (the port table in 07 §4 supersedes the coarse port names above where they differ — e.g. RainCardIssuer is implemented as CardIssuancePort + CardLifecyclePort + CardTransactionsQueryPort + RainWebhookVerifierPort, all satisfied by one adapter class), complexity caps, typed error hierarchy, and the review-rejection checklist that gates every task.

3. Bounded contexts & aggregates

Aggregate Key invariants Notes
Mandate pool allocations ≤ total budget; status machine (lowercase states) compiled by LLM in minor units/bp; sums re-derived by code
BudgetPool available balance never negative; movements via balanced journals; optional parent_id nesting (demo seeds 3 flat pools) the transactional money boundary
PaymentProposal state machine §5; one live attempt; verdict recorded; settled proposals immutable — new spend = new proposal references mandate/pool/provider by id
EvidenceLedger append-only, per-mission seq via locked ledger head, hash-chained domain service over events; appends serialized (design/02 §2)
AgentDefinition status draft→evaluating→active→frozen; active requires gated PromptVersion; holds NO spend authority — reads active spend_grants (ADR-003 rev 2) user-created agents
PromptVersion immutable rows (trigger-enforced); content_hash + boilerplate_hash + eval report DB is canonical; Langfuse mirrors for metrics UI

Cross-aggregate consistency via domain events → outbox → in-process handlers. Allowed exception: reserve-funds + create-proposal in one UoW transaction (financial atomicity).

4. Patterns map

Pattern Where Why
Ports & Adapters application/ports ↔ infrastructure Rain issuer swap = 1 file; facilitator ladder = env var; merchant simulator = explicit mock-only port
Specification domain/policy rules return machine-readable results; the demo displays which rule fired
State machine domain/payment lowercase enum + transitions dict + transition() raising InvalidTransition; rejects anything the graph shouldn't request
Strategy rail routing X402Rail / RainCardRail behind PaymentRailPort
Outbox infrastructure/db events persist in the same tx; NOTIFY on commit (ADR-001)
Saga (orchestration) LangGraph graph compensations: release_reservation, void_card, freeze_agent; progress = checkpoint + proposal states
Repository + UoW infrastructure/db explicit to_domain()/from_domain() converters

5. PaymentProposal state machine (canonical lowercase everywhere)

proposed → policy_allowed | policy_blocked | policy_escalated
policy_allowed → reserved → executing → settled | failed
policy_escalated → policy_allowed | policy_blocked   (human decision)
failed → compensated

Two block classes carried on the verdict, not extra states: validation_reject (bad/ambiguous input; no freeze) vs security_block (spend-rule violation; freezes the agent's mission-scoped grant).

6. LangGraph topology (rev 5 — adds atomic authorize+reserve and settlement

reconciliation branching from the production review; research 03 primitives)

Authority reconciliation (2026-08-08). Where an older topology in this section disagreed with the locked demo chronology, docs/design/00-product-spec.md §4 is the higher authority. The graph therefore performs credential retirement, provider-response parsing/flagging, the labelled attack simulation and policy block, and only then the independent outcome verification (product-spec beats 5→6→7).

flowchart TD
    S((START)) --> CM[compile_mandate · LLM strict schema]
    CM -->|valid| D[discover · registry + scout-normalize LLM]
    CM -->|validation_reject| R[reconcile]
    D --> EP[decide_evidence_plan · deterministic]
    EP -->|no auditions planned| R
    EP -->|Send per PLANNED audition<br/>x402 AND trial| AUD[audition subgraph<br/>x402: check_policy → pay → invoke production → measure<br/>trial: invoke trial endpoint → measure<br/>→ typed AuditionResult, never uncaught]
    AUD --> Q[assess_evidence_quorum]
    Q -->|insufficient / all failed| R
    Q -->|sufficient incl. partial| SEL[select · deterministic scorer]
    SEL -->|no feasible provider<br/>all below floor| R
    SEL -->|winner| PROP[propose_payment]
    PROP --> AUTH{authorize_and_reserve<br/>ONE UoW: lock rows in order →<br/>re-check policy → verdict →<br/>reservation journal → reserved}
    AUTH -->|allow, funds reserved| PS[purchase saga subgraph<br/>prepare_rail → execute_payment<br/>→ confirm_settlement]
    AUTH -->|block, nothing reserved| PB[policy_block · freeze if security_block]
    AUTH -->|escalate, nothing reserved| ESC[escalate · interrupt]
    ESC -->|approved + typed payload;<br/>full authorize_and_reserve re-check succeeds| PS
    ESC -->|approval stale / re-escalate| ESC
    ESC -->|full re-check blocks| PB
    ESC -->|rejected| PB
    PS -->|settled| RET[retire_credential]
    PS -->|failed / unknown / timeout| RREC{reconcile_rail_status<br/>query rail by idempotency key}
    RREC -->|remote SETTLED| CONFL[confirm_settlement<br/>reserved→spent journal FIRST] --> RET
    RREC -->|remote definitely failed<br/>or never submitted| COMP[compensate<br/>release reservation, void card] --> R
    RREC -->|still pending / unknown| HOLD[hold: keep reservation,<br/>freeze credential, attempt →<br/>settlement_unknown, manual review] --> R
    RREC -->|amount / merchant mismatch| SEC[security incident:<br/>freeze authority, NEVER auto-release] --> R
    RET --> PARSE[parse_provider_response<br/>load bounded artifact by ID]
    PARSE --> FLAG[record_injection_flag<br/>ledger write BEFORE the gate]
    FLAG --> GATE{demo_attack_gate}
    GATE -->|skip| V[verify_outcome · holdout set]
    GATE -->|simulate| SUB[substitute buyer output<br/>fixed schema-valid malicious] --> PROP2[propose_payment · FRESH id] --> AUTH
    PB -->|post-purchase simulated attack| V
    PB -->|ordinary pre-purchase block| R
    V -->|pass| R
    V -->|fail| REM[remediate_outcome<br/>accept_degraded OR reconcile_failed<br/>NEVER back to purchase] --> R
    R --> E((END))

Corrections encoded here (graph review 2026-08-04):

  1. Auditions fan out over the evidence PLAN, not over x402 providers — the trial audition (LegacyBatch) now has an explicit evidence-producing path; decide_evidence_plan is a real node (spec beat 2), with a typed "no auditions" exit. Reuse/quick/full-RFP triage stays roadmap.
  2. Escalation approval re-enters the authorize_and_reserve use case (full policy re-check — budget, grants, price, card scope may have changed during the interrupt). The resume payload is typed and validated: {mission_id, proposal_id, proposal_version, merchant_amount_digest, approver, decided_at} — a bare {"approved": true} is rejected. Approval converts the escalate trigger, it never bypasses the gate. The resumed interrupt node passes that payload directly to the use case and checkpoints only the typed verdict flag; the approval object is never copied into MissionState.
  3. Post-purchase verification runs after response parsing and the attack stages, and failure routes to remediate_outcome (accept-degraded or reconcile-failed for this build; refund/reselect = roadmap). It can NEVER re-enter the purchase saga — pre-purchase approval and post-purchase remediation are different business processes.
  4. Purchase is a saga subgraph, one thin node per externally-significant boundary (prepare_rail [issue card / prep x402] → execute_payment → confirm_settlement — reservation already happened atomically inside authorize_and_reserve, correction 7), each with its own checkpoint (durability="sync") and a typed rail_operations idempotency record — so a crash between "card issued", "auth submitted", "settled", and "DB updated" resumes with minimal uncertainty. Failure/unknown → reconcile_rail_status, which BRANCHES per correction 8 (settled → confirm; proven-failed → compensate; unknown → hold + reconciliation case; mismatch → security incident) — compensation is never unconditional.
  5. Typed audition results + quorum node: expected provider failures (timeout before pay, paid-but-no-response, invalid output) become AuditionResult = Success(evidence_id) | Failure(provider_id, stage, retryable, payment_consumed) — one dead provider cannot prevent selection when enough valid evidence exists; all-failed and none-above-floor exit via reconcile with typed terminal reasons.
  6. The compromise switch is a reachable node after credential retirement and before independent verification: record_injection_flag (ledger side effect BEFORE the gate — interrupt nodes re-run on resume) → demo_attack_gate, which reads the mission's attack_simulation input (set by the demo driver; can also be an interrupt() for live judge control). No update_state(as_node=...) tricks.
  7. Authorization and reservation are ONE atomic use case (authorize_and_reserve, production review 2026-08-04): a single UoW locks proposal + grant + account rows in deterministic order, validates proposal version + digest, re-evaluates every mutable policy rule, records the verdict, writes the reservation journal, transitions the proposal to reserved, appends ledger events, commits. No funds-check-then-reserve TOCTOU window across checkpoints; database locks never span LangGraph nodes. The graph node stays thin. Escalation approval re-enters this same atomic use case.
  8. Unknown settlements never auto-compensate: reconcile_rail_status queries the rail by idempotency key and BRANCHES — remote settled → confirm (reserved→spent journal, retire credential, verify: the money moved, so the mission proceeds); proven failure/never-submitted → compensate; still pending/unknown → reservation HELD, credential frozen, attempt marked settlement_unknown, manual review (typed terminal, never silent release); amount/merchant mismatch → security incident, authority frozen, nothing auto-released. Attempt states submitted / reconciling / settlement_unknown are persistent (design/02); a proposal stays executing until its attempt is terminal.

Mission state (keyed, versioned — IDs only, never blobs)

class MissionState(TypedDict, total=False):
    schema_version: int
    graph_version: str
    mission_id: str
    evidence_plan_decision: str | None               # typed routing flag
    audition_offer_ids: list[str]
    audition_ids_by_provider: dict[str, str]        # keyed, not order-dependent
    failed_audition_provider_ids: list[str]
    selected_offer_id: str | None
    proposal_ids: list[str]                          # full history, legitimate + malicious
    active_proposal_id: str | None
    settled_proposal_id: str | None
    provider_response_artifact_id: str | None
    verification_id: str | None
    attack_simulation: bool                         # typed routing flag
    terminal_reason: str | None                     # typed routing flag

Rules: state stores database IDs plus small typed routing flags, not offers, merchant or endpoint facts, receipts, provider bodies, approvals, prompts, events, or secrets. Use cases reload immutable authorized facts within their UoW; the bounded provider response is an append-only provider_response_artifacts row and only its ID is checkpointed. The ledger is never mirrored into checkpoints; parallel results land in provider-keyed dicts (parallel branch ordering is not guaranteed); providers are explicitly sorted before scoring and tie-breaking; legitimate and malicious proposal histories stay distinct.

Concurrency, versioning, timeouts

  • Single-flight per mission: a Postgres advisory lock on mission_id around every graph invocation/resume + idempotent command IDs — double-clicks, concurrent resumes, and repeated attack-switch requests cannot run one thread twice.
  • Version pinning: graph_version + schema_version stored in state; resume with a mismatched deployed graph fails loudly (no silent topology migration — LangGraph resumes on the latest deployed graph otherwise). Migrations are out of scope.
  • Timeouts — two node classes (round-10 fix of a self-contradiction): LLM, read-only, and discovery nodes get bounded retries (retry_on incl. NodeTimeoutError, which is NOT builtin TimeoutError). Payment and credential nodes get NO blind node retry — LangGraph retries NodeTimeoutError by default, so these nodes set an explicit no-retry policy; a timeout means UNKNOWN and routes immediately to reconcile_rail_status. A later resubmission is allowed only with the same durable operation id and the exact persisted payload, or after the rail proves the first request never executed. No graph-wide retry default on payment paths.

Other non-negotiables (unchanged from research 03): TypedDict + reducers where branches append; AsyncPostgresSaver in lifespan, thread_id = mission_id; durability="sync"; interrupts never inside broad try/except; SSE is ledger-only; pins incl. langgraph-checkpoint>=4.1,<5; no secrets in state (blob-scan test).

The two attack paths remain distinct by design (spec beat 6): the normal path proves the prompt defense (flags → no proposal → reconcile); the gate's simulate branch is the only source of a malicious proposal and flows through the standard propose/authorize use cases — zero demo-only code beyond the output substitution.

Role → node wiring (resolves review A6): mandate compiler → compile_mandate; provider scout → the normalize step inside discover (registry listing itself is deterministic); buyer → propose_payment; outcome verifier → structuring step inside verify; narrator → out-of-graph read-only service consuming the ledger for the UI (no node, no authority).

Non-negotiables (research 03): - State = TypedDict + model_dump()ed values; reducers on benchmarks/events. - AsyncPostgresSaver open for app lifetime; setup() at boot; thread_id = mission_id. - durability="sync"; payment nodes follow claim-then-act, never check-then-act (round-10 review): UoW 1 claims the durable rail_operations row via INSERT … ON CONFLICT (persisting the exact request — for x402, the encrypted signed payload + nonce, status prepared) and COMMITS; only then the external call happens (never inside a DB transaction); UoW 2 finalizes with compare-and-set. Ambiguous result → reconcile_rail_status, never a guess. Nodes replay from the top on resume and land on the claimed row. - Idempotency keys derive from durable business IDs, never node names: audition payment → audition_id; purchase → payment_attempt_id; card issue/freeze/retire → their rail_operation_id; API commands → client command id. Graph topology changes can never change financial identity. - Escalation via interrupt(); never in broad try/except; resume Command(resume=…). - Retries/timeouts via RetryPolicy/TimeoutPolicy (async nodes); compensation and retry-exhaustion paths are tested with a fake clock (plan Task 4.3). - SSE is ledger-only (review A8): nodes emit events exclusively by appending ledger rows through use cases (transactional, ordered, hash-chained); the SSE endpoint streams committed rows via LISTEN/NOTIFY with Last-Event-ID replay. astream modes are a dev/debug surface only, never the public event feed. - Pins: langgraph~=1.2.10, langgraph-checkpoint-postgres~=3.1.1, explicit langgraph-checkpoint>=4.1,<5 constraint + lockfile audit in CI (CVE-2025-64439), langchain-openai~=1.4.1. - No secrets in graph state; enforced by an allowlisted state schema + a Postgres integration test scanning stored checkpoint blobs for forbidden fields/sentinels.

7. Payment rails

x402: x402[fastapi,httpx,evm]==2.17.0; scheme exact, network eip155:10143, USDC 0x534b2f3A21130d7a60830c2Df862319e593943A3; facilitator ladder (env var): x402-facilitator.molandak.org → self-hosted x402Facilitator (compose service with its own funded signer + MON canary) → mock. All three modes contract-tested. Settlement verified via RPC; tx hash persisted on the attempt and deep-linked to the explorer in the UI. payment-identifier dedup in our resource servers. Never re-sign on retry.

Rain: RainCardIssuer port — normative contract = research/01 §4 (port, field names, cents, enums) + §5 (mock decision order, decline reasons, transaction store, webhook events, 5-min timestamp window, seeded uuid5 ids, injectable clock) in full; contract tests use those sections as the oracle, and the future RainSandboxCardIssuer must pass the same suite. Idempotency keys are mandatory on issue AND freeze/retire; webhook receiver dedupes by event id. The webhook HMAC scheme is mock-only (research/01 marks it INFERRED); the sandbox adapter maps whatever scheme organizers provide. simulate_authorization lives behind a separate mock-only MerchantAuthSimulatorPort used by the LegacyBatch merchant mock — so the "one-file swap" claim is scoped to issuer operations; the merchant checkout remains an explicit mock either way (review B23).

8. Agent Registry (user-created agents — ADR-003 rev 2)

Flow: user input (untrusted) → static validation → master prompt generates ROLE/CONTEXT/INSTRUCTIONS/EXAMPLES (each example individually re-validated by code against the role schema) → code injects security block + no-spend clause + output contract → candidate PromptVersion (immutable row) → bounded background evaluation job (agent status evaluating; progress events to the timeline; hard budget ≤60 s and ≤25 LLM calls against the cached pinned corpus; timeout → draft with report) → activation sets gated_at only if every gate metric passes at 100% (design/05 §4) → agent active. Runtime always executes the exact pinned prompt_versions.id; DB content is canonical; Langfuse labels are promotion metadata/metrics UI only (review A24). Demo fallback: one pre-generated, pre-gated agent ready to activate instantly. Authority: agents hold zero spend capability; grants are separate human-approved spend_grants rows; mission-scoped freeze revokes the mission grant only (design/02 §3).

9. Interfaces

  • REST (OpenAPI → generated TS client): missions (create/run/approve/reject), agents (CRUD/generate/activate/freeze), providers, ledger query, prompt versions + eval reports.
  • GET /missions/{id}/events — SSE from the ledger, Last-Event-ID resume.
  • POST /webhooks/rain — mock-HMAC verified, event-id deduped.
  • No gRPC (ADR-002). No broker (ADR-001).

10. UI (single mission-execution screen — the evidence graph is the hero)

The name promises a graph; the primary visual IS the animated evidence graph (rendered from the same ledger events the SSE feed delivers — one data source, two projections):

mandate ──► discovery ──► auditions ──► tournament ──► policy ──► Rain credential
                │  ├ ExactFlow  → x402 tx → evidence          │
                │  ├ BudgetFlow → x402 tx → evidence (floor ✕)│
                │  └ LegacyBatch→ trial   → evidence          ▼
                │                                     purchase + receipt ──► verified outcome
                └── [red branch] compromised proposal ──► POLICY GATE ✕ (terminates here)

Nodes light up as ledger events arrive; the malicious proposal appears as a red branch that dies at the policy gate with rule-by-rule results. Secondary panels: provider tournament table (quality, evidence class, price, risk-adjusted cost — every penalty term numeric, raw sample outputs inspectable), pools bar, timeline list, outcome card. A "Proof" drawer holds the deep surfaces (Langfuse trace links, explorer links, JSON logs, journal entries, hash-chain verify) — available live and for Q&A, never the main show. The "simulate compromised model" demo switch is an on-screen, clearly-labeled control that forces a crafted proposal through the real gate (design/04 Layer 7). Agent-creation modal: describe → generate → eval progress → scores → activate.