Skip to content

ProcureGraph — Testing Strategy (100% coverage, honestly)

Sources: research/06-architecture-data.md §d, research/03-langgraph.md §e. 100% is achievable because the architecture keeps domain/application pure and small. Budget ~40% of build time for tests, written with the code. Every exclude_also pragma is a reviewed decision, not an escape hatch.

1. Backend gates

pytest with --cov=procuregraph --cov-branch --cov-fail-under=100 --cov-config=pyproject.toml (explicit config path — pytest-cov discovery is flaky). branch = true; omit only alembic/; exclude_also limited to TYPE_CHECKING, NotImplementedError, @overload, ....

Layer Approach
domain/ pure unit tests, zero doubles: state-machine transitions, Money arithmetic, every specification rule's ALLOW/BLOCK/ESCALATE branch, ledger hash chain
application/ hand-rolled fakes (FakeUoW, dict repos, fake rails/LLM/clock) — Cosmic-Python fakes-over-mocks; tests read as user stories, no IO
infrastructure/db/ real Postgres via testcontainers: repos, UoW commit/rollback, outbox write, balanced-journal trigger, append-only trigger, partial unique indexes — DDL guardrails get real-DB tests
x402 adapter respx/httpx mock transport unit tests + one integration test vs local mock providers; real Monad testnet only in the demo path, never CI
Rain adapter contract tests against the RainCardIssuer port (mock passes them today; the sandbox adapter must pass the same suite to swap in)
LLM adapter recorded deterministic fixtures; schema re-validation; refusal-path test; no live OpenAI in tests

2. LangGraph coverage (the part plain coverage can't see)

  • Nodes as pure functions with injected ports → direct unit tests.
  • Every conditional edge is a named router function → parametrized over its full output range (one case per edge target).
  • Topology lock test: compiled graph's edge set asserted explicitly so refactors can't silently drop an edge.
  • Path tests through the compiled graph with InMemorySaver + stub ports: happy, block, escalate-approve, escalate-reject, injection-block; assert visited-node sequences from updates streaming.
  • The money test: payments port raises after "payment sent" on first call → re-invoke same thread_id → exactly one ledger row (replay idempotency).
  • One integration test against dockerized Postgres for AsyncPostgresSaver.setup() + resume.

3. Prompt evals as tests — two tiers (review B8)

  • Tier 1, CI (no live OpenAI): adapter tests over recorded fixtures — schema re-validation, refusal path, injection corpus replayed against the policy engine. Gates merges. Both canonical attack payloads — $97 over-budget and $5 within-budget GiftCardHub — are pinned regression cases forever (design/04 Layer 1).
  • Tier 2, credentialed candidate eval (make eval): runs the actual candidate prompt against the pinned corpus on live gpt-4o-mini (bounded: ≤25 calls, ≤60 s), stores raw outputs + system_fingerprint + scores into prompt_versions.eval_report, and is the ONLY path that sets gated_at. Thresholds per design/05 §4 (all hard gates at 100%, determinism 5/5). Replayed fixtures can never activate a new prompt.

3b. Security-effects tests (review B5)

Compiled-graph attack-path test asserts EFFECTS, not just the verdict: zero calls on both rail ports after the injection, no new card row, no new signature, mission-scoped spend_grants.revoked_at set, original card canceled, machine-readable reasons in the ledger. Plus the checkpoint-secret scan (review B10): integration test reads stored checkpoint blobs from Postgres and fails on forbidden fields/secret sentinels (PAN, CVC, private keys, API keys, signed payloads).

4. Frontend

  • Vitest (jsdom + Testing Library): 100% branch thresholds on ALL authored frontend code (components included — render + interaction tests); exclusions enumerated in config: generated OpenAPI client, vite config, type-only files (product spec §9; review B6).
  • Playwright: exactly three demo-critical journeys — happy path (incl. explorer link for the on-chain settlement — review B3), BLOCK moment renders rule reasons, timeline streams over SSE. No coverage gate on e2e. Never merge Vitest+Playwright coverage.

5. CI / local

GitHub Actions: (1) backend — ruff + mypy + import-linter architecture contracts (domain imports nothing third-party; dependency arrows inward only; graph nodes contain no business logic — review B19) + pytest unit + testcontainers integration, fail_under=100; (2) frontend — eslint + tsc --noEmit + vitest coverage; (3) Playwright, non-blocking on coverage. Merges blocked on all three. uv.lock committed; CI audits the resolved langgraph-checkpoint version against the explicit >=4.1,<5 constraint and fails on known-critical advisories (review B20). Pre-commit: ruff, ruff-format, mypy incremental, unit subset only. Daily canary this week: facilitator /supported + buyer balance + one $0.01 real settlement early in the week (review B22).

6. "Tested" for the demo means

  1. The gates above green. 2. One scripted end-to-end run passing deterministically with seeded data. 3. The recorded backup video of a clean run against the real facilitator.