Testing guide

Eventboat’s testing philosophy: every reliability property is a named, retrievable test; every externally visible surface (help screens, plugin schemas) is pinned by a golden file; and anything that claims to work against the real world (Kafka, long runs) is proven against the real thing in CI. The full suite is go test ./...; CI runs it with -race.

The test taxonomy

The eight engine invariants

internal/engine/invariants_test.go pins redesign-v3.md §6.2 — these names are load-bearing; do not rename them:

• TestInvariant_SpoolBeforeVisible — a failing spool append refuses the message; it never reaches a sink.
• TestInvariant_CheckpointAdvancesOnlyAfterCommit — the checkpoint trails committed messages.
• TestInvariant_Kill9ReplayReplaysAllUncommitted — crash recovery replays a superset of the uncommitted set.
• TestInvariant_DeadLetterWriteFailureBlocksCommit — a failing dead-letter write never drops the message.
• TestInvariant_RequiredFalseFailureDoesNotBlockSiblingBranches — optional edge failure isolates to its branch.
• TestInvariant_RedeliveryKeepsMessageIdStable — duplicate delivery keeps meta.message_id, so idempotent sinks can dedup.
• TestInvariant_CursorWatermarkNeverExceedsCommitted — pull-source watermarks only advance over committed messages.
• TestInvariant_BranchIsolation — fan-out siblings share the underlying msg.Decoded/msg.Meta maps, so a transform mutating them in place (the script binding’s lazy remove(), pre-fix) must never be able to race a sibling branch’s encode: the copy-on-write binding isolates the branches, and the process must survive a wide-map fan-out under -race.

The -race policy: the engine and store packages must stay race-clean — CI runs go test -race ./... on every push. If you add concurrency to either package, run the race detector locally first; a data race here is a correctness bug in the durability model, not a hygiene issue.

Golden tests

Three golden surfaces, each with its own regenerate flag:

• CLI help snapshots: internal/cli/testdata/help/*.txt pin every verb’s help screen. After changing a flag or usage string: go test ./internal/cli -update and commit the diff.
• Plugin schema goldens: internal/registry/builtin/testdata/schemas/*.json pin the JSON Schema generated from every builtin config struct. After changing a config struct: go test ./internal/registry/builtin -update-schemas. The test failure text tells you the flag.
• Protobuf descriptor golden: internal/registry/builtin/testdata/example.descr, regenerated with -update-descr (the protobuf codec test’s fixture; no protoc involved).

A golden diff in a PR must be reviewable line by line — that is the point.

Integration tests (need Docker)

Both suites skip silently without their env var; CI sets it:

# real Kafka via testcontainers (KRaft, confluent-local):
# roundtrip through the engine, dead-letter path, consumer-group rebalance
EVENTBOAT_KAFKA_TEST=1 go test ./internal/inttests/kafka/ -v -timeout 8m

# long-run stability: mixed load + injected faults; asserts at-least-once,
# full checkpoint advance and no goroutine leaks
EVENTBOAT_SOAK_TEST=1 EVENTBOAT_SOAK_DURATION=2m go test ./internal/inttests/soak/

EVENTBOAT_SOAK_DURATION bounds the soak run (CI default 25m; nightly plus manual dispatch via .github/workflows/soak.yml). The wasm tests optionally honor EVENTBOAT_WASM_GUEST — CI rebuilds the guest from source and points the env at the fresh artifact; locally the committed .wasm under internal/wasmhost/testdata/ is used, so plain go test ./... needs no wasm toolchain.

Benchmarks

bash scripts/bench-gate.sh   # what CI enforces

scripts/bench-gate.sh is a loose order-of-magnitude gate, not a noise gate — limits sit ~5–15x above the reference dev machine (i5-14600KF) to absorb shared-runner variance:

It covers the per-message hot paths (predicate eval, script exec, commit throughput against the memory store) and deliberately does not cover WASM (runner variance too high — BenchmarkHeavyTransform in internal/wasmhost stays informational) or SQLite-backed throughput. What a gate failure means: an algorithmic regression landed; profile before optimizing.

testkit and testrun

internal/testkit provides the deterministic primitives engine and contract tests build on:

• ManualSource — a source whose Emit blocks until the engine accepted the message (spooled + dispatched), making tests deterministic; Commit records the cursor watermark like a real pull source would.
• CaptureSink/Recorder — capture delivered messages; DiscardSink; FlakySink — fault-inject sink writes by attempt count.
• StoreWrapper — hook AppendSpool/WriteDeadLetter/SetCheckpoint to inject store faults.
• FixedClock/CounterID — frozen clock and deterministic IDs.
• RegisterFakeTransform/TransformFunc — register a function as a transform plugin with an accept-anything schema (see Plugins).

internal/testrun is the contract-test runner: it builds the pipeline IR, runs each case against a real engine with an ephemeral memory store, fixed clock, capture-wrapped sinks (the plugin sink never touches the real world), injects at the named node, waits for commit, and evaluates expectations (in-order subset matching on payload.*/meta.* dotted paths, DLQ count/reason). Job pipelines run with sources disabled so injections at the pull node are deterministic.

The suite format

The project convention is <root>/pipeline.yaml + <root>/tests/<suite>.yaml (+ tests/fixtures/); the suite references its pipeline as pipeline: ../pipeline.yaml (path resolution is contained within the suite root — an escaping path is an error):

suite: order-branching
pipeline: ../pipeline.yaml
cases:
  - name: eu-order-routed-to-eu-only
    inject: { at: ingest, messages: [fixtures/eu-order.json] }
    expect:
      capture:
        at: eu-out
        messages:
          - payload.total: 12000     # subset match
            meta.tier: vip
  - name: malformed-json-to-dlq
    inject: { at: ingest, raw: "{not json" }
    expect:
      dlq: { count: 1, reason_contains: decode }

eventboat test <path> consumes it: a file recurses (.yaml files without a top-level suite: key are skipped and counted; unreadable/unparseable YAML is a hard error, ruling D3). Exit codes: failing cases → 1; a directory with no valid suites but broken files → 2 (ruling D4).

Practical recipes

Run the fast loop (seconds, no Docker):

go build ./... && go test ./... 

Run everything locally before a PR:

go test -race ./...                          # full suite, race on (engine+store policy)
bash scripts/bench-gate.sh                   # perf gate
EVENTBOAT_KAFKA_TEST=1 go test ./internal/inttests/kafka/   # needs Docker

Add a regression test for a delivery bug: reproduce with the smallest pipeline YAML (see invYAML in invariants_test.go for the minimal shape), drive it with testkit.ManualSource + CaptureSink + StoreWrapper fault hooks, assert on Recorder.Captured() or the store’s dead letters. If it touches commit/checkpoint ordering, check whether one of the eight invariants already covers the class — extend that test’s scenario instead of asserting something weaker.

Add a contract test for a user-visible pipeline behavior: write the suite under examples/<pipeline>/tests/, fixtures under tests/fixtures/, and rely on the CI examples gate (internal/cli/examples_test.go) to keep it green.