Developer guide on Eventboat

Architecture & package map

Mon, 01 Jan 0001 00:00:00 +0000

Eventboat is a single Go binary that routes events: messages arrive from sources (Kafka, HTTP, cron, file, SQL), flow through an explicit DAG of transforms and filtered edges, and land at sinks — at-least-once, verifiable before anything runs, and replayable after a crash. Pipelines are declared in YAML, statically verified against plugin JSON Schemas and topology rules, and then executed by an engine whose entire reliability model is pinned by eight dedicated invariant tests. Predicates are CEL, transforms are Starlark (or WASM), so there is no custom language to learn.

Engine internals

Mon, 01 Jan 0001 00:00:00 +0000

The engine (internal/engine) executes one compiled pipeline against one store: source admission through a durable spool, in-memory DAG execution with commit tracking, per-edge delivery retries, dead lettering, checkpointing and backpressure. The package is five files plus tests: engine.go (lifecycle, persistence), admission.go (the inbound path), nodes.go (transform/sink workers, dead letters), commit.go (the commit frontier), plus sourcecommit.go (the per-source watermark committers). Every reliability property is phrased as one of the eight invariants (see Architecture), each with a dedicated test in internal/engine/invariants_test.go.

Plugin system & registry

Mon, 01 Jan 0001 00:00:00 +0000

Every source, sink, codec and transform in eventboat is a plugin registered in a registry.Registry (internal/registry/registry.go). Registration carries a JSON Schema; configuration blocks are validated strictly against it (unknown fields are errors), which is the mechanism that keeps agents from inventing plugins or fields that do not exist (redesign-v3.md §5.6). The registry is a leaf package: it imports nothing internal, so any Go program — including your own build of eventboat — can register plugins into it.

Configuration & diagnostics

Mon, 01 Jan 0001 00:00:00 +0000

A pipeline is one YAML file loaded by internal/config and compiled by internal/ir. Loading is deliberately strict: unknown keys, unknown plugins and un-compilable expressions are errors, not warnings — the config is a contract with agents as much as with humans. Every finding surfaces as a config.Diagnostic with a stable code, severity, file/line, message and hint (internal/config/config.go); --json output, the LSP, the MCP verify tool and the admin UI all render the same structs.

Expressions & scripting sandbox

Mon, 01 Jan 0001 00:00:00 +0000

Eventboat hosts three languages, each with one job: CEL is the default edge predicate dialect, CESQL is an opt-in CloudEvents-flavored alternative for edges, and Starlark is the scripting language of the built-in script transform. All three are sandboxed, deterministic and side-effect free — a property explain relies on to dry-run pipelines, and the reason replay produces the same answers as production. The WASM tier covers compute-heavy transforms that outgrow Starlark.

Observability & operations

Mon, 01 Jan 0001 00:00:00 +0000

Eventboat exposes three operations surfaces that share one implementation: the MCP tools, the Admin REST endpoints (plus SSE and an embedded read-only UI), and the --json CLI outputs are all thin shells over internal/ops.Service — REST, MCP and CLI emit the same Go structs and JSON shapes. Telemetry is OpenTelemetry with dual export (Prometheus pull + OTLP/HTTP push); the metric set is fixed and enumerated in internal/obs/obs.go.

Metrics

One MeterProvider, two readers (internal/obs/obs.go): Prometheus exposition at /metrics and, when telemetry.otlp_endpoint is set, an OTLP/HTTP push reader (10s interval). Fully disabled telemetry costs nothing (noop providers; every Obs helper is nil-receiver safe). Every instrument carries the eventboat_ prefix (the static set below, plus the per-source counters created lazily on first use):

Testing guide

Mon, 01 Jan 0001 00:00:00 +0000

Eventboat’s testing philosophy: every reliability property is a named, retrievable test; every externally visible surface (help screens, plugin schemas) is pinned by a golden file; and anything that claims to work against the real world (Kafka, long runs) is proven against the real thing in CI. The full suite is go test ./...; CI runs it with -race.

The test taxonomy

Layer	Where	What it proves
Unit tests	beside the code (`*_test.go` per package)	package-level behavior: loader rules, schema generation, host evaluation, store semantics
Engine invariants	`internal/engine/invariants_test.go`	the eight §6.2 reliability invariants, one test each
Engine behavior	`internal/engine/*_test.go`	delivery, fan-out, split, transform plugins, wasm, CESQL, admission pooling
Persistence/recovery	`internal/engine/commit_persist_test.go`, `retention_test.go`, `internal/store`	flush ordering, retention bounds, crash replay
Golden tests	`internal/cli/testdata/help/`, `internal/registry/builtin/testdata/schemas/`	CLI help screens; generated plugin JSON Schemas
Contract tests	`internal/testrun` + `examples/*/tests/`	pipeline behavior through the real engine via suite YAML
Integration	`internal/inttests/kafka`, `internal/inttests/soak`	real broker roundtrips; long-run stability (env-gated, need Docker)
Benchmarks	`internal/engine`, `internal/lang/*`, `internal/wasmhost`	performance gates and the Starlark-vs-WASM comparison
Acceptance	`internal/cli/examples_test.go`, `acceptance_test.go` (root), `internal/rpcplugin/acceptance_test.go`, `internal/cli/agent_loop_test.go`	every example verifies and passes its suites; the custom-build binary links `pkg/plugin` plugins and runs them; the third-party gRPC plugin builds and runs; the full agent loop over MCP stdio

The eight engine invariants

internal/engine/invariants_test.go pins redesign-v3.md §6.2 — these names are load-bearing; do not rename them:

Building & release

Mon, 01 Jan 0001 00:00:00 +0000

Eventboat builds with the standard Go toolchain and no code generation step in the normal path: go build ./... is the whole story for contributors. This page covers the local build, the container image, the CI workflows, and how releases are cut.

Prerequisites

Go 1.25 (matches go.mod; the CI jobs pin the same version).
Docker — only for the env-gated integration suites (the Kafka testcontainers test) and for building/running the container image.
Nothing else: all drivers are pure Go (SQLite via modernc.org/sqlite, Postgres via pgx, Kafka via kafka-go), so CGO is never required.

Local build and run

go build -o eventboat ./cmd/eventboat # the single binary
./eventboat # bare invocation prints help, exit 0
./eventboat verify --config examples/linear/pipeline.yaml
./eventboat test examples
./eventboat run --config examples/linear/pipeline.yaml
./eventboat run --config my.yaml --ephemeral # in-memory store, no ./data

The example job pipeline’s sqlite source database regenerates with go run ./examples/job-sync/seed. The WASM test guest is committed as a built artifact (internal/wasmhost/testdata/aggregate.wasm), so plain go test ./... needs no wasm toolchain; rebuild it with GOOS=wasip1 GOARCH=wasm go build -buildmode=c-shared -o ../aggregate.wasm . from internal/wasmhost/testdata/guest (CI does exactly this).

Contributing

Mon, 01 Jan 0001 00:00:00 +0000

This repo has a strong house style — it is what keeps a codebase with this many guarantees reviewable. Read this page before your first PR; most of it is enforceable by CI, but the commit message convention and the comment voice are reviewed by humans.

Dev workflow

Branch from main; keep one logical change per commit — the git log reads as a series of self-contained rationales, and bisectability depends on it.
Run the fast loop before pushing: go build ./..., go test -race ./... (CI runs the race build on every push), and bash scripts/bench-gate.sh if you touched a hot path.
User-visible changes get a CHANGELOG.md Unreleased entry at commit time, not at release time. Entries carry a Changed/Added/Fixed/ Removed heading and the same paragraph voice as the commit — the Settled→Commit rename entry, for example, includes the complete old→new mapping because external consumers needed it to migrate.

The commit message convention

This repo’s commits are single, detailed, paragraph-led messages written in the imperative that explain WHAT and WHY, with concrete file and identifier references. No one-line subjects, no bullet laundry lists (a short list is acceptable inside the paragraph when enumerating parallel changes). Read a few from git log — e.g. the Rename Settled → Commit everywhere commit or the Promote transforms to registered plugins commit — they record the motivation, the ruling that decided contested points, the migration/compat story, and the verification done (which tests were run). A good test: could a reviewer reconstruct the change from the message alone in a year?