Configuration & diagnostics

A pipeline is one YAML file loaded by internal/config and compiled by internal/ir. Loading is deliberately strict: unknown keys, unknown plugins and un-compilable expressions are errors, not warnings — the config is a contract with agents as much as with humans. Every finding surfaces as a config.Diagnostic with a stable code, severity, file/line, message and hint (internal/config/config.go); --json output, the LSP, the MCP verify tool and the admin UI all render the same structs.

The load pipeline

LoadBytes(file, data) (internal/config/loader.go) runs four passes:

1. Parse + env substitution. The document parses into a yaml.Node; every string scalar goes through ${VAR} / ${?VAR} substitution (unset ${VAR} is a cfg_env_unset error; unset ${?VAR} omits the key; substituted values are never re-scanned — single-pass, ruling D1).
2. Structural decode into generic maps.
3. Constants substitution. ${constants.x} expands (unknown constant = error); ${parameters.x} passes through untouched only in job pipelines (run.mode: job) — anywhere else it is a cfg_scope_unknown error. The optional marker ? is only legal for plain environment variables.
4. Structural validation with whitelists: top-level keys, metadata.name validation (metadata is a strictly checked mapping: unknown keys and a non-mapping value are errors), the limits/run/parameters/hooks/telemetry/codecs/dlq sections, the three node sections with their framework-field whitelists, and finally manifest reads for every external (grpc:) node. The whitelists, the top-level keys, the edge attributes, the reserved plugin names and the node-level default constants all come from the single leaf package internal/schema (candidate 06) — config, registry and the LSP read one vocabulary, so a name like grpc or version cannot register as a plugin that config would parse as a framework field.

metadata.name is a conservative identifier (cfg_name_invalid): 1–64 characters of [a-zA-Z0-9._-], starting alphanumeric, no .. (path traversal), and no Windows reserved device name (CON, PRN, AUX, NUL, COM1-9, LPT1-9) — the name becomes the deployed file name and the store key.

Pipeline.Order is the YAML document order of node declarations (candidate 06), preserved from the node stream: jobs binds cursor to the first source in that order, explain picks its default entry node from it and the diagnostics iterate it, so a shuffled document still behaves like the one a human reads.

The three-section topology

sources, transforms, sinks — each node is name: {plugin block, ...framework fields}. depends_on edges join them. Sources and sinks are required; the transforms section is optional. Exact node-level whitelists (internal/schema, read by internal/config/sections.go) —

— everything else at node level must be exactly one plugin key. workers is transform-only: sinks.workers was accepted and ignored before and is now rejected with cfg_sink_workers (sink concurrency is engine-owned; implement or refuse). Edge attributes on a depends_on element: when, route, buffer, delivery, required. when accepts a string (CEL) or {lang: cel|cesql, expr: "..."}; route is sugar compiled to meta.route == "<name>" and is mutually exclusive with when. edge_defaults accepts only delivery, required and buffer — a global default predicate is a footgun and a default route is meaningless, so when/route there are cfg_edge_defaults_field errors (candidate 06). Pipeline-level defaults (decoder/encoder json, delivery retries: 3 / backoff: exponential, required: true, buffer max_events: 128, workers 1) are materialized into the typed config at load: downstream layers read the typed field instead of re-applying a default (the json fallback used to be copied in ten places).

Runtime configuration

Deployment-level settings live in a separate kind: Runtime document (internal/runtimecfg/runtimecfg.go), resolved from --runtime, then ./eventboat.yaml, then defaults; CLI flags override. The decode is typed and strict (candidate 06): unknown keys AND type errors are errors (data_dir: 123, enable: "yes", sample_ratio: -1 used to fall through to the defaults), and sample_ratio must be in [0, 1] (0 = tracing off).

Verify and IR resolution

Production code builds pipelines only through internal/verify (candidate 05): verify.File/verify.Bytes load, build and judge in one call and return a Result with the built *ir.Pipeline, the merged diagnostics and the strict verdict; verify.LoadBytes + verify.Build is the two-stage form for consumers with a middle step (jobs substitutes parameters between the stages). ir.Build(cfg, reg, starOpts, parameters) is the build stage it wraps. Per node, in order:

1. Plugin lookup — the plugin key must resolve in the registry (plugin_unknown) or, for external nodes, match its manifest (grpc_manifest_name, grpc_builtin_conflict for compiled-in names).
2. Version pin check — a node version: must equal the registered or manifest version (plugin_version_mismatch); this is the §6.5 rule that makes “documented but absent” impossible.
3. JSON Schema validation — the plugin block validates against the generated/manifest schema (plugin_schema).
4. Factory instantiation — the registry factory builds a real instance at verify time: the script plugin compiles the Starlark program (expr_starlark_compile), the wasm plugin compiles the module and checks ABI exports (expr_wasm_compile), sources/sinks run their typed build functions (cross-field rules JSON Schema cannot express live here).

Transform instances declaring explain-safe are retained on the IR for explain dry-runs — but only in the retaining build (verify.Options.ForExplain, used by the CLI/ops explain entries and replay --dry-run); the default verify-only build validates every instance and closes it immediately, failure paths included, so nothing outlives a verify. An explain caller owns the pipeline and closes the retained instances with ir.Pipeline.Close when the walkthrough is done. Others (wasm — explain never executes guest code) are always closed immediately. Then: codec resolution (codec_unknown, codec_config), order-key compilation, topological sort, job semantics, telemetry pattern checks and lint.

explain renders the resolved semantics (candidate 09), never a per-plugin re-derivation: the sample message is decoded with the entry source’s declared decoder (the same codec instance the engine resolves — a raw/csv source is not fed JSON), each edge predicate is evaluated once per walk, every inbound edge of a sink is rendered with its own delivery policy plus the engine’s mixed-batch rule (max retries, max explicit timeout; the default timeout applies only when no edge in the batch sets one), and a required: false edge shows its terminal drop — the engine drops and commits the message instead of dead-lettering it. A wasm symbolic line shows the resolved per-invoke mode (fast mode (no per-invoke kill switch) when timeout_ms is unset).

The diagnostics table

Every diagnostic code that exists in the code, by emitting layer. Severity error fails verify; warning is lint (upgraded to error by verify --strict). Example messages are abbreviated.

Loader: file, syntax, document shape

Loader: sections

Loader/sections: nodes and edges

IR: topology, plugins, expressions, jobs, lint

Retired codes

Spec v1.19 retired the hand-parsed transform diagnostics — cfg_transform_main_field, cfg_script_type, cfg_split_type, cfg_wasm_type, cfg_wasm_module, cfg_wasm_range, cfg_wasm_allow — their checks now surface as cfg_missing_plugin/cfg_multiple_plugins or plugin_schema issues like every other kind. expr_starlark_compile, expr_wasm_compile and wasm_no_kill_switch survive via error classification (see Plugins).

Where diagnostics are consumed

• verify / run / test CLI verbs print severity[code] file:line: message plus hints; --json emits the structs.
• The LSP converts the same diagnostics into editor squiggles (internal/lsp/diagnostics.go).
• The MCP verify tool returns them with an ok verdict so agents iterate; deploy fails the tool call when verification fails — verify-first has no bypass.