The Pipeline Framework

Appearance

Durable Coordinator ​

The durable coordinator is the self-hostable control-plane boundary for QUEUE_ASYNC execution.

It owns execution state, leases, retry/DLQ, await units, release activation, worker dispatch, and status/result APIs. Step code still runs in workers: local in-process workers, REST workers, gRPC workers, or SQS request/reply workers.

This section is implementation-facing. Application usage remains in Orchestrator Runtime. The first runnable reference is examples/restaurant-approval/self-host.

Current Shape ​

AreaCurrent state
Execution stateExecutionRecord with leases, attempts, status, result, pinned pipeline/contract/release identity
Await stateAwaitUnitRecord plus pending/completion interaction records
Worker boundaryportable command/result envelopes over local, REST, gRPC, or SQS
Contract/release identitygenerated META-INF/pipeline/pipeline-contract.json, release descriptor registration, activation, execution pinning, and worker identity validation
Self-host pathbatteries-included local coordinator using the restaurant approval example

Guides ​

  1. Worker Protocols explains local, REST, gRPC, and SQS transition workers.
  2. Step-Aware Invocation Runtime explains the shared invocation seam used by pipeline steps and transition workers.
  3. Brokered Runtime Boundaries is the entry point for Kafka/SQS-style substrates under TPF-owned semantics.
  4. Boundary Taxonomy maps broker concepts into TPF runtime boundaries.
  5. Dispatch Substrates separates substrate policy from transport, platform, and payload policy.
  6. Envelope And Data Policy separates loose payloads from strict TPF control metadata.
  7. Contract And Release Identity explains generated contracts, release activation, and execution pinning.
  8. Pipeline Contract And Release Model describes contract/release descriptors, artifacts, deployment plans, and drift detection.
  9. Runtime Boundaries And Performance explains runtime mapping, patterns, package boundaries, and hot-path guardrails.
  10. Local APIs documents the current default-disabled control-plane and admin APIs.
  11. Self-Hosted Deployment gives the production-ish self-host topology, configuration, and operator runbooks.
  12. Self-Hosted HA Roadmap tracks the hardening work left after the container reference.
  13. Self-Hosted Milestone tracks what remains after the current self-host proof.

Limits ​

The current coordinator path does not dynamically load registered JAR code. Workers must already host matching pipeline code and validate active pipelineId + contractVersion + releaseVersion identity.

The Dynamo release registry provides multi-coordinator release metadata, while the file-backed registry remains local/dev oriented. Minimal worker lifecycle now gates new hosted submissions. Single-execution re-drive is present; bulk DLQ-message replay and append-only execution/await state remain follow-up runtime substrate work.