The Pipeline Framework

Appearance

Archived Docsv26.7.1

Durable Coordinator ​

The durable coordinator is the self-hostable control-plane boundary for QUEUE_ASYNC execution.

It owns execution state, leases, retry/DLQ, await units, release activation, worker dispatch, and status/result APIs. Step code still runs in workers: local in-process workers, REST workers, gRPC workers, or SQS request/reply workers.

If you are trying to understand what happened to the old "orchestrator", start with Coordinator And Worker Topology. The short version is that orchestrator-svc and pipeline.orchestrator.* remain historical module/config names, while self-host HA splits runtime responsibility into a coordinator role and one or more transition worker roles.

This section is implementation-facing. Application usage remains in Orchestrator Runtime. The first runnable reference is examples/restaurant-approval/self-host.

The current self-host HA path is compute-first. For the future FUNCTION/all-serverless HA design track, see All-Serverless Durable Coordinator.

Current Shape ​

AreaCurrent state
Execution stateExecutionRecord with leases, attempts, status, result, pinned pipeline/contract/release identity
Await stateAwaitUnitRecord plus pending/completion interaction records
Worker boundaryportable command/result envelopes over local, REST, gRPC, or SQS
Contract/release identitygenerated META-INF/pipeline/pipeline-contract.json, release descriptor registration, activation, execution pinning, and worker identity validation
Self-host pathcompute-first HA references using restaurant approval and CSV Payments

Guides ​

  1. Coordinator And Worker Topology explains the role split behind orchestrator-svc, coordinator processes, and transition workers.
  2. Worker Protocols explains local, REST, gRPC, and SQS transition workers.
  3. Step-Aware Invocation Runtime explains the shared invocation seam used by pipeline steps and transition workers.
  4. Brokered Runtime Boundaries is the entry point for Kafka/SQS-style substrates under TPF-owned semantics.
  5. Boundary Taxonomy maps broker concepts into TPF runtime boundaries.
  6. Dispatch Substrates separates substrate policy from transport, platform, and payload policy.
  7. Envelope And Data Policy separates loose payloads from strict TPF control metadata.
  8. Contract And Release Identity explains generated contracts, release activation, and execution pinning.
  9. Pipeline Contract And Release Model describes contract/release descriptors, artifacts, deployment plans, and drift detection.
  10. Runtime Boundaries And Performance explains runtime mapping, patterns, package boundaries, and hot-path guardrails.
  11. All-Serverless Durable Coordinator evaluates whether QueueAsyncCoordinator can become single-shot actions for FUNCTION/all-serverless HA.
  12. Local APIs documents the current default-disabled control-plane and admin APIs.
  13. Self-Hosted Deployment gives the production-ish self-host topology, configuration, and operator runbooks.
  14. Self-Hosted HA Roadmap records the milestone closeout and deferred hardening.
  15. Self-Hosted Milestone gives the adoption entry points and current proof matrix.

Limits ​

The current coordinator path does not dynamically load registered JAR code. Workers must already host matching pipeline code and validate active pipelineId + contractVersion + releaseVersion identity.

The Dynamo release registry provides multi-coordinator release metadata, while the file-backed registry remains local/dev oriented. Minimal worker lifecycle gates new hosted submissions. Single-execution re-drive is present. Bulk DLQ-message replay and append-only execution/await state are deferred hardening work, not blockers for the current compute-first self-host HA milestone.