Not another workflow engine or CI/CD pipeline. TPF pipelines are strongly typed application flows: each step transforms explicit business types while TPF handles persistence, transport, replay, observability, retries, and deployment concerns.
Recent DEV Community posts tagged tpf.
Fresh from GitHub releases for The Pipeline Framework.
Model business flows as explicit transformations between typed contracts. TPF validates application boundaries, cardinality, mappings, and transport compatibility before deployment, reducing semantic drift as systems evolve.
Write business logic as ordinary Java functions. TPF isolates persistence, messaging, APIs, file ingestion, asynchronous workflows, retries, and deployment concerns from application decisions.
Databases, APIs, Kafka, S3, files, human approvals, long-running jobs, and external providers are modeled through connectors and runtime primitives instead of custom glue code.
Run locally, over REST, gRPC, Kafka, SQS, or serverless platforms without changing business logic. Infrastructure choices become deployment concerns rather than application concerns.
Execution lineage, captured external state, persistence, caching, retries, and idempotency help make distributed systems easier to reason about, debug, and recover.
Execution tracking, telemetry, audit trails, checkpointing, and runtime validation are available from the beginning rather than added later through custom instrumentation.
Pause execution while waiting for human approvals, webhook callbacks, provider responses, and long-running jobs without polling loops, correlation tables, or custom state machines.
Distributed-system complexity often begins as semantic drift rather than network failures. TPF makes contracts explicit through strongly typed application boundaries and generated transports, helping teams evolve safely as systems grow.
TPF pipelines are strongly typed application flows. They are not CI/CD job chains, coarse batch jobs, or generic workflow diagrams. A step is one business transformation: it receives an explicit Java input type, produces an explicit Java output type, and leaves distributed-system concerns to the framework-owned shell.
Focused Java business functions such as validating a payment, enriching a record, parsing a document, or deciding how to handle a domain event.
Connectors, adapters, mappers, persistence, caching, await boundaries, and transport-specific calls that connect the core to external reality.
The runtime starts the flow, calls each step, records lineage when configured, retries failed work, exposes status, and routes terminal failures.
You write typed Java functions and declare their application flow in YAML. TPF checks the flow at build time, including whether each function shape is valid, whether outputs match the next inputs, whether mappers convert types correctly, whether operators are valid, and whether generated transports can call each boundary cleanly.
A mapper translates between your domain types and transport or external-system types. An operator is an existing Java method or remote endpoint reused as a pipeline function.
Start with pipeline compilation when you want the build-time generation model, or code a step when you want to implement the function itself.
TPF creates the connector and transport shells that would otherwise become handwritten service glue. An adapter is generated code around your business function: it lets another component call the function through the selected transport without moving transport logic into the function.
Connectors own I/O admission such as file and object ingest. Await steps own durable suspend/resume for human approvals, webhook callbacks, provider responses, and long-running jobs. Generated transports keep business logic independent of whether a caller uses local calls, REST, gRPC, messaging, or a serverless function entry point. See Object Ingest, Await Boundaries, and runtime layouts when you want the implementation details.
The runtime starts the flow, invokes each step, records execution state when configured, retries failed transitions, and exposes status/result endpoints for background work. For crash-surviving background execution, TPF can store accepted work outside the current JVM, recover it after restart, and send terminal execution failures to a DLQ, a dead-letter channel for investigation or replay.
The exact config value for this mode is QUEUE_ASYNC; start with the Orchestrator Runtime guide before using it in production.
Persistence and caching matter here more than the word "plugin" suggests. Persistence stores business outputs developers can query later from APIs, reports, or UIs. Caching protects expensive steps, speeds recomputation, and supports replay or rewind scenarios when downstream logic changes. Together they reduce the need to invent separate state stores, replay workflows, or read-model plumbing.
Plugins add declared cross-cutting work such as persistence, cache, telemetry, and logging. An aspect is the rule that says where the plugin runs, for example before or after a step; the plugin provides the implementation.
That keeps business functions focused on domain behaviour while TPF keeps generated plugin calls aligned across transports and runtime layouts.
See State, Replay, and Queryable Data for the full persistence-and-caching story.
Validate and enrich records, await provider decisions through Kafka, produce status output, reject one malformed item, and continue the workload.
Admit documents through object connectors, parse, tokenize, index, cache intermediate results, and replay safely when source content changes.
Pass stable checkpoints from one pipeline to the next without inventing custom handoff code.
Reuse embedding, vector-search, or LLM helper libraries as operators inside a typed Java flow.
Pause while a person, webhook, provider, or long-running job decides, then resume the same execution without custom correlation state.
Run the same typed flow locally, behind APIs, through messaging, on serverless platforms, or as distributed services when the deployment model requires it.
FUNCTION platform guide.