The Pipeline Framework

Appearance

Pipeline Compilation and Generation ​

This guide explains how The Pipeline Framework's annotation processor works to automatically generate pipeline applications and adapters at build time.

For the architecture overview of the processor, see Annotation Processor Guide

Important The default project structure and parent POM lifecycle described in this guide come from the scaffold generated by the Web UI Canvas Designer (or template generator). Runtime mapping (pipeline.runtime.yaml) controls logical placement and transport-aware generation, but it does not automatically rewrite your Maven module topology. For topology migration guidance, see Runtime Layouts and Build Topologies in the section below.

For a build-phase walkthrough of the generated parent POM, see:

For runtime layout usage and build migration, see:

For runnable await examples, see:

For await suspend/resume internals, see Await Unit Runtime.

Overview ​

The Pipeline Framework uses YAML-first compilation to automatically generate the necessary infrastructure for pipeline execution. For internal service: steps:

  1. pipeline.yaml declares the service class, cardinality, domain input/output types, and optional inbound/outbound mappers
  2. @PipelineStep marks the Java service class, so the compiler can discover it
  3. The compiler validates the YAML contract against the implemented reactive service interface
  4. It generates transport adapters and client steps for the configured transport (gRPC or REST)
  5. It expands configured aspects into synthetic side effect steps when a plugin host is present
  6. It registers all generated components with the dependency injection container

This eliminates the need for manual configuration and ensures consistency across your pipeline.

Annotation Processing Workflow ​

Proto Generation (Pre-Processing) ​

Before annotation processing, pipeline protobuf descriptors are generated from the pipeline template. The authoritative source is:

  • framework/runtime/src/main/java/org/pipelineframework/proto/PipelineProtoGenerator.java

Build Timeline (gRPC) ​

text
Pipeline template
      |
      v
PipelineProtoGenerator
      |
      v
protoc -> descriptor set (.desc)
      |
      v
Annotation processor -> adapters/clients/resources/CLI
      |
      v
CDI registration

1. Build-Time Discovery ​

During the Maven build process, the compiler reads pipeline.yaml and resolves each internal service: step against a class annotated with @PipelineStep:

yaml
steps:
  - name: process-payment
    service: com.app.payment.ProcessPaymentService
    cardinality: ONE_TO_ONE
    input: com.app.domain.PaymentRecord
    output: com.app.domain.PaymentStatus
    inboundMapper: com.app.payment.PaymentRecordMapper
    outboundMapper: com.app.payment.PaymentStatusMapper
java
@PipelineStep
@ApplicationScoped
public class ProcessPaymentService implements ReactiveService<PaymentRecord, PaymentStatus> {
    @Override
    public Uni<PaymentStatus> process(PaymentRecord input) {
        // Implementation
    }
}

The annotation is the marker plus Java-local execution hints. Current internal-step contract metadata belongs in YAML.

1.1 Orchestrator and Plugin Annotations ​

The processor also reacts to:

  • @PipelineOrchestrator on a marker class to enable orchestrator endpoints and (optionally) CLI generation.
  • @PipelinePlugin on plugin services to enable plugin-server generation and plugin-aspect expansion.

These annotations do not define pipeline steps themselves, but they control which orchestrator and plugin artifacts are generated.

2. Compile-time Code Generation ​

The Pipeline Framework extension processor generates several classes:

  • If transport: GRPC, gRPC service adapters and gRPC client steps.
  • If transport: REST, REST resource adapters and REST client steps.
  • Synthetic client steps for configured plugin aspects (in a plugin host module).

2.5 Scaffolding ​

The template generator provides the necessary scaffolding for:

  • Service and orchestrator entry points
  • Step interfaces and DTO placeholders
  • REST/gRPC adapter wiring and routing
  • Configuration files and environment defaults
  • Tests and sample fixtures
  • CI/workflow stubs for build and release

3. Dependency Injection Registration ​

All generated classes are automatically registered with the CDI container, making them available for injection.

Generated Classes in Detail ​

Role-Specific Output Directories ​

Generated sources are written into role-specific directories under target/generated-sources/pipeline, one per deployment role. Packaging relies on these directories instead of class-name patterns.

Build configuration notes:

  • Pass -Apipeline.generatedSourcesDir=target/generated-sources/pipeline to the compiler during compile only, to avoid warnings during testCompile.
  • Register the role directories as sources for IDEs via build-helper-maven-plugin.
  • If tests reference generated classes (e.g., REST resources), register the same directories as test sources in generate-test-sources.

Generated Pipeline Metadata ​

The build also emits runtime metadata under META-INF/pipeline/:

  • order.json: resolved runtime execution order
  • telemetry.json: item-boundary and parent-step telemetry metadata
  • replay-topology.json: replay and live-topology metadata used by execution-event playback and trace-aware topology surfaces

If you package a grouped runtime such as monolith or pipeline-runtime, keep these resources aligned with the runtime artifact that will execute the pipeline.

gRPC Adapter Generation ​

The gRPC adapter acts as a server-side endpoint that:

  1. Receives gRPC requests
  2. Uses the inbound mapper to convert gRPC objects to domain objects
  3. Calls the actual service implementation
  4. Uses the outbound mapper to convert domain objects to gRPC responses
java
// Generated class structure (simplified)
public class ServiceNameGrpcService extends ServiceNameGrpc.ServiceNameImplBase {

    @Inject
    PaymentRecordInboundMapper inboundMapper;

    @Inject
    PaymentStatusOutboundMapper outboundMapper;

    @Inject
    ServiceName service;  // Your actual service implementation

    @Override
    public Uni<PaymentGrpcOut> remoteProcess(PaymentGrpcIn request) {
        // Delegates to an inline GrpcReactiveServiceAdapter based on the streaming shape
        return /* adapter */.remoteProcess(request);
    }
}

gRPC Step Class Generation ​

The step class acts as a client-side component that:

  1. Connects to the gRPC service
  2. Implements the pipeline step interface
  3. Handles the conversion between domain objects and gRPC calls
java
// Generated class structure
@ApplicationScoped
public class ServiceNameGrpcClientStep implements StepOneToOne<DomainIn, DomainOut> {

    @Inject
    @GrpcClient("service-name")
    StubClass grpcClient;

    public Uni<DomainOut> applyOneToOne(DomainIn input) {
        // Convert domain to gRPC
        GRpcIn grpcInput = convertDomainToGrpc(input);

        // Call remote service
        return grpcClient.remoteProcess(grpcInput);
    }
}

Orchestrator Application Structure ​

The orchestrator application coordinates pipeline execution by using the PipelineExecutionService to connect all generated steps:

java
// Orchestrator application that coordinates execution
@CommandLine.Command(...)
public class OrchestratorApplication implements QuarkusApplication, Callable<Integer> {

    @Inject
    PipelineExecutionService pipelineExecutionService;

    public Integer call() {
        // Create input stream from input parameter
        Multi<DomainInput> inputStream = createInputStream(input);

        // Execute pipeline using the injected service
        // The service discovers all registered step implementations through dependency injection
        pipelineExecutionService.executePipeline(inputStream)
            .collect().asList()
            .await().indefinitely();

        return CommandLine.ExitCode.OK;
    }
}

The actual pipeline execution is handled by the PipelineExecutionService which discovers all available step implementations through the StepsRegistry.

Module Ownership and Dependencies ​

This build is organized into three groups plus shared scaffolding:

  • common (scaffolded): DTOs and mappers used by services and orchestrator
  • services: step implementations + server adapters (gRPC or REST)
  • orchestrator: orchestrator endpoints + orchestrator CLI + client steps

Client steps are only used by the orchestrator (there is no direct step-to-step communication).

Ownership Matrix (Generated Artifacts) ​

ArtifactOwner ModuleConsumers
DTOs + mapperscommonservices, orchestrator
gRPC server adaptersservicesruntime/CDI
REST resourcesservicesruntime/CDI
gRPC client stepsorchestratororchestrator runtime
REST client stepsorchestratororchestrator runtime
Orchestrator endpointsorchestratorruntime/CDI
Orchestrator CLIorchestratoruser

Build Process Integration ​

Maven Configuration ​

The pipeline framework integrates with the Maven build process. Both runtime and deployment components are bundled in a single dependency:

xml
<!-- pom.xml dependencies -->
<dependency>
    <groupId>org.pipelineframework</groupId>
    <artifactId>pipelineframework</artifactId>
</dependency>

Annotation Processor Execution ​

The annotation processor runs during the compile phase:

bash
# During mvn compile
[INFO] --- quarkus:3.28.0.CR1:generate-code (default) @ service-module ---
[INFO] [org.pipelineframework.processor.PipelineStepProcessor] Generating adapters for annotated services
[INFO] [org.pipelineframework.processor.PipelineStepProcessor] Found 3 @PipelineStep annotated services
[INFO] [org.pipelineframework.processor.PipelineStepProcessor] Generated ProcessPaymentServiceGrpcService
[INFO] [org.pipelineframework.processor.PipelineStepProcessor] Generated ProcessPaymentGrpcClientStep
[INFO] [org.pipelineframework.processor.PipelineStepProcessor] Generated SendPaymentServiceGrpcService
[INFO] [org.pipelineframework.processor.PipelineStepProcessor] Generated SendPaymentGrpcClientStep
[INFO] [org.pipelineframework.processor.PipelineStepProcessor] Generated ProcessAckPaymentServiceGrpcService
[INFO] [org.pipelineframework.processor.PipelineStepProcessor] Generated ProcessAckPaymentGrpcClientStep
[INFO] [org.pipelineframework.processor.PipelineStepProcessor] Generated step implementations and service adapters

Required gRPC Descriptor Set Generation ​

The annotation processor resolves gRPC bindings from a protobuf descriptor set. Configure your build to emit a descriptor set (for example via Quarkus gRPC codegen) or pass protobuf.descriptor.file/protobuf.descriptor.path to the annotation processor if you have a custom descriptor location.

Generated Code Verification ​

Viewing Generated Sources ​

Generated sources can be found in the target directory:

bash
# Generated pipeline sources location
target/generated-sources/pipeline/

# Generated classes location
target/classes/

Debugging Generation Issues ​

Enable verbose logging to debug generation issues:

properties
# application.properties
quarkus.log.category."org.pipelineframework.processor".level=DEBUG

Customization Points ​

Extending Generated Classes ​

While generated classes are typically not modified directly, you can extend them:

java
// Custom extension of generated step
@ApplicationScoped
public class CustomProcessPaymentGrpcClientStep extends ProcessPaymentGrpcClientStep {

    @Override
    public Uni<PaymentStatus> applyOneToOne(PaymentRecord input) {
        // Add custom logic before/after calling super
        return super.applyOneToOne(input)
            .onItem().invoke(status -> {
                // Custom post-processing
                logPaymentStatus(status);
            });
    }

    private void logPaymentStatus(PaymentStatus status) {
        // Custom logging logic
    }
}

Customizing Generation ​

Use configuration and transport settings instead of transport-specific annotation fields:

  • Set transport: GRPC or transport: REST in pipeline.yaml.
  • Override REST paths with pipeline.rest.path.<ServiceName> in application.properties.

Troubleshooting ​

Common Issues ​

1. Missing Dependencies ​

Ensure the required dependency is present. Both runtime and deployment components are bundled in a single dependency:

xml
<dependency>
    <groupId>org.pipelineframework</groupId>
    <artifactId>pipelineframework</artifactId>
</dependency>

2. Annotation Processing Not Running ​

Verify the processor is on the classpath:

bash
# Check that the pipelineframework dependency is included
mvn dependency:tree | grep pipelineframework

3. Generated Classes Not Found ​

Check the generated sources directory:

bash
# List generated classes
find target/generated-sources -name "*.java" | grep -i pipeline

Debugging Tips ​

Enable Detailed Logging ​

properties
# application.properties
quarkus.log.category."org.pipelineframework".level=DEBUG
quarkus.log.category."org.pipelineframework.processor".level=TRACE

Verify Generated Classes ​

bash
# Check that step classes were generated
find target/classes -name "*Step.class" | head -5
# Check that gRPC service classes were generated
find target/classes -name "*GrpcService.class" | head -5

Clean and Rebuild ​

bash
# Clean build to force regeneration
mvn clean compile

Best Practices ​

Development Workflow ​

  1. Author Internal Step Contracts in YAML: Define service, cardinality, input, output, and optional inboundMapper / outboundMapper in pipeline.yaml.
  2. Build Project: Run mvn compile to trigger generation
  3. Verify Generation: Check that generated classes are created and the service interface matches the YAML cardinality
  4. Test Integration: Run integration tests to verify the pipeline works
  5. Deploy: Deploy the complete application with generated components

Maintenance ​

  1. Keep YAML and Service Signatures in Sync: Update pipeline.yaml whenever an internal service interface changes
  2. Review Generated Code: Periodically review generated code for correctness
  3. Monitor Build Logs: Watch for generation warnings or errors
  4. Test Changes: Thoroughly test after making changes to YAML-defined steps or service implementations

Performance Considerations ​

  1. Minimize Regeneration: Only rebuild when annotations change
  2. Optimize Mappers: Ensure mappers are efficient
  3. Configure Retries: Set appropriate retry limits and wait times
  4. Monitor Resource Usage: Watch memory and CPU usage of generated components

The Pipeline Framework's annotation processing provides a powerful way to automatically generate pipeline infrastructure while maintaining type safety and reducing boilerplate code. By understanding how this process works, you can leverage its full potential while troubleshooting any issues that may arise.