Observability Overview

Observability in The Pipeline Framework is designed for distributed pipelines: you should be able to see what each step did, how long it took, and where failures occurred.

What You Get Out of the Box

• Metrics: Step timings, throughput, and failure counts
• Tracing: End-to-end request visibility across steps
• Replay & Live Topology: Separate the offline replay viewer from live Tempo and Prometheus surfaces
• Logging: Structured logs with correlation identifiers
• Health Checks and In-flight Probe: Liveness, readiness and killswitch for orchestration
• Alerting: Dashboards and alert rules tuned for pipeline behavior
• Security Notes: Prevent accidental leakage of sensitive information
• Working with NewRelic OTel: Enabling OTel export to use NewRelic
• Test locally using LGTM: Enabling Prometheus metrics for Grafana dashboards on Quarkus LGTM stack

Managed external boundaries appear as first-class nodes. Await steps expose suspend/resume lifecycle events. Command steps appear as command nodes in replay topology and participate in normal step spans and metrics while their effect lifecycle is recorded by the command effect store.