Metrics

TPF creates framework metrics through its shared OpenTelemetry runtime abstraction. The deployable Quarkus application owns the metric capability and the integration that exports or exposes those instruments. A deployment can use Prometheus/Micrometer, OTLP, New Relic, or another supported platform route without changing TPF's semantic metric contract.

Built-in Metrics

Core pipeline metrics include:

1. pipeline run count, errors, and duration;
2. step duration, errors, retries, and in-flight work;
3. transition count and latency;
4. configured item-boundary throughput;
5. backpressure, Await, connector, and publication signals where those subsystems are used.

Prometheus Through Quarkus

The CSV operator dashboard uses Quarkus Micrometer's Prometheus registry. A deployable that uses this route must include the corresponding Quarkus extension at build time and enable the registry:

quarkus.micrometer.export.prometheus.enabled=true
quarkus.micrometer.export.prometheus.path=/q/metrics

Framework metrics are produced only when the deployable is metrics-capable and both pipeline.telemetry.enabled and pipeline.telemetry.metrics.enabled resolve to true. Prometheus, OTLP, New Relic, or another backend then decides where those metrics go; exporter availability is not an instrumentation switch. See Observability Overview for the full capability, policy, and exporter model.

Dashboards

Pair metrics with Grafana dashboards that show:

1. Step latency percentiles (p95/p99)
2. Throughput per step
3. Error rate by step
4. Pipeline end-to-end latency

For an asynchronous boundary, a total-latency panel is not enough. Keep semantic start/end boundaries visible so an operator can locate the accumulated time: interaction creation, provider admission and dispatch, completion admission, live handoff or durable fallback, continuation, and terminal publication. Use stage throughput, p50/p90/p99 histogram panels where supported, in-flight or queue state, retries/rejects/timeouts, and publication outcomes together.

Healthy runs may not exercise a retry, reject, fallback, or pressure path. Treat an absent series for one of those optional paths as a condition to interpret with the core stage counters, rather than as proof that the primary journey is missing.

Execution Channels and Signals

Queue-async operations involve three distinct channels that should be monitored separately:

Channel	What it means	Core signals
Worker/dispatcher control plane	orchestration coordination and progress	queue depth, worker lag, lease conflicts, stale commits, sweeper recoveries
Execution DLQ	terminal execution failures	DLQ publish count, provider queue depth, the oldest message age
Item Reject Sink	item-level recover-and-continue business rejects	tpf.step.reject.total, provider queue depth (when durable), reject fingerprint concentration

Operational interpretation:

1. High worker lag or stale/lease contention points to orchestration pressure or dependency latency.
2. Execution DLQ growth points to systemic execution failures that require execution-level triage.
3. Item reject growth often indicates data-quality/business-rule drift and should route to business remediation and selective re-drive.

LGTM Metrics Pipeline

LGTM Dev Services ship an OTLP collector and Prometheus. Grafana's built-in dashboards read from the Prometheus datasource, so Prometheus scraping must be enabled even if OTLP export is configured. For OTLP-first dashboards, you need a Grafana datasource that reads OTLP metrics storage (for example Mimir) instead of Prometheus.

Prometheus cadence only affects the metrics view. It does not affect live trace delivery to Tempo. For the split between metrics dashboards, Tempo live topology, and replay playback, see Replay & Live Topology.

Parallelism and Backpressure

TPF emits additional metrics and span attributes to showcase parallelism and buffer pressure:

Metrics (OTel/Micrometer):

• tpf.step.inflight (gauge): in-flight items per step (tpf.step.class attribute)
• tpf.step.buffer.queued (gauge): queued items in the backpressure buffer (tpf.step.class attribute)
• tpf.step.buffer.capacity (gauge): configured backpressure buffer capacity per step (tpf.step.class attribute)
• tpf.step.parent (attribute): parent step class for plugin steps (same as tpf.step.class for regular steps)
• tpf.pipeline.max_concurrency (gauge): configured max concurrency for the pipeline run
• tpf.item.produced (counter): items produced at the configured item boundary
• tpf.item.consumed (counter): items consumed at the configured item boundary
• tpf.slo.rpc.server.* (counters): SLO-ready totals for RPC server reliability and latency (gRPC + REST)
• tpf.slo.rpc.client.* (counters): SLO-ready totals for RPC client reliability and latency (gRPC + REST)
• tpf.slo.item.throughput.* (counters): SLO-ready totals for item throughput per run

Prometheus exports these as *_items because the unit is set to items.

Note: tpf.step.* metrics represent step executions (not domain items). Use the tpf.item.* counters when you want throughput for a specific domain type.

Note: New Relic dimensional metrics treat tpf.slo.item.throughput.* as event-counted counters, so SLOs should use COUNT (not SUM) over metricName = 'tpf.slo.item.throughput.total|good'.

Aspect position note: AFTER_STEP observes the output of each step. This captures every boundary except the very first input boundary (before the pipeline starts). Conversely, BEFORE_STEP captures every boundary except the final output boundary (after the pipeline completes). Use two aspects if you need complete boundary coverage.

Run-level span attributes (on tpf.pipeline.run):

• tpf.parallel.max_in_flight
• tpf.parallel.avg_in_flight

These are designed for batch-style pipelines where parallelism should be inspected while the pipeline is running.

Tip: gauges report the instantaneous value, so after a run finishes they will return to 0. When querying, use a max over time window to surface the peak:

max(tpf_step_inflight_items) by (tpf_step_class)
max(tpf_step_buffer_queued_items) by (tpf_step_class)

Application Metrics

Application code may use Micrometer for application-specific counters and timers. These instruments are separate from TPF's framework telemetry ownership and should not duplicate a TPF semantic fact:

@Inject
MeterRegistry registry;

Timer timer = registry.timer("payment.processing.duration");
Counter success = registry.counter("payment.processing.success");

return timer.recordCallable(() -> processPayment(record));

Connector And Await Boundary Metrics

Connector-owned I/O and await boundaries use low-cardinality metrics. Keep object keys, execution ids, await unit ids, interaction ids, and correlation ids in spans/replay events, not metric attributes.

Object Ingest metrics:

Metric	Type	Attributes	Meaning
tpf.object_ingest.list.total	counter	tpf.object_ingest.source, tpf.object_ingest.provider	Source listing attempts.
tpf.object_ingest.listed.objects.total	counter	tpf.object_ingest.source, tpf.object_ingest.provider	Objects returned by listing.
tpf.object_ingest.submitted.total	counter	tpf.object_ingest.source, tpf.object_ingest.provider	Objects accepted as queue-async execution inputs.
tpf.object_ingest.duplicate.total	counter	tpf.object_ingest.source, tpf.object_ingest.provider	Duplicate object admissions resolved by idempotency.
tpf.object_ingest.failed.total	counter	tpf.object_ingest.source, tpf.object_ingest.provider	Listing, mapping, or submission failures.

Object Publish metrics:

Metric	Type	Attributes	Meaning
tpf.object_publish.grouped.total	counter	tpf.object_publish.target	Terminal output grouping operations.
tpf.object_publish.grouped.items.total	counter	tpf.object_publish.target	Terminal items seen by Object Publish.
tpf.object_publish.grouped.groups.total	counter	tpf.object_publish.target	Object groups created for publication.
tpf.object_publish.published.total	counter	tpf.object_publish.target, tpf.object_publish.provider	Objects successfully written.
tpf.object_publish.published.bytes.total	counter	tpf.object_publish.target, tpf.object_publish.provider	Bytes written by successful publishes.
tpf.object_publish.skipped.total	counter	tpf.object_publish.target	Empty terminal outputs skipped.
tpf.object_publish.failed.total	counter	tpf.object_publish.target, tpf.object_publish.provider	Publish failures.
tpf.object_publish.write.duration	histogram	tpf.object_publish.target, tpf.object_publish.provider	Provider write duration in milliseconds.

Await boundary metrics:

Metric	Type	Attributes	Meaning
tpf.await.interaction.dispatched.total	counter	step, status, transport	Await interactions dispatched to an external actor.
tpf.await.unit.dispatch_complete.total	counter	step, cardinality, status	Await units whose dispatch phase completed.
tpf.await.completion.admitted.total	counter	step, status, transport	Completion envelopes admitted into await state.
tpf.await.item.completed.total	counter	step, cardinality, status, transport	Itemized await completions recorded.
tpf.await.completion.early_held.total	counter	step, cardinality, status, transport	Item completions held for durable fallback because live release was not available yet.
tpf.await.resume.released.total	counter	step, cardinality/status/transport when known	Await resumes released from durable fallback state.
tpf.await.unit.terminal.total	counter	step, cardinality, status, transport	Await units reaching terminal state.
tpf.await.completion.latency	histogram	step, status, transport	Time from interaction creation to completion admission.
tpf.await.unit.duration	histogram	step, cardinality, status, transport	Time from await unit creation to terminal state.
tpf.await.completion.dropped.total	counter	transport, reason	Completions that cannot be admitted because the target interaction is already terminal, stale, or otherwise not admissible.

Prometheus exports OpenTelemetry units in the metric name. For example, the duration histograms are exported with _milliseconds_* suffixes in the Quarkus Prometheus endpoint.

SLO-friendly derived indicators:

1. Await admission reliability: admitted completions divided by admitted plus dropped completions.
2. Await flow health: admitted completions followed by downstream step progress in the live path; durable resume releases and early-held completions draining in fallback paths.
3. Object publish reliability: published objects divided by published plus failed objects.
4. Object publish latency: p95/p99 of tpf.object_publish.write.duration by provider and target.
5. CSV output completeness: Object Publish grouped item count matches the terminal PaymentOutput count for the run.

Keep demo expectations separate from production SLOs. The CSV Payments demo may report provider permits/sec, wall time, throughput, output count, and checksum, but production thresholds should come from service-specific latency/error budgets and provider-native backlog/lag signals.

Use replay JSON for high-cardinality drill-down. It includes object keys, await unit ids, interaction ids, execution ids, and correlation fields that must not become metric dimensions.

Orchestrator Queue-Async Signals

For QUEUE_ASYNC, include control-plane metrics in addition to step metrics. Treat these as required operational signals for GA readiness:

1. lease claim conflicts (OCC contention),
2. stale commit rejections,
3. retry scheduling rate and retry-saturation ratio,
4. due-sweeper recovery count (persisted-before-dispatch gap recovery),
5. execution DLQ publish count and backlog depth,
6. item reject sink publish count and backlog depth,
7. queue depth and worker lag.

Use these to separate dependency outages (high retries, low success) from coordination issues (high stale/lease conflicts).

Implementation note:

1. TPF core already emits step/pipeline telemetry.
2. Control-plane metrics may be emitted by provider integration or surrounding platform telemetry (queue, datastore, worker runtime).
3. Keep metric names stable per environment even if data comes from different backends.

Step-level reject signal:

• tpf.step.reject.total (counter): rejected step items published to item reject sinks.

Await execution logs include a parked await unit when a QUEUE_ASYNC execution uses durable fallback to wait or resume. Replay and trace events expose those durable await-unit transitions, including dispatch, waiting, item completion, unit completion, resume release, and terminal timeout/failure states. A healthy live itemized handoff is observed through durable interaction completion and downstream step telemetry; it does not emit unit item-completion or resume-release events. Metrics expose aggregate await-boundary health without high-cardinality ids.

Command signal:

• command steps participate in normal tpf.step.* metrics and tpf.step spans,
• command effect lifecycle is exposed through tpf.command.effect.* metrics,
• effect lifecycle state is also recorded in the configured CommandEffectStore,
• provider backlog, throttling, and external latency should come from the provider connector or platform telemetry.

Replay topology marks command steps with renderRole: "command" and actorKind equal to the command name. That topology signal is for inspection and playback; it is not a replacement for provider-native command metrics.

Command effect metrics:

OpenTelemetry metric	Prometheus name	Type	Key attributes
tpf.command.effect.transition.total	tpf_command_effect_transition_total	counter	tpf.command, tpf.command.step, tpf.command.status
tpf.command.effect.duplicate.total	tpf_command_effect_duplicate_total	counter	tpf.command, tpf.command.step, tpf.command.duplicate_policy, tpf.command.duplicate_result
tpf.command.effect.duration	tpf_command_effect_duration_*	histogram	tpf.command, tpf.command.step, tpf.command.status

Command status values are pending, dispatching, succeeded, failed_retryable, ambiguous, user_action_required, and dlq. ambiguous and user_action_required are distinct durable barriers even though their enclosing pipeline execution follows the established terminal failure/DLQ path. Duplicate result values are returned_recorded, rejected, and in_progress.

Command step SLO examples:

SLO	Primary signal	Example objective
Command effect completion	tpf.command.effect.transition.total{tpf_command_status="succeeded"}	99.9% of command effects reach SUCCEEDED within 5 minutes.
Terminal command failure budget	tpf.command.effect.transition.total{tpf_command_status="dlq"}	Fewer than 0.1% of command effects enter terminal DLQ over 30 days.
Command step latency	tpf.step latency for the command step	p95 under the service target for the provider, such as 2 seconds for search indexing.
Replay duplicate safety	tpf.command.effect.duplicate.total{tpf_command_duplicate_result="returned_recorded"} plus provider write count	RETURN_RECORDED duplicates return stored output with zero extra provider writes.
Provider health	connector or provider metrics	Provider throttling and backlog stay below the connector's retry budget.

PromQL examples:

# Success ratio by command over 5 minutes.
sum by (tpf_command) (rate(tpf_command_effect_transition_total{tpf_command_status="succeeded"}[5m]))
/
sum by (tpf_command) (rate(tpf_command_effect_transition_total{tpf_command_status=~"succeeded|failed_retryable|ambiguous|user_action_required|dlq"}[5m]))

# Terminal DLQ rate by command.
sum by (tpf_command) (rate(tpf_command_effect_transition_total{tpf_command_status="dlq"}[5m]))

# Recorded duplicate replays by command.
sum by (tpf_command) (increase(tpf_command_effect_duplicate_total{tpf_command_duplicate_result="returned_recorded"}[1h]))

# p95 command effect duration by command.
histogram_quantile(
  0.95,
  sum by (le, tpf_command) (rate(tpf_command_effect_duration_bucket{tpf_command_status="succeeded"}[5m]))
)

For a Search/OpenSearch indexing command, use tpf.command.effect.transition.total to count succeeded, failed_retryable, and dlq effects by tpf_command="opensearch-index-document". Use tpf.step.duration for the Write Search Index Document step's pipeline latency, and use OpenSearch client or cluster metrics for provider latency, throttling, and indexing failures.

Backlog signal note:

1. TPF emits tpf.step.reject.total for reject throughput.
2. Backlog depth is provider-native:
3. use SQS queue depth/age for durable sinks (provider=sqs),
4. use retained-size logs/metrics for in-memory sink (provider=memory).

Design Tips

1. Prefer low-cardinality labels
2. Track user-visible latency
3. Align metrics with SLIs/SLOs
4. Measure queue depth if you use streaming steps