# OpenTelemetry Tracing

Helmfile can export [OpenTelemetry](https://opentelemetry.io/) traces of a run, which is especially useful in CI/CD: see where time is spent, which helm invocations dominate, how label selectors fan out, and what to target for improvement.

Tracing is **experimental** and **off by default**. It currently emits the command-level root span, state-loading spans (discover/load/render/parse), one span per external process (helm invocations, hooks, plugin execs), per-hook spans, and per-release spans for the sync/diff/delete/status/test/prepare paths.

## Enabling

```bash
helmfile --otel-tracing -e production -l tier=backend apply
```

or via environment variable (handy for CI):

```bash
export HELMFILE_OTEL_TRACING=true
helmfile sync
```

`--otel-tracing` overrides `HELMFILE_OTEL_TRACING` when specified.

## Configuration

Everything beyond the on/off switch uses the standard [OpenTelemetry environment variables](https://opentelemetry.io/docs/specs/otel/configuration/sdk-environment-variables/); helmfile defines no telemetry-specific variables of its own.

| Variable | Default | Notes |
|---|---|---|
| `OTEL_EXPORTER_OTLP_ENDPOINT` | `http://localhost:4318` | OTLP endpoint; use `http://host:4317` for gRPC |
| `OTEL_EXPORTER_OTLP_PROTOCOL` | `http/protobuf` | or `grpc` |
| `OTEL_EXPORTER_OTLP_HEADERS` | — | e.g. `authorization=Bearer <token>` |
| `OTEL_TRACES_EXPORTER` | `otlp` | `otlp` \| `console` \| `none` |
| `OTEL_METRICS_EXPORTER` | `otlp` | `otlp` \| `console` \| `prometheus` \| `none` — see [Metrics](#metrics) |
| `OTEL_TRACES_SAMPLER` (+ `OTEL_TRACES_SAMPLER_ARG`) | `parentbased_always_on` | |
| `OTEL_SERVICE_NAME` | `helmfile` | |
| `OTEL_RESOURCE_ATTRIBUTES` | — | e.g. `cicd.pipeline=deploy,cicd.run_id=4821` |
| `OTEL_PROPAGATORS` | `tracecontext,baggage` | |
| `OTEL_SDK_DISABLED` | `false` | standard kill switch |

HTTP proxies (`HTTPS_PROXY`/`HTTP_PROXY`) are honored automatically by the Go runtime.

## Quick start without a collector

To inspect what would be exported, print spans as JSON on stdout:

```bash
OTEL_TRACES_EXPORTER=console helmfile --otel-tracing -l name=myrelease template
```

## Quick start with Jaeger

```bash
docker run --rm -p 16686:16686 -p 4318:4318 jaegertracing/all-in-one:latest

OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318 \
  helmfile --otel-tracing -e production apply
# Open http://localhost:16686 and pick service "helmfile"
```

## What gets traced

The root span is named after the command (`helmfile apply`, `helmfile sync`, ...) and carries `helmfile.command`, `helmfile.file`, `helmfile.environment`, and `helmfile.selectors` attributes, plus `helmfile.exit_code` and the error (if any) at the end.

Every external process helmfile starts — each helm invocation, hook command, and plugin exec — gets its own span nested under the command span: `helm.exec` (with `helm.subcommand`) for helm binaries, `os.exec` otherwise, both carrying `exec.command`, redacted `exec.args`, and `exec.exit_code` on failure.

State loading is traced too: `helmfile.discover_states`, one `helmfile.load` per state file (including nested helmfiles), with `helmfile.render` and `helmfile.parse` children — rendering is frequently the hidden time sink. Each hook execution produces a `helmfile.hook` span (with `hook.event` and `hook.name`) that its subprocess span nests under — and release-scoped hooks (presync/postsync & co. within sync/diff/delete) nest under their release's span, keeping full release attribution.

Per-release spans (`helmfile.release.sync` / `.diff` / `.delete` / `.status` / `.test` / `.prepare`) carry the release name, namespace, chart, and labels, and nest under the state-file load span — with the release's helm subprocesses (upgrade, diff, delete, status, test) nested under the release span. The remaining loops (flag preparation, template/lint/unittest) are being added incrementally; see [the design proposal](proposals/otel-tracing.md) for the full taxonomy.

Secret-bearing command arguments (`--set`, `--set-file`, `--username`, `--password`, ...) are never recorded in span attributes — they are masked before export.

## Metrics

Tracing and metrics share the same switch and resource; metrics are exported through the standard metrics environment variables:

| Variable | Default | Notes |
|---|---|---|
| `OTEL_METRICS_EXPORTER` | `otlp` | `otlp` \| `console` \| `prometheus` \| `none` |
| `OTEL_METRIC_EXPORT_INTERVAL` | `60000` (ms) | periodic export interval; the final flush happens at exit |

Three instruments are emitted:

| Metric | Type | Attributes |
|---|---|---|
| `helmfile.helm.exec.duration` | histogram (seconds, buckets tuned for 5ms–600s helm invocations) | `subcommand`, `success` |
| `helmfile.release.duration` | histogram (seconds, same buckets) | `verb`, `result` — plus `helmfile.release` and `helmfile.namespace` when per-release metrics are enabled (below) |
| `helmfile.release.count` | counter (unit `{release}`) | `verb` (sync/diff/delete/status/test/prepare), `result` (success/error) |

### Per-release metrics (opt-in, high cardinality)

By default metric dimensions are bounded enumerations — release names never appear, so time-series count stays constant regardless of fleet size. If you need per-release durations on dashboards, opt in:

```bash
HELMFILE_OTEL_METRICS_PER_RELEASE=true helmfile --otel-tracing sync
```

This adds `helmfile.release` and `helmfile.namespace` to `helmfile.release.duration`. Time-series count then scales with your release fleet: well-suited to bounded CI runs, but long-lived centralized collection needs a backend capacity/TTL story. (Per-release timing is always available in traces, without this flag.)

Both dimensions are bounded enumerations — never release names — so metric cardinality stays constant regardless of fleet size. The instrumentation scope (`helmfile`) carries the helmfile version.

Inspect them without a collector with `OTEL_METRICS_EXPORTER=console`.

## CI trace correlation

If your CI system injects a W3C `TRACEPARENT` environment variable into jobs, helmfile joins that trace automatically, so helmfile spans appear inside your pipeline's trace.

## Troubleshooting

- **No spans arrive**: check `OTEL_EXPORTER_OTLP_ENDPOINT`/protocol match your collector (4318 for `http/protobuf`, 4317 for `grpc`); verify with `OTEL_TRACES_EXPORTER=console`.
- **Tracing breaks nothing**: exporter/sampler misconfiguration and export failures never fail a helmfile run; at worst you get no spans.
- **Still stuck?** Set `OTEL_SDK_DISABLED=false` explicitly and rerun with `--log-level debug`.