mirror of
https://github.com/helmfile/helmfile.git
synced 2026-09-30 14:01:46 +02:00
yxxhero
16259008d5
* feat(telemetry): add opt-in OpenTelemetry tracing (PR 1: lifecycle + root span) Implements the first increment of docs/proposals/otel-tracing.md (#2767): - pkg/telemetry: SDK setup from standard OTEL_* env vars (autoexport for exporter selection, env-driven sampler/propagators, OTEL_SDK_DISABLED), command-span lifecycle, no-op-by-default accessors - --otel-tracing flag / HELMFILE_OTEL_TRACING env switch - root span "helmfile <command>" with file/environment/selectors/exit_code attributes; TRACEPARENT-based remote-parent extraction for CI correlation - shutdown flush on both normal-exit and signal paths (nil-safe, 5s bound) - app.New derives its context from telemetry.CommandContext() (Background-identical when tracing is disabled) - docs: otel.md user guide, experimental-features entry, design proposal - tests: hermetic unit tests, app context-contract pinning, flag registration Telemetry problems never fail a run: exporter misconfiguration and export errors degrade to disabled with a warning. When disabled, behavior and performance are identical to before (no-op tracer, no goroutines, no network). Refs: #2767, #2758 Signed-off-by: yxxhero <aiopsclub@163.com> * feat(telemetry): trace every external process + trace-context bridges (PR 2) Implements the second increment of docs/proposals/otel-tracing.md (#2767): - pkg/helmexec/span.go: one span per external process started by helmfile (helm invocations, hooks, plugin execs) at the ShellRunner choke point — helm.exec (with helm.subcommand) vs os.exec, with redacted exec.args, exec.exit_code, and error status on failure - pkg/helmexec/redact.go: shared argument redaction with two profiles; legacy is byte-identical to the historical exit-error behavior (existing goldens unchanged), strict (spans) additionally covers --set=k=v and credential flags; exit_error.go now uses the shared helper - orphan-trace bridges with bit-identical cancellation semantics (context.WithoutCancel of the command context): both kubedog call sites (state.go) and hook execution (event.Bus gains an optional Ctx consumed by its default runner; state.go sets it, nil falls back to TODO as before) - OTLP end-to-end test (in-process httptest receiver, no external collector): span export, error status/exit code, redaction, and parent-linkage to the command span - docs/otel.md updated to the now-traced surface Verified end-to-end with the console exporter: helmfile template on a local chart yields the command span plus helm.exec spans for helm version/dependency/template, all nested under it. Refs: #2767 Signed-off-by: yxxhero <aiopsclub@163.com> * feat(telemetry): state-loading and hook spans (PR 3a) Implements the third increment of docs/proposals/otel-tracing.md (#2767): - helmfile.discover_states around findDesiredStateFiles and helmfile.load around loadDesiredStateFromYamlWithBaseDir; both cover all callers (incl. nested helmfiles) with no signature changes - helmfile.render / helmfile.parse children per document part, parented through a traceCtx field on the unexported desiredStateLoader struct (set once at its single construction site) - helmfile.hook span per hook execution: Trigger's per-hook body extracted into runHook (readability win on its own), the hook's subprocess span nests under it via a per-hook ctx-swapped ShellRunner clone (cancellation unchanged — Bus.Ctx never carries cancellation by contract) - pkg/telemetry/otlptest: shared in-process OTLP/HTTP receiver harness, now used by helmexec, event, and app span tests - golden span-tree test at the app layer (root -> discover -> load -> render/parse, via the exectest fake helm) and a hook-span nesting test - nil-ctx guard for App literals built directly by tests (App.spanParentCtx) Verified end-to-end with the console exporter: a template run over a gotmpl state file with a prepare hook yields the full tree with the hook's os.exec nested under helmfile.hook. Refs: #2767 Signed-off-by: yxxhero <aiopsclub@163.com> * feat(telemetry): per-release spans nested under the load span (PR 3b) Implements the per-release increment of docs/proposals/otel-tracing.md (#2767) — spans nest command -> load -> release -> helm exec: - helmexec.HelmContext gains an optional Ctx carrying the per-release span context; the execer's new execWithContext funnel consumes it via a per-call runner clone (runnerWithCtx) so the shared, cached execer is never mutated across concurrent workers. The seven Interface methods that take a HelmContext (Sync/Diff/ReleaseStatus/List/DecryptSecret/ Delete/Test) route through it; nil Ctx behaves exactly as before. exec() lost its always-nil override parameter on the way (unparam). - pkg/state/span.go: SetTraceContext + startReleaseSpan/endReleaseSpan helpers (release/namespace/chart/labels attributes, sorted for stable output); a typed-nil guard (releaseErrAsError) avoids the classic nil-pointer-in-interface trap on *ReleaseError. - release spans in the worker loops: SyncReleases, DiffReleases, DeleteReleasesForSync, PrepareCharts, and iterateOnReleases (status/ delete/test via a new verb parameter); their HelmContext is stamped with the release span context where one is built. - pkg/app sets st.SetTraceContext(loadCtx) right after loading a state file, rooting all per-release spans under helmfile.load. - bridged one more detached tracking call found on the way (trackReleaseIfEnabled's context.Background in the sync worker). - golden test: release span present, nested under load, correct attributes; unit test for the runnerWithCtx clone semantics. Verified with the console exporter: helmfile template yields release.prepare(demo) under load with full attributes. Refs: #2767 Signed-off-by: yxxhero <aiopsclub@163.com> * feat(telemetry): nest status/delete/test execs under their release spans Completes the per-release exec nesting for the iterateOnReleases-based loops (docs/proposals/otel-tracing.md §4.4 phase 2): the do closures now receive the release span context and stamp it into their HelmContext, so helm status/delete/test subprocess spans nest under helmfile.release.<verb> like sync/diff already did. - scatterGatherReleases/iterateOnReleases/doWithReleaseSpan: do gains a context parameter (the release span context) - ReleaseStatuses/DeleteReleases/TestReleases closures stamp HelmContext.Ctx from it - integration test with a real execer (version-probe shim binary): the release's status subprocess nests under helmfile.release.status, same trace, with helm.subcommand=status This also makes the otel.md claim ("upgrade, diff, delete, status, test nested under the release span") fully accurate. Refs: #2767 Signed-off-by: yxxhero <aiopsclub@163.com> * feat(telemetry): OTel metrics — helm exec duration and release results (PR 4) Implements the metrics increment of docs/proposals/otel-tracing.md (#2767) on the same provider, switch, and resource as traces: - pkg/telemetry/metrics.go: helmfile.helm.exec.duration histogram (subcommand, success) and helmfile.release.count counter (verb, result). Instruments come from the otel global meter, so recording at call sites is branch-free no-op when telemetry is disabled. - Setup builds the resource once and installs both providers; reader selection delegates to autoexport (OTEL_METRICS_EXPORTER: otlp | console | prometheus | none), the OTLP reader's interval honors OTEL_METRIC_EXPORT_INTERVAL (read by the SDK). Shutdown flushes both providers (errors.Join). StartCommandSpan now carries the meter provider across state transitions (fixes a nil-shutdown panic). - helmexec: finishExecSpan records exec duration for helm binaries; state: endReleaseSpan counts release outcomes for sync/diff/delete/ status/test/prepare (diff counted as success when no hard error). - otlptest: recorder routes by OTLP path (/v1/traces vs /v1/metrics) and decodes metrics; new FindMetric helper. - tests: metrics recorded as no-op when disabled, provider enabled with the none exporter, degradation on an invalid metrics exporter, and an integration assertion (status exec duration datapoint + one successful release.count) in the shim-based state test. Verified with the console exporter: helmfile template emits helmfile.helm.exec.duration per subcommand (version/dependency/ template) and helmfile.release.count{verb=prepare,result=success}=1. Refs: #2767 Signed-off-by: yxxhero <aiopsclub@163.com> * docs: complete OTel documentation coverage - docs/cli.md: --otel-tracing in the CLI reference help block (verbatim from the cobra output) - CHANGELOG.md: [Unreleased] Added entry for tracing + metrics - docs/index.md: Observability highlight linking docs/otel.md - docs/proposals/otel-tracing.md: add OTEL_METRICS_EXPORTER / OTEL_METRIC_EXPORT_INTERVAL rows to the env-var table and note the periodic reader + bounded metric cardinality in §7 Refs: #2767 Signed-off-by: yxxhero <aiopsclub@163.com> * fix: drop unused id parameter from parsePart (unparam) The id parameter was never used inside the span wrapper; the caller's id variable is still used for the render calls and error messages. Refs: #2769 Signed-off-by: yxxhero <aiopsclub@163.com> * fix(telemetry): address review — redaction gaps, kubedog valve, phantom metrics Addresses all Copilot review comments on #2769: Security (span payloads): - exec.args: positional arguments are additionally passed through helmexec.RedactedURL, so credentials embedded in chart/repository URLs (AddRepo, RegistryLogin, OCI refs) are masked exactly like log output - release spans sanitize helmfile.chart the same way - error statuses no longer embed raw errors (which contain rendered commands, arguments, and subprocess output): the command span, release spans, hook spans, and exec spans now use generic descriptions; the concrete exit code remains an attribute, and RecordError on the root span is dropped Correctness: - kubedog safety valve restored: execWithContext now attaches the per-release span into the runner's own context instead of replacing it, so trackHandle.Cancel() can interrupt a wedged helm again and app cancellation semantics stay exactly as before the PR - diff release spans/metrics: real failures are recorded (exit code 2 "changes detected" still counts as success); previously every diff was exported as successful - skipped releases no longer emit phantom spans and inflate helmfile.release.count: iterateOnReleases callers pass a skip predicate (skipUndesired for status/test; delete deletes undesired releases and passes nil) - Setup shuts down the already-constructed tracer provider (bounded) when the metrics provider fails, instead of abandoning its batch goroutine Tests: URL redaction cases (masked/untouched), spanAttachedContext preserves the runner cancellation chain while attaching the caller's span, skipUndesired, and a failing-hook span asserting the generic message. Refs: #2769 Signed-off-by: yxxhero <aiopsclub@163.com> * fix: lint — restore nolint placement and avoid nil context literal - the skipUndesired insertion had displaced the // nolint: unparam directive off iterateOnReleases (helm param is intentionally unused there); also fixes a skipDesired/skipUndesired comment typo - use a typed nil in TestSpanAttachedContext (staticcheck SA1012) Refs: #2769 Signed-off-by: yxxhero <aiopsclub@163.com> * fix(telemetry): address review round 2 — remote-ref redaction, wrapper helm binaries, hook release attribution Addresses all 6 new review comments on #2769: Security (remote references): - new helmexec.RedactedRef sanitizes go-getter style references for telemetry: forced-form prefixes (git::, s3::) preserved, whole URL userinfo masked (usernames carry tokens too), credential-bearing query parameters masked using pkg/remote's heuristic (token/password/secret/ key/signature). Applied to helmfile.file (command span), helmfile.path (discover_states), helmfile.chart (release spans), and exec.args — log-time RedactedURL is untouched so log output is unchanged Correctness: - wrapper helm binaries (--helm-binary custom names) are now classified as helm operations by an explicit context marker stamped in the execer funnel, instead of the executable-basename heuristic; the same classification gates helmfile.helm.exec.duration, so the metric no longer misses wrapper invocations (classifyExec) - release-scoped hooks (presync/postsync/preuninstall/postuninstall/ cleanup in the sync/delete/diff workers) now attach their helmfile.hook spans to the active helmfile.release.* span via a variadic parent on the trigger functions; global hooks keep the command context and all 29 existing call sites compile unchanged; hook cancellation stays detached (WithoutCancel) as before - signal-terminated runs (Shutdown with exitCode 130/143 and nil error) now mark the command span with error status, consistent with their nonzero exit code Tests: RedactedRef table (forced forms, userinfo, s3/token query params, untouched cases), classifyExec marker case, hookTraceContext parent attribution + non-cancellability + fallback. Refs: #2769 Signed-off-by: yxxhero <aiopsclub@163.com> * fix(telemetry): address review round 3 — redaction corner cases, value runners Addresses 5 of the 6 new review comments on #2769 (the sixth — an unused strings import in exit_error.go — is a false positive: Indent still uses strings.Split/Builder and the package compiles): - RedactArgs read the previous token from the progressively redacted output, so {--set, --set-string, secret} leaked the secret (the masked value hid the following flag). Read the previous token from the original input, restoring the legacy contract for adjacent secret flags - RedactedRef fails closed for malformed references: URL-like refs with invalid percent escapes export a fully redacted value, and an unparseable query is dropped entirely instead of exported verbatim - ShellRunner has value receivers, so a ShellRunner VALUE satisfies the Runner API; the helm marker stamping and the per-release span attachment now handle both value and pointer forms (matching WithContext), so value-runner callers keep release nesting and the helm.exec classification/metric Regression tests: adjacent secret flags (legacy + strict), malformed URL-like ref, malformed query, value-runner marker + span attachment. Refs: #2769 Signed-off-by: yxxhero <aiopsclub@163.com> * fix(telemetry): stamp the helm marker on the stdin funnel too execStdIn (registry login, repo add) called the runner directly, so wrapper --helm-binary names were misclassified as os.exec and omitted from helmfile.helm.exec.duration on that path. The marking now goes through a shared markHelmRunner helper (value and pointer ShellRunner forms) used by both execution funnels. Refs: #2769 Signed-off-by: yxxhero <aiopsclub@163.com> * fix(telemetry): redact helm's --kube-token in strict profile Helm's global --kube-token carries a bearer token; both the two-argument and inline forms are now masked in span exec.args (legacy exit-error output is untouched, matching its historical behavior). Refs: #2769 Signed-off-by: yxxhero <aiopsclub@163.com> * fix(telemetry): OTel metrics best-practice alignment - helmfile.helm.exec.duration now declares explicit bucket boundaries tuned for seconds-scale helm invocations (5ms…600s); the SDK defaults are millisecond-oriented and lumped every sub-5s invocation — the common case — into the first bucket, defeating the histogram - instruments are re-created under the installed provider with the instrumentation scope version stamped (Setup-time, race-free) - helmfile.release.count declares the {release} curly-annotation unit per the metrics naming conventions Tested end-to-end via the OTLP integration test: exported bounds are the tuned set, units are asserted, and the scope carries the version. Refs: #2769 Signed-off-by: yxxhero <aiopsclub@163.com> * feat(telemetry): per-release duration metrics behind an opt-in switch New helmfile.release.duration histogram (seconds, same tuned buckets) with bounded dimensions by default (verb, result). Setting HELMFILE_OTEL_METRICS_PER_RELEASE=true adds helmfile.release and helmfile.namespace, answering "which release is slow" from dashboards: - well-suited to bounded CI runs; long-lived centralized collection needs a backend capacity/TTL story (documented in docs/otel.md) - per-release timing remains available in traces without the flag - env read per call (release operations are low-frequency, and tests toggle it) endReleaseSpan now takes the release and the operation start time; the five worker-loop call sites pass them (doWithReleaseSpan, SyncReleases, DeleteReleasesForSync, PrepareCharts, DiffReleases). Verified end-to-end with the console exporter (default dims vs per-release) and OTLP integration tests pinning both modes. Refs: #2767, #2769 Signed-off-by: yxxhero <aiopsclub@163.com> * refactor(telemetry): maintainability pass over the runner/metric plumbing - StartCommandSpan copies the tracingState struct instead of enumerating fields by hand — that pattern dropped the meter provider once already - the two value/pointer ShellRunner switches (helm marker, span attachment) are unified into one withRunnerCtx helper; the duplication caused two review rounds of value-form misses - classifyExec derives the helm classification from the span name (helmExecSpanName constant) instead of returning a third parallel bool - metrics: shared outcomeAttrs for the verb/result dimensions, and the bucket slice renamed to durationBuckets with a comment covering both histograms that use it No behavior change; full -race suite green, lint clean. Refs: #2769 Signed-off-by: yxxhero <aiopsclub@163.com> * refactor(telemetry): consolidate test env lists, trace bridges, and hook prep; sync the design doc Maintainability: - HermeticEnvVars is now exported from pkg/telemetry (the owner of the env surface) and used by both telemetry tests and otlptest — the two copies had already drifted once (HELMFILE_OTEL_METRICS_PER_RELEASE needed updating in both) - kubedogTraceContext and hookTraceContext were the same concept written twice; unified into traceOnlyContext(parent...) in span.go Readability: - runHook's nested kubectl rewrite extracted into prepareKubectlHook with guard-clause structure Accuracy (docs ↔ code, drifted over five review rounds): - §4.4 now describes the implemented mechanism: the release span is INJECTED into the runner's own context (preserving the kubedog safety valve) rather than the runner context being replaced, and helm classification is marker-based for wrapper binaries - §5 exec span rows list the actual attributes incl. URL/query masking - §6 strict profile documents RedactedRef, --kube-token, and the adjacent-token guarantee No behavior change; full -race suite green, lint clean. Refs: #2769 Signed-off-by: yxxhero <aiopsclub@163.com> * refactor(telemetry): drop the dead noop state, relocate skipUndesired, sync user-facing accuracy - tracingState.noop was dead weight in the enabled state and a copy-surface in every transition; a single package-level noopTracerProvider now backs Tracer while disabled - skipUndesired moved next to doWithReleaseSpan in span.go, its only conceptual home (span/metric suppression, not run plumbing) - accuracy: the package doc, --otel-tracing flag help, experimental-features entry, and CHANGELOG now all say tracing AND metrics and list the third instrument (helmfile.release.duration with the HELMFILE_OTEL_METRICS_PER_RELEASE opt-in) — these had drifted when the metric was added; the PR description's metric table is updated to match as well No behavior change; full -race suite green (except the pre-existing network-dependent TestStorage_resolveFile flake), lint clean. Refs: #2769 Signed-off-by: yxxhero <aiopsclub@163.com> * refactor(telemetry): flatten Setup, name the prefix bound, dedupe test fake; fix instrument-count drift Readability/maintainability: - Setup drops from 56 to 39 lines: provider construction (including the shutdown-tracer-on-meter-failure recovery) moves to newProviders in exporter.go next to the constructors it composes - refredact's magic 16 becomes maxForcedFormPrefix with a comment - span_test's hand-rolled fakeRunner removed in favor of the existing mockRunner (same package) Accuracy: - "Two instruments" wording survived in docs/otel.md and the design proposal §7 after helmfile.release.duration was added; both now say three and mention the per-release opt-in No behavior change; full -race suite green (except the pre-existing network flake), lint clean. Refs: #2769 Signed-off-by: yxxhero <aiopsclub@163.com> * refactor(telemetry): co-locate span machinery, drop a dead export, fix docs nits - the span plumbing helpers (markHelmExec, withRunnerCtx, markHelmRunner, spanAttachedContext) move from exec.go to span.go, next to the marker type and classifiers they serve — exec.go keeps only the funnel call sites - otlptest.SpanNames was never used outside the package; unexported - isHelmBinary's comment now states it is the FALLBACK classifier (funnel invocations are marker-classified), replacing the outdated "cosmetic distinction" framing from before the marker existed - docs/otel.md: release-scoped hooks nest under their release span (added in review round 2, never documented) No behavior change; full -race suite green, lint clean. Refs: #2769 Signed-off-by: yxxhero <aiopsclub@163.com> --------- Signed-off-by: yxxhero <aiopsclub@163.com>
111 lines
6.5 KiB
Markdown
111 lines
6.5 KiB
Markdown
# OpenTelemetry Tracing
|
||
|
||
Helmfile can export [OpenTelemetry](https://opentelemetry.io/) traces of a run, which is especially useful in CI/CD: see where time is spent, which helm invocations dominate, how label selectors fan out, and what to target for improvement.
|
||
|
||
Tracing is **experimental** and **off by default**. It currently emits the command-level root span, state-loading spans (discover/load/render/parse), one span per external process (helm invocations, hooks, plugin execs), per-hook spans, and per-release spans for the sync/diff/delete/status/test/prepare paths.
|
||
|
||
## Enabling
|
||
|
||
```bash
|
||
helmfile --otel-tracing -e production -l tier=backend apply
|
||
```
|
||
|
||
or via environment variable (handy for CI):
|
||
|
||
```bash
|
||
export HELMFILE_OTEL_TRACING=true
|
||
helmfile sync
|
||
```
|
||
|
||
`--otel-tracing` overrides `HELMFILE_OTEL_TRACING` when specified.
|
||
|
||
## Configuration
|
||
|
||
Everything beyond the on/off switch uses the standard [OpenTelemetry environment variables](https://opentelemetry.io/docs/specs/otel/configuration/sdk-environment-variables/); helmfile defines no telemetry-specific variables of its own.
|
||
|
||
| Variable | Default | Notes |
|
||
|---|---|---|
|
||
| `OTEL_EXPORTER_OTLP_ENDPOINT` | `http://localhost:4318` | OTLP endpoint; use `http://host:4317` for gRPC |
|
||
| `OTEL_EXPORTER_OTLP_PROTOCOL` | `http/protobuf` | or `grpc` |
|
||
| `OTEL_EXPORTER_OTLP_HEADERS` | — | e.g. `authorization=Bearer <token>` |
|
||
| `OTEL_TRACES_EXPORTER` | `otlp` | `otlp` \| `console` \| `none` |
|
||
| `OTEL_METRICS_EXPORTER` | `otlp` | `otlp` \| `console` \| `prometheus` \| `none` — see [Metrics](#metrics) |
|
||
| `OTEL_TRACES_SAMPLER` (+ `OTEL_TRACES_SAMPLER_ARG`) | `parentbased_always_on` | |
|
||
| `OTEL_SERVICE_NAME` | `helmfile` | |
|
||
| `OTEL_RESOURCE_ATTRIBUTES` | — | e.g. `cicd.pipeline=deploy,cicd.run_id=4821` |
|
||
| `OTEL_PROPAGATORS` | `tracecontext,baggage` | |
|
||
| `OTEL_SDK_DISABLED` | `false` | standard kill switch |
|
||
|
||
HTTP proxies (`HTTPS_PROXY`/`HTTP_PROXY`) are honored automatically by the Go runtime.
|
||
|
||
## Quick start without a collector
|
||
|
||
To inspect what would be exported, print spans as JSON on stdout:
|
||
|
||
```bash
|
||
OTEL_TRACES_EXPORTER=console helmfile --otel-tracing -l name=myrelease template
|
||
```
|
||
|
||
## Quick start with Jaeger
|
||
|
||
```bash
|
||
docker run --rm -p 16686:16686 -p 4318:4318 jaegertracing/all-in-one:latest
|
||
|
||
OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318 \
|
||
helmfile --otel-tracing -e production apply
|
||
# Open http://localhost:16686 and pick service "helmfile"
|
||
```
|
||
|
||
## What gets traced
|
||
|
||
The root span is named after the command (`helmfile apply`, `helmfile sync`, ...) and carries `helmfile.command`, `helmfile.file`, `helmfile.environment`, and `helmfile.selectors` attributes, plus `helmfile.exit_code` and the error (if any) at the end.
|
||
|
||
Every external process helmfile starts — each helm invocation, hook command, and plugin exec — gets its own span nested under the command span: `helm.exec` (with `helm.subcommand`) for helm binaries, `os.exec` otherwise, both carrying `exec.command`, redacted `exec.args`, and `exec.exit_code` on failure.
|
||
|
||
State loading is traced too: `helmfile.discover_states`, one `helmfile.load` per state file (including nested helmfiles), with `helmfile.render` and `helmfile.parse` children — rendering is frequently the hidden time sink. Each hook execution produces a `helmfile.hook` span (with `hook.event` and `hook.name`) that its subprocess span nests under — and release-scoped hooks (presync/postsync & co. within sync/diff/delete) nest under their release's span, keeping full release attribution.
|
||
|
||
Per-release spans (`helmfile.release.sync` / `.diff` / `.delete` / `.status` / `.test` / `.prepare`) carry the release name, namespace, chart, and labels, and nest under the state-file load span — with the release's helm subprocesses (upgrade, diff, delete, status, test) nested under the release span. The remaining loops (flag preparation, template/lint/unittest) are being added incrementally; see [the design proposal](proposals/otel-tracing.md) for the full taxonomy.
|
||
|
||
Secret-bearing command arguments (`--set`, `--set-file`, `--username`, `--password`, ...) are never recorded in span attributes — they are masked before export.
|
||
|
||
## Metrics
|
||
|
||
Tracing and metrics share the same switch and resource; metrics are exported through the standard metrics environment variables:
|
||
|
||
| Variable | Default | Notes |
|
||
|---|---|---|
|
||
| `OTEL_METRICS_EXPORTER` | `otlp` | `otlp` \| `console` \| `prometheus` \| `none` |
|
||
| `OTEL_METRIC_EXPORT_INTERVAL` | `60000` (ms) | periodic export interval; the final flush happens at exit |
|
||
|
||
Three instruments are emitted:
|
||
|
||
| Metric | Type | Attributes |
|
||
|---|---|---|
|
||
| `helmfile.helm.exec.duration` | histogram (seconds, buckets tuned for 5ms–600s helm invocations) | `subcommand`, `success` |
|
||
| `helmfile.release.duration` | histogram (seconds, same buckets) | `verb`, `result` — plus `helmfile.release` and `helmfile.namespace` when per-release metrics are enabled (below) |
|
||
| `helmfile.release.count` | counter (unit `{release}`) | `verb` (sync/diff/delete/status/test/prepare), `result` (success/error) |
|
||
|
||
### Per-release metrics (opt-in, high cardinality)
|
||
|
||
By default metric dimensions are bounded enumerations — release names never appear, so time-series count stays constant regardless of fleet size. If you need per-release durations on dashboards, opt in:
|
||
|
||
```bash
|
||
HELMFILE_OTEL_METRICS_PER_RELEASE=true helmfile --otel-tracing sync
|
||
```
|
||
|
||
This adds `helmfile.release` and `helmfile.namespace` to `helmfile.release.duration`. Time-series count then scales with your release fleet: well-suited to bounded CI runs, but long-lived centralized collection needs a backend capacity/TTL story. (Per-release timing is always available in traces, without this flag.)
|
||
|
||
Both dimensions are bounded enumerations — never release names — so metric cardinality stays constant regardless of fleet size. The instrumentation scope (`helmfile`) carries the helmfile version.
|
||
|
||
Inspect them without a collector with `OTEL_METRICS_EXPORTER=console`.
|
||
|
||
## CI trace correlation
|
||
|
||
If your CI system injects a W3C `TRACEPARENT` environment variable into jobs, helmfile joins that trace automatically, so helmfile spans appear inside your pipeline's trace.
|
||
|
||
## Troubleshooting
|
||
|
||
- **No spans arrive**: check `OTEL_EXPORTER_OTLP_ENDPOINT`/protocol match your collector (4318 for `http/protobuf`, 4317 for `grpc`); verify with `OTEL_TRACES_EXPORTER=console`.
|
||
- **Tracing breaks nothing**: exporter/sampler misconfiguration and export failures never fail a helmfile run; at worst you get no spans.
|
||
- **Still stuck?** Set `OTEL_SDK_DISABLED=false` explicitly and rerun with `--log-level debug`.
|