* feat(telemetry): add opt-in OpenTelemetry tracing (PR 1: lifecycle + root span)
Implements the first increment of docs/proposals/otel-tracing.md (#2767):
- pkg/telemetry: SDK setup from standard OTEL_* env vars (autoexport for
exporter selection, env-driven sampler/propagators, OTEL_SDK_DISABLED),
command-span lifecycle, no-op-by-default accessors
- --otel-tracing flag / HELMFILE_OTEL_TRACING env switch
- root span "helmfile <command>" with file/environment/selectors/exit_code
attributes; TRACEPARENT-based remote-parent extraction for CI correlation
- shutdown flush on both normal-exit and signal paths (nil-safe, 5s bound)
- app.New derives its context from telemetry.CommandContext()
(Background-identical when tracing is disabled)
- docs: otel.md user guide, experimental-features entry, design proposal
- tests: hermetic unit tests, app context-contract pinning, flag registration
Telemetry problems never fail a run: exporter misconfiguration and export
errors degrade to disabled with a warning. When disabled, behavior and
performance are identical to before (no-op tracer, no goroutines, no
network).
Refs: #2767, #2758
Signed-off-by: yxxhero <aiopsclub@163.com>
* feat(telemetry): trace every external process + trace-context bridges (PR 2)
Implements the second increment of docs/proposals/otel-tracing.md (#2767):
- pkg/helmexec/span.go: one span per external process started by helmfile
(helm invocations, hooks, plugin execs) at the ShellRunner choke point —
helm.exec (with helm.subcommand) vs os.exec, with redacted exec.args,
exec.exit_code, and error status on failure
- pkg/helmexec/redact.go: shared argument redaction with two profiles;
legacy is byte-identical to the historical exit-error behavior (existing
goldens unchanged), strict (spans) additionally covers --set=k=v and
credential flags; exit_error.go now uses the shared helper
- orphan-trace bridges with bit-identical cancellation semantics
(context.WithoutCancel of the command context): both kubedog call sites
(state.go) and hook execution (event.Bus gains an optional Ctx consumed
by its default runner; state.go sets it, nil falls back to TODO as before)
- OTLP end-to-end test (in-process httptest receiver, no external
collector): span export, error status/exit code, redaction, and
parent-linkage to the command span
- docs/otel.md updated to the now-traced surface
Verified end-to-end with the console exporter: helmfile template on a
local chart yields the command span plus helm.exec spans for helm
version/dependency/template, all nested under it.
Refs: #2767
Signed-off-by: yxxhero <aiopsclub@163.com>
* feat(telemetry): state-loading and hook spans (PR 3a)
Implements the third increment of docs/proposals/otel-tracing.md (#2767):
- helmfile.discover_states around findDesiredStateFiles and helmfile.load
around loadDesiredStateFromYamlWithBaseDir; both cover all callers
(incl. nested helmfiles) with no signature changes
- helmfile.render / helmfile.parse children per document part, parented
through a traceCtx field on the unexported desiredStateLoader struct
(set once at its single construction site)
- helmfile.hook span per hook execution: Trigger's per-hook body extracted
into runHook (readability win on its own), the hook's subprocess span
nests under it via a per-hook ctx-swapped ShellRunner clone
(cancellation unchanged — Bus.Ctx never carries cancellation by contract)
- pkg/telemetry/otlptest: shared in-process OTLP/HTTP receiver harness,
now used by helmexec, event, and app span tests
- golden span-tree test at the app layer (root -> discover -> load ->
render/parse, via the exectest fake helm) and a hook-span nesting test
- nil-ctx guard for App literals built directly by tests (App.spanParentCtx)
Verified end-to-end with the console exporter: a template run over a
gotmpl state file with a prepare hook yields the full tree with the hook's
os.exec nested under helmfile.hook.
Refs: #2767
Signed-off-by: yxxhero <aiopsclub@163.com>
* feat(telemetry): per-release spans nested under the load span (PR 3b)
Implements the per-release increment of docs/proposals/otel-tracing.md
(#2767) — spans nest command -> load -> release -> helm exec:
- helmexec.HelmContext gains an optional Ctx carrying the per-release span
context; the execer's new execWithContext funnel consumes it via a
per-call runner clone (runnerWithCtx) so the shared, cached execer is
never mutated across concurrent workers. The seven Interface methods
that take a HelmContext (Sync/Diff/ReleaseStatus/List/DecryptSecret/
Delete/Test) route through it; nil Ctx behaves exactly as before.
exec() lost its always-nil override parameter on the way (unparam).
- pkg/state/span.go: SetTraceContext + startReleaseSpan/endReleaseSpan
helpers (release/namespace/chart/labels attributes, sorted for stable
output); a typed-nil guard (releaseErrAsError) avoids the classic
nil-pointer-in-interface trap on *ReleaseError.
- release spans in the worker loops: SyncReleases, DiffReleases,
DeleteReleasesForSync, PrepareCharts, and iterateOnReleases (status/
delete/test via a new verb parameter); their HelmContext is stamped with
the release span context where one is built.
- pkg/app sets st.SetTraceContext(loadCtx) right after loading a state
file, rooting all per-release spans under helmfile.load.
- bridged one more detached tracking call found on the way
(trackReleaseIfEnabled's context.Background in the sync worker).
- golden test: release span present, nested under load, correct
attributes; unit test for the runnerWithCtx clone semantics.
Verified with the console exporter: helmfile template yields
release.prepare(demo) under load with full attributes.
Refs: #2767
Signed-off-by: yxxhero <aiopsclub@163.com>
* feat(telemetry): nest status/delete/test execs under their release spans
Completes the per-release exec nesting for the iterateOnReleases-based
loops (docs/proposals/otel-tracing.md §4.4 phase 2): the do closures now
receive the release span context and stamp it into their HelmContext, so
helm status/delete/test subprocess spans nest under
helmfile.release.<verb> like sync/diff already did.
- scatterGatherReleases/iterateOnReleases/doWithReleaseSpan: do gains a
context parameter (the release span context)
- ReleaseStatuses/DeleteReleases/TestReleases closures stamp
HelmContext.Ctx from it
- integration test with a real execer (version-probe shim binary): the
release's status subprocess nests under helmfile.release.status, same
trace, with helm.subcommand=status
This also makes the otel.md claim ("upgrade, diff, delete, status, test
nested under the release span") fully accurate.
Refs: #2767
Signed-off-by: yxxhero <aiopsclub@163.com>
* feat(telemetry): OTel metrics — helm exec duration and release results (PR 4)
Implements the metrics increment of docs/proposals/otel-tracing.md
(#2767) on the same provider, switch, and resource as traces:
- pkg/telemetry/metrics.go: helmfile.helm.exec.duration histogram
(subcommand, success) and helmfile.release.count counter (verb,
result). Instruments come from the otel global meter, so recording at
call sites is branch-free no-op when telemetry is disabled.
- Setup builds the resource once and installs both providers; reader
selection delegates to autoexport (OTEL_METRICS_EXPORTER: otlp |
console | prometheus | none), the OTLP reader's interval honors
OTEL_METRIC_EXPORT_INTERVAL (read by the SDK). Shutdown flushes both
providers (errors.Join). StartCommandSpan now carries the meter
provider across state transitions (fixes a nil-shutdown panic).
- helmexec: finishExecSpan records exec duration for helm binaries;
state: endReleaseSpan counts release outcomes for sync/diff/delete/
status/test/prepare (diff counted as success when no hard error).
- otlptest: recorder routes by OTLP path (/v1/traces vs /v1/metrics)
and decodes metrics; new FindMetric helper.
- tests: metrics recorded as no-op when disabled, provider enabled with
the none exporter, degradation on an invalid metrics exporter, and an
integration assertion (status exec duration datapoint + one successful
release.count) in the shim-based state test.
Verified with the console exporter: helmfile template emits
helmfile.helm.exec.duration per subcommand (version/dependency/
template) and helmfile.release.count{verb=prepare,result=success}=1.
Refs: #2767
Signed-off-by: yxxhero <aiopsclub@163.com>
* docs: complete OTel documentation coverage
- docs/cli.md: --otel-tracing in the CLI reference help block (verbatim
from the cobra output)
- CHANGELOG.md: [Unreleased] Added entry for tracing + metrics
- docs/index.md: Observability highlight linking docs/otel.md
- docs/proposals/otel-tracing.md: add OTEL_METRICS_EXPORTER /
OTEL_METRIC_EXPORT_INTERVAL rows to the env-var table and note the
periodic reader + bounded metric cardinality in §7
Refs: #2767
Signed-off-by: yxxhero <aiopsclub@163.com>
* fix: drop unused id parameter from parsePart (unparam)
The id parameter was never used inside the span wrapper; the caller's id
variable is still used for the render calls and error messages.
Refs: #2769
Signed-off-by: yxxhero <aiopsclub@163.com>
* fix(telemetry): address review — redaction gaps, kubedog valve, phantom metrics
Addresses all Copilot review comments on #2769:
Security (span payloads):
- exec.args: positional arguments are additionally passed through
helmexec.RedactedURL, so credentials embedded in chart/repository URLs
(AddRepo, RegistryLogin, OCI refs) are masked exactly like log output
- release spans sanitize helmfile.chart the same way
- error statuses no longer embed raw errors (which contain rendered
commands, arguments, and subprocess output): the command span, release
spans, hook spans, and exec spans now use generic descriptions; the
concrete exit code remains an attribute, and RecordError on the root
span is dropped
Correctness:
- kubedog safety valve restored: execWithContext now attaches the
per-release span into the runner's own context instead of replacing it,
so trackHandle.Cancel() can interrupt a wedged helm again and app
cancellation semantics stay exactly as before the PR
- diff release spans/metrics: real failures are recorded (exit code 2
"changes detected" still counts as success); previously every diff was
exported as successful
- skipped releases no longer emit phantom spans and inflate
helmfile.release.count: iterateOnReleases callers pass a skip predicate
(skipUndesired for status/test; delete deletes undesired releases and
passes nil)
- Setup shuts down the already-constructed tracer provider (bounded) when
the metrics provider fails, instead of abandoning its batch goroutine
Tests: URL redaction cases (masked/untouched), spanAttachedContext
preserves the runner cancellation chain while attaching the caller's
span, skipUndesired, and a failing-hook span asserting the generic
message.
Refs: #2769
Signed-off-by: yxxhero <aiopsclub@163.com>
* fix: lint — restore nolint placement and avoid nil context literal
- the skipUndesired insertion had displaced the // nolint: unparam
directive off iterateOnReleases (helm param is intentionally unused
there); also fixes a skipDesired/skipUndesired comment typo
- use a typed nil in TestSpanAttachedContext (staticcheck SA1012)
Refs: #2769
Signed-off-by: yxxhero <aiopsclub@163.com>
* fix(telemetry): address review round 2 — remote-ref redaction, wrapper helm binaries, hook release attribution
Addresses all 6 new review comments on #2769:
Security (remote references):
- new helmexec.RedactedRef sanitizes go-getter style references for
telemetry: forced-form prefixes (git::, s3::) preserved, whole URL
userinfo masked (usernames carry tokens too), credential-bearing query
parameters masked using pkg/remote's heuristic (token/password/secret/
key/signature). Applied to helmfile.file (command span), helmfile.path
(discover_states), helmfile.chart (release spans), and exec.args —
log-time RedactedURL is untouched so log output is unchanged
Correctness:
- wrapper helm binaries (--helm-binary custom names) are now classified
as helm operations by an explicit context marker stamped in the execer
funnel, instead of the executable-basename heuristic; the same
classification gates helmfile.helm.exec.duration, so the metric no
longer misses wrapper invocations (classifyExec)
- release-scoped hooks (presync/postsync/preuninstall/postuninstall/
cleanup in the sync/delete/diff workers) now attach their helmfile.hook
spans to the active helmfile.release.* span via a variadic parent on
the trigger functions; global hooks keep the command context and all
29 existing call sites compile unchanged; hook cancellation stays
detached (WithoutCancel) as before
- signal-terminated runs (Shutdown with exitCode 130/143 and nil error)
now mark the command span with error status, consistent with their
nonzero exit code
Tests: RedactedRef table (forced forms, userinfo, s3/token query params,
untouched cases), classifyExec marker case, hookTraceContext parent
attribution + non-cancellability + fallback.
Refs: #2769
Signed-off-by: yxxhero <aiopsclub@163.com>
* fix(telemetry): address review round 3 — redaction corner cases, value runners
Addresses 5 of the 6 new review comments on #2769 (the sixth — an
unused strings import in exit_error.go — is a false positive: Indent
still uses strings.Split/Builder and the package compiles):
- RedactArgs read the previous token from the progressively redacted
output, so {--set, --set-string, secret} leaked the secret
(the masked value hid the following flag). Read the previous token
from the original input, restoring the legacy contract for adjacent
secret flags
- RedactedRef fails closed for malformed references: URL-like refs with
invalid percent escapes export a fully redacted value, and an
unparseable query is dropped entirely instead of exported verbatim
- ShellRunner has value receivers, so a ShellRunner VALUE satisfies the
Runner API; the helm marker stamping and the per-release span
attachment now handle both value and pointer forms (matching
WithContext), so value-runner callers keep release nesting and the
helm.exec classification/metric
Regression tests: adjacent secret flags (legacy + strict), malformed
URL-like ref, malformed query, value-runner marker + span attachment.
Refs: #2769
Signed-off-by: yxxhero <aiopsclub@163.com>
* fix(telemetry): stamp the helm marker on the stdin funnel too
execStdIn (registry login, repo add) called the runner directly, so
wrapper --helm-binary names were misclassified as os.exec and omitted
from helmfile.helm.exec.duration on that path. The marking now goes
through a shared markHelmRunner helper (value and pointer ShellRunner
forms) used by both execution funnels.
Refs: #2769
Signed-off-by: yxxhero <aiopsclub@163.com>
* fix(telemetry): redact helm's --kube-token in strict profile
Helm's global --kube-token carries a bearer token; both the
two-argument and inline forms are now masked in span exec.args
(legacy exit-error output is untouched, matching its historical
behavior).
Refs: #2769
Signed-off-by: yxxhero <aiopsclub@163.com>
* fix(telemetry): OTel metrics best-practice alignment
- helmfile.helm.exec.duration now declares explicit bucket boundaries
tuned for seconds-scale helm invocations (5ms…600s); the SDK defaults
are millisecond-oriented and lumped every sub-5s invocation — the
common case — into the first bucket, defeating the histogram
- instruments are re-created under the installed provider with the
instrumentation scope version stamped (Setup-time, race-free)
- helmfile.release.count declares the {release} curly-annotation unit
per the metrics naming conventions
Tested end-to-end via the OTLP integration test: exported bounds are
the tuned set, units are asserted, and the scope carries the version.
Refs: #2769
Signed-off-by: yxxhero <aiopsclub@163.com>
* feat(telemetry): per-release duration metrics behind an opt-in switch
New helmfile.release.duration histogram (seconds, same tuned buckets)
with bounded dimensions by default (verb, result). Setting
HELMFILE_OTEL_METRICS_PER_RELEASE=true adds helmfile.release and
helmfile.namespace, answering "which release is slow" from dashboards:
- well-suited to bounded CI runs; long-lived centralized collection
needs a backend capacity/TTL story (documented in docs/otel.md)
- per-release timing remains available in traces without the flag
- env read per call (release operations are low-frequency, and tests
toggle it)
endReleaseSpan now takes the release and the operation start time; the
five worker-loop call sites pass them (doWithReleaseSpan, SyncReleases,
DeleteReleasesForSync, PrepareCharts, DiffReleases).
Verified end-to-end with the console exporter (default dims vs
per-release) and OTLP integration tests pinning both modes.
Refs: #2767, #2769
Signed-off-by: yxxhero <aiopsclub@163.com>
* refactor(telemetry): maintainability pass over the runner/metric plumbing
- StartCommandSpan copies the tracingState struct instead of enumerating
fields by hand — that pattern dropped the meter provider once already
- the two value/pointer ShellRunner switches (helm marker, span
attachment) are unified into one withRunnerCtx helper; the duplication
caused two review rounds of value-form misses
- classifyExec derives the helm classification from the span name
(helmExecSpanName constant) instead of returning a third parallel bool
- metrics: shared outcomeAttrs for the verb/result dimensions, and the
bucket slice renamed to durationBuckets with a comment covering both
histograms that use it
No behavior change; full -race suite green, lint clean.
Refs: #2769
Signed-off-by: yxxhero <aiopsclub@163.com>
* refactor(telemetry): consolidate test env lists, trace bridges, and hook prep; sync the design doc
Maintainability:
- HermeticEnvVars is now exported from pkg/telemetry (the owner of the
env surface) and used by both telemetry tests and otlptest — the two
copies had already drifted once (HELMFILE_OTEL_METRICS_PER_RELEASE
needed updating in both)
- kubedogTraceContext and hookTraceContext were the same concept written
twice; unified into traceOnlyContext(parent...) in span.go
Readability:
- runHook's nested kubectl rewrite extracted into prepareKubectlHook
with guard-clause structure
Accuracy (docs ↔ code, drifted over five review rounds):
- §4.4 now describes the implemented mechanism: the release span is
INJECTED into the runner's own context (preserving the kubedog safety
valve) rather than the runner context being replaced, and helm
classification is marker-based for wrapper binaries
- §5 exec span rows list the actual attributes incl. URL/query masking
- §6 strict profile documents RedactedRef, --kube-token, and the
adjacent-token guarantee
No behavior change; full -race suite green, lint clean.
Refs: #2769
Signed-off-by: yxxhero <aiopsclub@163.com>
* refactor(telemetry): drop the dead noop state, relocate skipUndesired, sync user-facing accuracy
- tracingState.noop was dead weight in the enabled state and a
copy-surface in every transition; a single package-level
noopTracerProvider now backs Tracer while disabled
- skipUndesired moved next to doWithReleaseSpan in span.go, its only
conceptual home (span/metric suppression, not run plumbing)
- accuracy: the package doc, --otel-tracing flag help,
experimental-features entry, and CHANGELOG now all say tracing AND
metrics and list the third instrument (helmfile.release.duration with
the HELMFILE_OTEL_METRICS_PER_RELEASE opt-in) — these had drifted
when the metric was added; the PR description's metric table is
updated to match as well
No behavior change; full -race suite green (except the pre-existing
network-dependent TestStorage_resolveFile flake), lint clean.
Refs: #2769
Signed-off-by: yxxhero <aiopsclub@163.com>
* refactor(telemetry): flatten Setup, name the prefix bound, dedupe test fake; fix instrument-count drift
Readability/maintainability:
- Setup drops from 56 to 39 lines: provider construction (including the
shutdown-tracer-on-meter-failure recovery) moves to newProviders in
exporter.go next to the constructors it composes
- refredact's magic 16 becomes maxForcedFormPrefix with a comment
- span_test's hand-rolled fakeRunner removed in favor of the existing
mockRunner (same package)
Accuracy:
- "Two instruments" wording survived in docs/otel.md and the design
proposal §7 after helmfile.release.duration was added; both now say
three and mention the per-release opt-in
No behavior change; full -race suite green (except the pre-existing
network flake), lint clean.
Refs: #2769
Signed-off-by: yxxhero <aiopsclub@163.com>
* refactor(telemetry): co-locate span machinery, drop a dead export, fix docs nits
- the span plumbing helpers (markHelmExec, withRunnerCtx,
markHelmRunner, spanAttachedContext) move from exec.go to span.go,
next to the marker type and classifiers they serve — exec.go keeps
only the funnel call sites
- otlptest.SpanNames was never used outside the package; unexported
- isHelmBinary's comment now states it is the FALLBACK classifier
(funnel invocations are marker-classified), replacing the outdated
"cosmetic distinction" framing from before the marker existed
- docs/otel.md: release-scoped hooks nest under their release span
(added in review round 2, never documented)
No behavior change; full -race suite green, lint clean.
Refs: #2769
Signed-off-by: yxxhero <aiopsclub@163.com>
---------
Signed-off-by: yxxhero <aiopsclub@163.com>
* Do fail on a possible typo in `needs` entries
Helmfile kindly fails with a friendly error when you made a typo in a `needs` entry, i.e. a `needs` entry included a reference to a release that is not defined in the helmfile config.
Example Output:
```
in ./helmfile.needs.yaml: release(s) "app" depend(s) on an undefined release "infrastructure/cert-manager2". Perhaps you made a typo in `needs` or forgot defining a release named "cert-manager2" with appropriate `namespace` and `kubeContext`?
```
This prevents issues like #1959
* Fix regression in helmfile-diff (This may break when you had two or more duplicated releases that are intended to be de-duplicated before DAG calculation using selectors
* Fix regression when you used selector to deduplicate releases before DAG calculation
* Comments
* Fix regressions in helmfile-apply and helmfile-sync
* Fix regression in duplicate release detection
#1772 broke `--selector` with `needs` in many ways.
The two biggest problems I've encountered were:
- duplicate releases even if you've provided a proper `selector` to deduplicate
- sync/deletion ordering broken when you have `needs`
For the first issue, we had to update `getSelectedReleases` function to also calculate the "selected releases and releases needed by the selected releases", and use that to calculate the DAG. That should have been done in #1772.
The latter started happening after I've fixed the first issue. The source of the issue was that `needs` turned out to be ambiguous in a few cases.
Previously, `needs: ["foo/bar"]` had two meanings. One for "needs release bar in kubecontext foo", another for "needs release bar in namespace foo".
Moreover, `needs: ["foo/bar/baz"]` had three meanings.
- `needs release baz in tiller namespace foo and namespace baz`
- `needs release baz in namespace bar in kubecontext foo`
- `needs release baz in tiller namespace bar in kubecontext foo`.
Especially, the first meaning doesn't make sense at all. Helm 2 solely use tillerNamespace for namespacing the release and Helm 3 uses namespace for that.
This fix sorts all the bugs and issues I've found so far around that, by changing the meanings of the above two examples as follows:
- `foo/bar` means `namespace=foo,name=bar` for Helm 3 and `tillerNamespace=foo,name=bar` for Helm 2
- `needs release bar in kubecontext foo` is now `foo//bar`. Notice the extra `/` between `foo` and `bar`.
- `foo/bar/baz` means `kubecontext=foo,namespace=bar,name=baz` for Helm 3 and `kubecontext=foo,tillerNamespace=bar,name=baz` in Helm 2
Fixes#1818
* Add --{include,skip}-needs to helmfile-sync and helmfile-apply
* Add --include-needs to helmfile-template
* Add TODO related to #1018
* Add a few new test files to cover new functionalities
* Update apply test to incorporate the change that the destroy and sync steps target affected releases only
This is the GA version of the helm-x integration #673 developed last year.
You get all the following benefits without an extra helm plugin:
- Ability to add ad-hoc chart dependencies/aliases, without forking the chart (Fixes#876 )
- Ability to patch resulting K8s resources before installing the helm chart
- Ability to install a kustomization as a chart (Requires `kustomize` binary to be available in `$PATH`
- Ability to install a directory of K8s manifests as a chart
- etc.
The problem was that `--namespace NS` had been not taken into account while deleting releases, that resulted in releases that should be deleted are not deleted.
* fix: Fix `needs` to work for upgrades and when selectors are provided
Fixes#919
* Add test framework for `helmfile apply`
* Various enhancements and fixes to the DAG support
- Make the order of upgrades/deletes more deterministic for testability
- Fix the test framework so that we can validate log outputs and errors
- Add more test cases for `helmfile apply`, along with bug fixes.
- Make sure it fails with an intuitive error when you have non-existent releases referenced from witin "needs"
Introduces DAG-aware installation/deletion ordering to Helmfile.
`needs` controls the order of the installation/deletion of the release:
```yaml
relesaes:
- name: somerelease
needs:
- [TILLER_NAMESPACE/][NAMESPACE/]anotherelease
```
All the releases listed under `needs` are installed before(or deleted after) the release itself.
For the following example, `helmfile [sync|apply]` installs releases in this order:
1. logging
2. servicemesh
3. myapp1 and myapp2
```yaml
- name: myapp1
chart: charts/myapp
needs:
- servicemesh
- logging
- name: myapp2
chart: charts/myapp
needs:
- servicemesh
- logging
- name: servicemesh
chart: charts/istio
needs:
- logging
- name: logging
chart: charts/fluentd
```
Note that all the releases in a same group is installed concurrently. That is, myapp1 and myapp2 are installed concurrently.
On `helmdile [delete|destroy]`, deleations happen in the reverse order.
That is, `myapp1` and `myapp2` are deleted first, then `servicemesh`, and finally `logging`.
Resolves#715
yxxhero