Toolchain and CI:
- Bump go directive from 1.26.8 to 1.27.1 (go.mod)
- Use golang:1.27-alpine builder images in all Dockerfiles
- Bump golangci-lint to v2.13.2 (first release with go1.27 support)
- Add CI gate: `go fix -diff` fails when outdated Go patterns are
detected (locally: `make check-modernize`)
Note: darwin binaries now require macOS 13 or later.
Lint fixes required by golangci-lint v2.13.2:
- goconst: ignore tests (all 436 findings were test-only; goconst
got stricter since v2.12 and this option was added for it)
- openai.go: keep deprecated MaxTokens deliberately with a nolint
rationale (max_tokens is the only form universally supported by
OpenAI-compatible backends like One-API, LiteLLM, Ollama shim)
- state.go: drop always-nil flags param from appendChartVersionFlags
(renamed to chartVersionFlags, unparam)
Modernization (go fix ./..., 62 files):
- interface{} -> any, maps.Copy, strings.SplitSeq, range-over-int,
builtin min/max, slices.Contains/ContainsFunc/Sort, WaitGroup.Go,
reflect.Type.Fields(), new(expr)
- exit_error.go: strings.Builder + fmt.Fprintf instead of string
concatenation and WriteString(fmt.Sprintf(...)) (QF1012)
- chart_dependency.go: strings.CutLast for OCI dependency helpers
Signed-off-by: yxxhero <aiopsclub@163.com>
Co-authored-by: Claude <noreply@anthropic.com>
* feat(telemetry): add opt-in OpenTelemetry tracing (PR 1: lifecycle + root span)
Implements the first increment of docs/proposals/otel-tracing.md (#2767):
- pkg/telemetry: SDK setup from standard OTEL_* env vars (autoexport for
exporter selection, env-driven sampler/propagators, OTEL_SDK_DISABLED),
command-span lifecycle, no-op-by-default accessors
- --otel-tracing flag / HELMFILE_OTEL_TRACING env switch
- root span "helmfile <command>" with file/environment/selectors/exit_code
attributes; TRACEPARENT-based remote-parent extraction for CI correlation
- shutdown flush on both normal-exit and signal paths (nil-safe, 5s bound)
- app.New derives its context from telemetry.CommandContext()
(Background-identical when tracing is disabled)
- docs: otel.md user guide, experimental-features entry, design proposal
- tests: hermetic unit tests, app context-contract pinning, flag registration
Telemetry problems never fail a run: exporter misconfiguration and export
errors degrade to disabled with a warning. When disabled, behavior and
performance are identical to before (no-op tracer, no goroutines, no
network).
Refs: #2767, #2758
Signed-off-by: yxxhero <aiopsclub@163.com>
* feat(telemetry): trace every external process + trace-context bridges (PR 2)
Implements the second increment of docs/proposals/otel-tracing.md (#2767):
- pkg/helmexec/span.go: one span per external process started by helmfile
(helm invocations, hooks, plugin execs) at the ShellRunner choke point —
helm.exec (with helm.subcommand) vs os.exec, with redacted exec.args,
exec.exit_code, and error status on failure
- pkg/helmexec/redact.go: shared argument redaction with two profiles;
legacy is byte-identical to the historical exit-error behavior (existing
goldens unchanged), strict (spans) additionally covers --set=k=v and
credential flags; exit_error.go now uses the shared helper
- orphan-trace bridges with bit-identical cancellation semantics
(context.WithoutCancel of the command context): both kubedog call sites
(state.go) and hook execution (event.Bus gains an optional Ctx consumed
by its default runner; state.go sets it, nil falls back to TODO as before)
- OTLP end-to-end test (in-process httptest receiver, no external
collector): span export, error status/exit code, redaction, and
parent-linkage to the command span
- docs/otel.md updated to the now-traced surface
Verified end-to-end with the console exporter: helmfile template on a
local chart yields the command span plus helm.exec spans for helm
version/dependency/template, all nested under it.
Refs: #2767
Signed-off-by: yxxhero <aiopsclub@163.com>
* feat(telemetry): state-loading and hook spans (PR 3a)
Implements the third increment of docs/proposals/otel-tracing.md (#2767):
- helmfile.discover_states around findDesiredStateFiles and helmfile.load
around loadDesiredStateFromYamlWithBaseDir; both cover all callers
(incl. nested helmfiles) with no signature changes
- helmfile.render / helmfile.parse children per document part, parented
through a traceCtx field on the unexported desiredStateLoader struct
(set once at its single construction site)
- helmfile.hook span per hook execution: Trigger's per-hook body extracted
into runHook (readability win on its own), the hook's subprocess span
nests under it via a per-hook ctx-swapped ShellRunner clone
(cancellation unchanged — Bus.Ctx never carries cancellation by contract)
- pkg/telemetry/otlptest: shared in-process OTLP/HTTP receiver harness,
now used by helmexec, event, and app span tests
- golden span-tree test at the app layer (root -> discover -> load ->
render/parse, via the exectest fake helm) and a hook-span nesting test
- nil-ctx guard for App literals built directly by tests (App.spanParentCtx)
Verified end-to-end with the console exporter: a template run over a
gotmpl state file with a prepare hook yields the full tree with the hook's
os.exec nested under helmfile.hook.
Refs: #2767
Signed-off-by: yxxhero <aiopsclub@163.com>
* feat(telemetry): per-release spans nested under the load span (PR 3b)
Implements the per-release increment of docs/proposals/otel-tracing.md
(#2767) — spans nest command -> load -> release -> helm exec:
- helmexec.HelmContext gains an optional Ctx carrying the per-release span
context; the execer's new execWithContext funnel consumes it via a
per-call runner clone (runnerWithCtx) so the shared, cached execer is
never mutated across concurrent workers. The seven Interface methods
that take a HelmContext (Sync/Diff/ReleaseStatus/List/DecryptSecret/
Delete/Test) route through it; nil Ctx behaves exactly as before.
exec() lost its always-nil override parameter on the way (unparam).
- pkg/state/span.go: SetTraceContext + startReleaseSpan/endReleaseSpan
helpers (release/namespace/chart/labels attributes, sorted for stable
output); a typed-nil guard (releaseErrAsError) avoids the classic
nil-pointer-in-interface trap on *ReleaseError.
- release spans in the worker loops: SyncReleases, DiffReleases,
DeleteReleasesForSync, PrepareCharts, and iterateOnReleases (status/
delete/test via a new verb parameter); their HelmContext is stamped with
the release span context where one is built.
- pkg/app sets st.SetTraceContext(loadCtx) right after loading a state
file, rooting all per-release spans under helmfile.load.
- bridged one more detached tracking call found on the way
(trackReleaseIfEnabled's context.Background in the sync worker).
- golden test: release span present, nested under load, correct
attributes; unit test for the runnerWithCtx clone semantics.
Verified with the console exporter: helmfile template yields
release.prepare(demo) under load with full attributes.
Refs: #2767
Signed-off-by: yxxhero <aiopsclub@163.com>
* feat(telemetry): nest status/delete/test execs under their release spans
Completes the per-release exec nesting for the iterateOnReleases-based
loops (docs/proposals/otel-tracing.md §4.4 phase 2): the do closures now
receive the release span context and stamp it into their HelmContext, so
helm status/delete/test subprocess spans nest under
helmfile.release.<verb> like sync/diff already did.
- scatterGatherReleases/iterateOnReleases/doWithReleaseSpan: do gains a
context parameter (the release span context)
- ReleaseStatuses/DeleteReleases/TestReleases closures stamp
HelmContext.Ctx from it
- integration test with a real execer (version-probe shim binary): the
release's status subprocess nests under helmfile.release.status, same
trace, with helm.subcommand=status
This also makes the otel.md claim ("upgrade, diff, delete, status, test
nested under the release span") fully accurate.
Refs: #2767
Signed-off-by: yxxhero <aiopsclub@163.com>
* feat(telemetry): OTel metrics — helm exec duration and release results (PR 4)
Implements the metrics increment of docs/proposals/otel-tracing.md
(#2767) on the same provider, switch, and resource as traces:
- pkg/telemetry/metrics.go: helmfile.helm.exec.duration histogram
(subcommand, success) and helmfile.release.count counter (verb,
result). Instruments come from the otel global meter, so recording at
call sites is branch-free no-op when telemetry is disabled.
- Setup builds the resource once and installs both providers; reader
selection delegates to autoexport (OTEL_METRICS_EXPORTER: otlp |
console | prometheus | none), the OTLP reader's interval honors
OTEL_METRIC_EXPORT_INTERVAL (read by the SDK). Shutdown flushes both
providers (errors.Join). StartCommandSpan now carries the meter
provider across state transitions (fixes a nil-shutdown panic).
- helmexec: finishExecSpan records exec duration for helm binaries;
state: endReleaseSpan counts release outcomes for sync/diff/delete/
status/test/prepare (diff counted as success when no hard error).
- otlptest: recorder routes by OTLP path (/v1/traces vs /v1/metrics)
and decodes metrics; new FindMetric helper.
- tests: metrics recorded as no-op when disabled, provider enabled with
the none exporter, degradation on an invalid metrics exporter, and an
integration assertion (status exec duration datapoint + one successful
release.count) in the shim-based state test.
Verified with the console exporter: helmfile template emits
helmfile.helm.exec.duration per subcommand (version/dependency/
template) and helmfile.release.count{verb=prepare,result=success}=1.
Refs: #2767
Signed-off-by: yxxhero <aiopsclub@163.com>
* docs: complete OTel documentation coverage
- docs/cli.md: --otel-tracing in the CLI reference help block (verbatim
from the cobra output)
- CHANGELOG.md: [Unreleased] Added entry for tracing + metrics
- docs/index.md: Observability highlight linking docs/otel.md
- docs/proposals/otel-tracing.md: add OTEL_METRICS_EXPORTER /
OTEL_METRIC_EXPORT_INTERVAL rows to the env-var table and note the
periodic reader + bounded metric cardinality in §7
Refs: #2767
Signed-off-by: yxxhero <aiopsclub@163.com>
* fix: drop unused id parameter from parsePart (unparam)
The id parameter was never used inside the span wrapper; the caller's id
variable is still used for the render calls and error messages.
Refs: #2769
Signed-off-by: yxxhero <aiopsclub@163.com>
* fix(telemetry): address review — redaction gaps, kubedog valve, phantom metrics
Addresses all Copilot review comments on #2769:
Security (span payloads):
- exec.args: positional arguments are additionally passed through
helmexec.RedactedURL, so credentials embedded in chart/repository URLs
(AddRepo, RegistryLogin, OCI refs) are masked exactly like log output
- release spans sanitize helmfile.chart the same way
- error statuses no longer embed raw errors (which contain rendered
commands, arguments, and subprocess output): the command span, release
spans, hook spans, and exec spans now use generic descriptions; the
concrete exit code remains an attribute, and RecordError on the root
span is dropped
Correctness:
- kubedog safety valve restored: execWithContext now attaches the
per-release span into the runner's own context instead of replacing it,
so trackHandle.Cancel() can interrupt a wedged helm again and app
cancellation semantics stay exactly as before the PR
- diff release spans/metrics: real failures are recorded (exit code 2
"changes detected" still counts as success); previously every diff was
exported as successful
- skipped releases no longer emit phantom spans and inflate
helmfile.release.count: iterateOnReleases callers pass a skip predicate
(skipUndesired for status/test; delete deletes undesired releases and
passes nil)
- Setup shuts down the already-constructed tracer provider (bounded) when
the metrics provider fails, instead of abandoning its batch goroutine
Tests: URL redaction cases (masked/untouched), spanAttachedContext
preserves the runner cancellation chain while attaching the caller's
span, skipUndesired, and a failing-hook span asserting the generic
message.
Refs: #2769
Signed-off-by: yxxhero <aiopsclub@163.com>
* fix: lint — restore nolint placement and avoid nil context literal
- the skipUndesired insertion had displaced the // nolint: unparam
directive off iterateOnReleases (helm param is intentionally unused
there); also fixes a skipDesired/skipUndesired comment typo
- use a typed nil in TestSpanAttachedContext (staticcheck SA1012)
Refs: #2769
Signed-off-by: yxxhero <aiopsclub@163.com>
* fix(telemetry): address review round 2 — remote-ref redaction, wrapper helm binaries, hook release attribution
Addresses all 6 new review comments on #2769:
Security (remote references):
- new helmexec.RedactedRef sanitizes go-getter style references for
telemetry: forced-form prefixes (git::, s3::) preserved, whole URL
userinfo masked (usernames carry tokens too), credential-bearing query
parameters masked using pkg/remote's heuristic (token/password/secret/
key/signature). Applied to helmfile.file (command span), helmfile.path
(discover_states), helmfile.chart (release spans), and exec.args —
log-time RedactedURL is untouched so log output is unchanged
Correctness:
- wrapper helm binaries (--helm-binary custom names) are now classified
as helm operations by an explicit context marker stamped in the execer
funnel, instead of the executable-basename heuristic; the same
classification gates helmfile.helm.exec.duration, so the metric no
longer misses wrapper invocations (classifyExec)
- release-scoped hooks (presync/postsync/preuninstall/postuninstall/
cleanup in the sync/delete/diff workers) now attach their helmfile.hook
spans to the active helmfile.release.* span via a variadic parent on
the trigger functions; global hooks keep the command context and all
29 existing call sites compile unchanged; hook cancellation stays
detached (WithoutCancel) as before
- signal-terminated runs (Shutdown with exitCode 130/143 and nil error)
now mark the command span with error status, consistent with their
nonzero exit code
Tests: RedactedRef table (forced forms, userinfo, s3/token query params,
untouched cases), classifyExec marker case, hookTraceContext parent
attribution + non-cancellability + fallback.
Refs: #2769
Signed-off-by: yxxhero <aiopsclub@163.com>
* fix(telemetry): address review round 3 — redaction corner cases, value runners
Addresses 5 of the 6 new review comments on #2769 (the sixth — an
unused strings import in exit_error.go — is a false positive: Indent
still uses strings.Split/Builder and the package compiles):
- RedactArgs read the previous token from the progressively redacted
output, so {--set, --set-string, secret} leaked the secret
(the masked value hid the following flag). Read the previous token
from the original input, restoring the legacy contract for adjacent
secret flags
- RedactedRef fails closed for malformed references: URL-like refs with
invalid percent escapes export a fully redacted value, and an
unparseable query is dropped entirely instead of exported verbatim
- ShellRunner has value receivers, so a ShellRunner VALUE satisfies the
Runner API; the helm marker stamping and the per-release span
attachment now handle both value and pointer forms (matching
WithContext), so value-runner callers keep release nesting and the
helm.exec classification/metric
Regression tests: adjacent secret flags (legacy + strict), malformed
URL-like ref, malformed query, value-runner marker + span attachment.
Refs: #2769
Signed-off-by: yxxhero <aiopsclub@163.com>
* fix(telemetry): stamp the helm marker on the stdin funnel too
execStdIn (registry login, repo add) called the runner directly, so
wrapper --helm-binary names were misclassified as os.exec and omitted
from helmfile.helm.exec.duration on that path. The marking now goes
through a shared markHelmRunner helper (value and pointer ShellRunner
forms) used by both execution funnels.
Refs: #2769
Signed-off-by: yxxhero <aiopsclub@163.com>
* fix(telemetry): redact helm's --kube-token in strict profile
Helm's global --kube-token carries a bearer token; both the
two-argument and inline forms are now masked in span exec.args
(legacy exit-error output is untouched, matching its historical
behavior).
Refs: #2769
Signed-off-by: yxxhero <aiopsclub@163.com>
* fix(telemetry): OTel metrics best-practice alignment
- helmfile.helm.exec.duration now declares explicit bucket boundaries
tuned for seconds-scale helm invocations (5ms…600s); the SDK defaults
are millisecond-oriented and lumped every sub-5s invocation — the
common case — into the first bucket, defeating the histogram
- instruments are re-created under the installed provider with the
instrumentation scope version stamped (Setup-time, race-free)
- helmfile.release.count declares the {release} curly-annotation unit
per the metrics naming conventions
Tested end-to-end via the OTLP integration test: exported bounds are
the tuned set, units are asserted, and the scope carries the version.
Refs: #2769
Signed-off-by: yxxhero <aiopsclub@163.com>
* feat(telemetry): per-release duration metrics behind an opt-in switch
New helmfile.release.duration histogram (seconds, same tuned buckets)
with bounded dimensions by default (verb, result). Setting
HELMFILE_OTEL_METRICS_PER_RELEASE=true adds helmfile.release and
helmfile.namespace, answering "which release is slow" from dashboards:
- well-suited to bounded CI runs; long-lived centralized collection
needs a backend capacity/TTL story (documented in docs/otel.md)
- per-release timing remains available in traces without the flag
- env read per call (release operations are low-frequency, and tests
toggle it)
endReleaseSpan now takes the release and the operation start time; the
five worker-loop call sites pass them (doWithReleaseSpan, SyncReleases,
DeleteReleasesForSync, PrepareCharts, DiffReleases).
Verified end-to-end with the console exporter (default dims vs
per-release) and OTLP integration tests pinning both modes.
Refs: #2767, #2769
Signed-off-by: yxxhero <aiopsclub@163.com>
* refactor(telemetry): maintainability pass over the runner/metric plumbing
- StartCommandSpan copies the tracingState struct instead of enumerating
fields by hand — that pattern dropped the meter provider once already
- the two value/pointer ShellRunner switches (helm marker, span
attachment) are unified into one withRunnerCtx helper; the duplication
caused two review rounds of value-form misses
- classifyExec derives the helm classification from the span name
(helmExecSpanName constant) instead of returning a third parallel bool
- metrics: shared outcomeAttrs for the verb/result dimensions, and the
bucket slice renamed to durationBuckets with a comment covering both
histograms that use it
No behavior change; full -race suite green, lint clean.
Refs: #2769
Signed-off-by: yxxhero <aiopsclub@163.com>
* refactor(telemetry): consolidate test env lists, trace bridges, and hook prep; sync the design doc
Maintainability:
- HermeticEnvVars is now exported from pkg/telemetry (the owner of the
env surface) and used by both telemetry tests and otlptest — the two
copies had already drifted once (HELMFILE_OTEL_METRICS_PER_RELEASE
needed updating in both)
- kubedogTraceContext and hookTraceContext were the same concept written
twice; unified into traceOnlyContext(parent...) in span.go
Readability:
- runHook's nested kubectl rewrite extracted into prepareKubectlHook
with guard-clause structure
Accuracy (docs ↔ code, drifted over five review rounds):
- §4.4 now describes the implemented mechanism: the release span is
INJECTED into the runner's own context (preserving the kubedog safety
valve) rather than the runner context being replaced, and helm
classification is marker-based for wrapper binaries
- §5 exec span rows list the actual attributes incl. URL/query masking
- §6 strict profile documents RedactedRef, --kube-token, and the
adjacent-token guarantee
No behavior change; full -race suite green, lint clean.
Refs: #2769
Signed-off-by: yxxhero <aiopsclub@163.com>
* refactor(telemetry): drop the dead noop state, relocate skipUndesired, sync user-facing accuracy
- tracingState.noop was dead weight in the enabled state and a
copy-surface in every transition; a single package-level
noopTracerProvider now backs Tracer while disabled
- skipUndesired moved next to doWithReleaseSpan in span.go, its only
conceptual home (span/metric suppression, not run plumbing)
- accuracy: the package doc, --otel-tracing flag help,
experimental-features entry, and CHANGELOG now all say tracing AND
metrics and list the third instrument (helmfile.release.duration with
the HELMFILE_OTEL_METRICS_PER_RELEASE opt-in) — these had drifted
when the metric was added; the PR description's metric table is
updated to match as well
No behavior change; full -race suite green (except the pre-existing
network-dependent TestStorage_resolveFile flake), lint clean.
Refs: #2769
Signed-off-by: yxxhero <aiopsclub@163.com>
* refactor(telemetry): flatten Setup, name the prefix bound, dedupe test fake; fix instrument-count drift
Readability/maintainability:
- Setup drops from 56 to 39 lines: provider construction (including the
shutdown-tracer-on-meter-failure recovery) moves to newProviders in
exporter.go next to the constructors it composes
- refredact's magic 16 becomes maxForcedFormPrefix with a comment
- span_test's hand-rolled fakeRunner removed in favor of the existing
mockRunner (same package)
Accuracy:
- "Two instruments" wording survived in docs/otel.md and the design
proposal §7 after helmfile.release.duration was added; both now say
three and mention the per-release opt-in
No behavior change; full -race suite green (except the pre-existing
network flake), lint clean.
Refs: #2769
Signed-off-by: yxxhero <aiopsclub@163.com>
* refactor(telemetry): co-locate span machinery, drop a dead export, fix docs nits
- the span plumbing helpers (markHelmExec, withRunnerCtx,
markHelmRunner, spanAttachedContext) move from exec.go to span.go,
next to the marker type and classifiers they serve — exec.go keeps
only the funnel call sites
- otlptest.SpanNames was never used outside the package; unexported
- isHelmBinary's comment now states it is the FALLBACK classifier
(funnel invocations are marker-classified), replacing the outdated
"cosmetic distinction" framing from before the marker existed
- docs/otel.md: release-scoped hooks nest under their release span
(added in review round 2, never documented)
No behavior change; full -race suite green, lint clean.
Refs: #2769
Signed-off-by: yxxhero <aiopsclub@163.com>
---------
Signed-off-by: yxxhero <aiopsclub@163.com>
* feat: allow helmfile to continue on failed releases
Co-authored-by: Peter Honeder <peter.honeder@unwired.at>
Signed-off-by: Niklas Ott <niklas.ott@unwired.at>
* fix: skip failed-prep releases, complete flag wiring, add tests and docs (#64)
Review follow-ups for --allow-failed-releases (#2616):
- Track per-release chart preparation failures in PrepareCharts (returned
as a map keyed by release) and remove those releases from the state in
Run.WithPreparedCharts when --allow-failed-releases is set, so a failed
release is never executed against its original, un-prepared chart
reference (which could either fail again with a duplicate error or, for
charts requiring chartify, bypass patches/dependency modifications and
produce an unintended result). All failures are still reported at the
end via the aggregated MultiError.
- Complete the release identity on error results from
prepareChartForRelease so failures are attributed to the correct
release.
- With --allow-failed-releases, continue building dependencies of the
remaining charts when 'helm dep build' fails for one release, and skip
the affected releases during execution.
- Wire --allow-failed-releases into 'helmfile unittest' and 'helmfile
status'; remove the dead flag wiring for write-values and list (both
never prepare charts, see commandsSkipChartPrep).
- Simplify control flow (guard clauses, errors.As, drop dead code and
redundant else branches).
- Add end-to-end coverage in pkg/app/issue_2616_test.go and extend the
state-level tests; document the flag in docs/cli.md and CHANGELOG.md.
Signed-off-by: yxxhero <11087727+yxxhero@users.noreply.github.com>
---------
Signed-off-by: Niklas Ott <niklas.ott@unwired.at>
Signed-off-by: yxxhero <11087727+yxxhero@users.noreply.github.com>
Co-authored-by: Peter Honeder <peter.honeder@unwired.at>
Co-authored-by: yxxhero <11087727+yxxhero@users.noreply.github.com>
* feat: add --repo-retries for retrying helm repo and registry login commands
Add a configurable retry mechanism for chart repository operations to
handle unstable networks (corporate proxies, slow internal registries).
Closes#1894
- New --repo-retries N flag and HELMFILE_REPO_RETRIES env var (default 0
= opt-in, backward compatible)
- Retry applies to helm repo add, helm repo update (incl. ACR), and
helm registry login with exponential backoff (1s, 2s, 4s, ..., capped 30s)
- Single retryRepoOp helper; per-attempt args/buffer are local to avoid
state leaking across retries
- Tests cover succeed-after-retry, exhausted-retries, disabled-by-default,
and regression guards for password-buffer and args-accumulation
Signed-off-by: yxxhero <aiopsclub@163.com>
* fix: address PR review (overflow guard, cancellable sleep, flag-override, docs)
Address Copilot review feedback on #2683:
- Cap backoff shift exponent at 5 to prevent time.Duration overflow on
large --repo-retries values
- Make retry sleep context-aware (sleepCtx) so Ctrl+C aborts the retry
loop promptly via the ShellRunner context
- Log a concise exit status instead of the verbose ExitError dump, and
clarify the retry-counter wording ('retry N/M')
- Use -1 sentinel as the CLI default so --repo-retries=0 can explicitly
disable retries even when HELMFILE_REPO_RETRIES is set
- Align help text and docs: retry applies 'on failure' (not just
transient errors), document the 0-disables behavior
- Add tests for overflow guard, cancellable sleep, and flag-zero-disables
Signed-off-by: yxxhero <aiopsclub@163.com>
* fix: abort retries on canceled context, hide sentinel default, align comment
Address follow-up Copilot review on #2683:
- Fix tight-loop bug: sleepCtx now returns whether it completed vs was
interrupted by context cancellation, and retryRepoOp aborts the retry
loop on interruption so Ctrl+C no longer spins into rapid helm calls
- Hide the -1 sentinel from --help by overriding the displayed default
to 0 (pflag DefValue), matching the documented default while keeping
the flag-override semantics
- Correct HelmExecOptions.RepoRetry comment: 'on failure' not 'transient
network errors', matching the actual retry behavior
- Add Test_Retry_AbortsOnCanceledContext covering the no-tight-loop path
Signed-off-by: yxxhero <aiopsclub@163.com>
* fix: copy args per retry in RegistryLogin, make cancel test deterministic
Address follow-up Copilot review on #2683:
- RegistryLogin: pass a per-attempt copy of args to execStdIn so its
internal append (for helm.extra) can't alias the shared slice across
retries
- Test_Retry_AbortsOnCanceledContext: cancel the context deterministically
inside the op closure after the first attempt, replacing the flaky
time.Sleep(20ms) goroutine
Signed-off-by: yxxhero <aiopsclub@163.com>
* fix: return error on unknown managed repo type instead of silent skip
Address Copilot review on #2683: AddRepo logged an error for an unknown
managed type but returned nil, silently succeeding while skipping the
repo add. Now returns an error so misconfigurations fail loudly.
Signed-off-by: yxxhero <aiopsclub@163.com>
---------
Signed-off-by: yxxhero <aiopsclub@163.com>
* feat: add `helmfile doctor` command for AI-assisted diff analysis
`helmfile doctor` runs `helmfile diff` and asks an OpenAI-compatible LLM to
summarize the changes and flag risks (data loss, security exposure, breaking
changes, downtime, performance, best-practice issues).
Key design decisions:
- When no LLM is configured, doctor is equivalent to `helmfile diff` with
one exception: --show-secrets is always forced off (secrets never reach
stdout, even without an LLM).
- Secrets are ALWAYS redacted via two layers: (1) ShowSecrets() forced to
false so helm-diff emits <REDACTED> placeholders; (2) a defense-in-depth
text redactor strips residual secret-looking content (Secret YAML blocks,
sensitive key/value lines, base64 blobs, JWT tokens) before LLM transmission.
- LLM configuration precedence: env (HELMFILE_LLM_*) < helmfile.yaml (llm:)
< CLI flags (--llm-*).
- Supports any OpenAI-compatible backend (OpenAI, Azure, One-API, LiteLLM,
Ollama, etc.) with automatic response_format fallback for backends that
don't support JSON mode.
- Prompt injection defense: release names and environment values are
JSON-encoded before insertion into the LLM prompt.
- Exit codes: 0 (success/low-risk), 2 (high-risk gate, bypass with --force),
1 (other errors). Helm-diff's 'detected changes' exit-2 is swallowed.
New packages:
- pkg/agent/llm: OpenAI-compatible client with JSON response parsing, mock
client for testing, prompt builder with injection defense.
- pkg/agent/doctor: secret redactor (state machine + regex), report renderer
(markdown + JSON), config resolver (env < yaml < flag merge).
Testing: 70+ unit tests covering redaction patterns, prompt injection,
response_format fallback, JSON parsing, yaml roundtrip, concurrency safety,
panic recovery, and error propagation. go test -race passes.
Documentation: full doctor section in docs/cli.md, llm: block reference in
docs/configuration.md, updated skills/helmfile for AI agents.
Signed-off-by: yxxhero <aiopsclub@163.com>
* docs: fix doctor equivalence wording per PR review
Per review feedback (PR #2660): the docs claimed doctor is 'equivalent to
helmfile diff — same flags, same output, same exit codes' in the unconfigured
path, but this over-promises because:
1. doctor --output is the report format (not helm-diff's output format)
2. helm-diff's --output is exposed as --diff-output in doctor
3. --show-secrets is silently ignored
Updated all three locations (cli.md, cmd/doctor.go Long + godoc, pkg/app/doctor.go
godoc) to say 'falls back to helmfile diff with --show-secrets forced off' and
explicitly note the --output / --diff-output flag difference.
Signed-off-by: yxxhero <aiopsclub@163.com>
---------
Signed-off-by: yxxhero <aiopsclub@163.com>
* feat: support more HELMFILE_* env vars as flag fallbacks
Adds env-var fallbacks for global flags, mirroring the existing
HELMFILE_ENVIRONMENT / HELMFILE_KUBE_CONTEXT pattern:
* --helm-binary -> HELMFILE_HELM_BINARY
* --kustomize-binary -> HELMFILE_KUSTOMIZE_BINARY
* --log-level -> HELMFILE_LOG_LEVEL
* --debug -> HELMFILE_DEBUG (expecting "true" lower case)
* --quiet -> HELMFILE_QUIET (expecting "true" lower case)
* --no-color -> HELMFILE_NO_COLOR (expecting "true" lower case),
additionally honors NO_COLOR per no-color.org
(any non-empty value disables color)
Flag values still take precedence; env vars are consulted only when the
flag is unset. The string-flag default values ("helm", "kustomize",
"info") move into the accessor methods so the env-var fallback can
actually trigger when no flag is passed.
Signed-off-by: Dominik Schmidt <dev@dominik-schmidt.de>
* docs: mention new HELMFILE_* env vars in cli.md and templating.md
Signed-off-by: Dominik Schmidt <dev@dominik-schmidt.de>
* fix: make Color/NoColor/env interaction consistent
Two issues with the env-aware NoColor() introduced together with
HELMFILE_NO_COLOR / NO_COLOR support:
1. Color() consulted the raw GlobalOptions.NoColor field instead of
NoColor(), so in a TTY with only the env set, Color() fell through
to terminal autodetect and ValidateConfig() spuriously errored with
"--color and --no-color cannot be specified at the same time".
2. NoColor() returned true via env even when --color was explicitly
passed, so `helmfile --color` with NO_COLOR (or HELMFILE_NO_COLOR=true)
in the environment hit the same ValidateConfig() error. A flag should
always win over an env var.
Fix both by routing Color() through NoColor() and giving NoColor() an
explicit --color short-circuit. Regression tests added for both paths.
Signed-off-by: Dominik Schmidt <dev@dominik-schmidt.de>
---------
Signed-off-by: Dominik Schmidt <dev@dominik-schmidt.de>
* feat: support HELMFILE_NAMESPACE env var for default namespace
Mirrors the existing HELMFILE_ENVIRONMENT pattern: the --namespace
CLI flag takes precedence, falling back to HELMFILE_NAMESPACE when
unset.
Signed-off-by: Dominik Schmidt <dev@dominik-schmidt.de>
* docs: mention HELMFILE_NAMESPACE in cli.md and templating.md
Signed-off-by: Dominik Schmidt <dev@dominik-schmidt.de>
---------
Signed-off-by: Dominik Schmidt <dev@dominik-schmidt.de>
* feat: support HELMFILE_KUBE_CONTEXT env var for default kube context
Mirrors the existing HELMFILE_ENVIRONMENT pattern: the --kube-context
CLI flag takes precedence, falling back to HELMFILE_KUBE_CONTEXT when
unset.
Refs #1213.
Signed-off-by: Dominik Schmidt <dev@dominik-schmidt.de>
* docs: mention HELMFILE_KUBE_CONTEXT in cli.md and templating.md
Signed-off-by: Dominik Schmidt <dev@dominik-schmidt.de>
---------
Signed-off-by: Dominik Schmidt <dev@dominik-schmidt.de>
* feat: add 'create' subcommand to scaffold helmfile deployment projects
Add 'helmfile create [NAME]' command that generates a best-practice
helmfile project structure with:
- helmfile.yaml with commented examples (helmDefaults, repositories,
environments, releases)
- environments/default.yaml for environment-specific values
- values/.gitkeep placeholder for release values
Supports --output-dir/-o for custom output path and --force to
overwrite existing files. Validates project name to prevent path
traversal.
Signed-off-by: yxxhero <aiopsclub@163.com>
* fix: add overwrite protection for all scaffold files and unit tests for create command
- pkg/app/create.go: extract writeFileIfNotExists helper that respects the
--force flag; all three scaffold files (helmfile.yaml,
environments/default.yaml, values/.gitkeep) now refuse to overwrite
without --force
- pkg/config/create.go: ValidateConfig now checks all three scaffold paths
and reports every already-existing file in a single error before
proceeding, instead of only checking helmfile.yaml
- pkg/app/create_test.go: add unit tests covering new directory, current
directory, per-file overwrite rejection without --force, and full
overwrite with --force
Agent-Logs-Url: https://github.com/helmfile/helmfile/sessions/eb6d9e4b-0f72-4e26-b841-e1e39a2b2e83
Co-authored-by: yxxhero <11087727+yxxhero@users.noreply.github.com>
* fix: remove redundant absDir from ValidateConfig error message
Agent-Logs-Url: https://github.com/helmfile/helmfile/sessions/eb6d9e4b-0f72-4e26-b841-e1e39a2b2e83
Co-authored-by: yxxhero <11087727+yxxhero@users.noreply.github.com>
* fix: address create command review feedback
- cmd/create.go: add config.NewCLIConfigImpl() call for consistency with other
subcommands; update --force flag help text to list all overwritten files
- pkg/config/create.go: delegate to c.GlobalImpl.ValidateConfig() at end of
ValidateConfig() for global option validation (--color/--no-color)
- pkg/config/create_test.go: add unit tests for CreateImpl.ValidateConfig()
covering path separator rejection, '..' rejection, existing-file detection
per-file and with --force, and global color conflict delegation
Agent-Logs-Url: https://github.com/helmfile/helmfile/sessions/6327d657-5888-4b94-85fb-def80c0a193f
Co-authored-by: yxxhero <11087727+yxxhero@users.noreply.github.com>
* fix: clarify test helper name and comment in create_test.go
Agent-Logs-Url: https://github.com/helmfile/helmfile/sessions/6327d657-5888-4b94-85fb-def80c0a193f
Co-authored-by: yxxhero <11087727+yxxhero@users.noreply.github.com>
* fix: atomic preflight check in App.Create before any writes
Refactor Create to collect all conflicting scaffold paths up front
before writing anything. When --force is not set and any scaffold file
already exists, the command returns a single error listing all
conflicts without touching the filesystem.
Also removes the now-unnecessary writeFileIfNotExists helper and adds a
test (TestCreate_PreflightAtomicOnLaterConflict) verifying that a
conflict on a later file (e.g. environments/default.yaml) prevents even
the first file (helmfile.yaml) from being created.
Agent-Logs-Url: https://github.com/helmfile/helmfile/sessions/aae6f2e6-7f9e-42b8-afa3-78edd3215127
Co-authored-by: yxxhero <11087727+yxxhero@users.noreply.github.com>
* fix: handle non-IsNotExist Stat errors in preflight check; add whitespace name test; fix gci formatting
- pkg/app/create.go: treat os.Stat errors that are NOT os.IsNotExist as
hard errors in the preflight scan, surfacing permission/IO issues before
any writes happen; remove trailing blank line that caused gci failure
- pkg/config/create.go: same non-IsNotExist error handling in ValidateConfig
- pkg/config/create_test.go: add TestCreateImpl_ValidateConfig_WhitespaceOnlyName
covering the " " (whitespace-only) name rejection branch
Agent-Logs-Url: https://github.com/helmfile/helmfile/sessions/d6574f56-f46d-46f7-99d9-e0b0b897b3b5
Co-authored-by: yxxhero <11087727+yxxhero@users.noreply.github.com>
* refactor: eliminate duplicated scaffold existence check; use O_EXCL for TOCTOU protection
- pkg/config/create.go: remove file-existence check from ValidateConfig
(duplicate of App.Create's preflight); ValidateConfig now only validates
the project name and delegates to GlobalImpl.ValidateConfig. Remove unused
os/path/filepath imports.
- pkg/app/create.go: add writeScaffoldFile helper that uses O_CREATE|O_EXCL
when force=false, so a file appearing between the preflight check and the
actual write is caught rather than silently overwritten (TOCTOU protection).
- pkg/config/create_test.go: remove four file-existence tests that tested the
now-deleted ValidateConfig logic; file-conflict coverage remains in
pkg/app/create_test.go. Simplify ValidName and GlobalColorConflict tests.
Agent-Logs-Url: https://github.com/helmfile/helmfile/sessions/82f82e72-934f-416c-8662-5060e92284fa
Co-authored-by: yxxhero <11087727+yxxhero@users.noreply.github.com>
* fix: wrap O_EXCL error with --force hint; add writeScaffoldFile unit tests
- pkg/app/create.go: wrap os.IsExist error from writeScaffoldFile with a
message that names the conflicting file and suggests --force, so the user
gets actionable output even in the TOCTOU case
- pkg/app/create_test.go: add TestWriteScaffoldFile_CreatesNewFile,
TestWriteScaffoldFile_ExistingFileNoForce, and
TestWriteScaffoldFile_ExistingFileWithForce to cover the helper directly
Agent-Logs-Url: https://github.com/helmfile/helmfile/sessions/82f82e72-934f-416c-8662-5060e92284fa
Co-authored-by: yxxhero <11087727+yxxhero@users.noreply.github.com>
* fix: wrap App.Create errors in *app.Error; reject '.' as project name; add '.' name test
- pkg/app/create.go: wrap all App.Create fmt.Errorf returns with appError("", ...)
so toCLIError produces a clean user-friendly message instead of
"unexpected error: *fmt.wrapError: ..."
- pkg/config/create.go: reject "." as a NAME alongside ".." to prevent
accidentally scaffolding into the current directory via a named argument
- pkg/config/create_test.go: add TestCreateImpl_ValidateConfig_NameDot
Agent-Logs-Url: https://github.com/helmfile/helmfile/sessions/6d64508e-2d66-47e9-a02a-7669a2f481b7
Co-authored-by: yxxhero <11087727+yxxhero@users.noreply.github.com>
* fix: drop unused outputDir param from test helper to fix unparam lint error
Agent-Logs-Url: https://github.com/helmfile/helmfile/sessions/11cd65e9-c5ef-4195-9375-bc929169616b
Co-authored-by: yxxhero <11087727+yxxhero@users.noreply.github.com>
* fix: drop unused force param from test helper to fix unparam lint error
Agent-Logs-Url: https://github.com/helmfile/helmfile/sessions/0e1bdac5-708f-4615-ae6d-e22fc1e921f2
Co-authored-by: yxxhero <11087727+yxxhero@users.noreply.github.com>
---------
Signed-off-by: yxxhero <aiopsclub@163.com>
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
* fix: eliminate os.Chdir in sequential helmfiles to fix relative path resolution
The sequential code path used within() → os.Chdir() to change the
process-wide working directory when processing helmfile.d files.
This broke relative environment variable paths (e.g. KUBECONFIG=kubeconfig.yaml)
because they resolved from the wrong directory after chdir.
Replace the chdir-based approach with the same baseDir parameter pattern
used by the parallel code path, passing explicit directory context through
loadDesiredStateFromYamlWithBaseDir() instead of mutating global process state.
Closes#2409
Signed-off-by: Aditya Menon <amenon@canarytechnologies.com>
* fix: restore within() for single-file sequential to preserve chart path format
The previous approach used baseDir for all sequential processing, which
changed chart path format in output (e.g. from "../../../../charts/raw"
to "test/integration/charts/raw"). This broke integration tests that
compare chart paths in expected output.
Now the sequential branch uses two strategies:
- Single file: use os.Chdir via within() to preserve backward-compatible
relative chart paths in output
- Multiple files with --sequential-helmfiles: use baseDir parameter to
avoid os.Chdir, fixing relative env var paths like KUBECONFIG (#2409)
Signed-off-by: Aditya Menon <amenon@canarytechnologies.com>
* fix: revert e2e snapshot outputs to match within() behavior
The previous commit restored within() for single-file sequential
processing, which produces relative chart paths (e.g. ../../charts/raw)
and filename-only FilePath. Revert the e2e snapshot expected outputs
to match main branch since single-file behavior is now identical.
Signed-off-by: Aditya Menon <amenon@canarytechnologies.com>
* fix: restructure integration test for multi-file sequential processing
- Point -f at helmfile.d/ directly (not parent dir) so findDesiredStateFiles
discovers the yaml files
- Add second helmfile to trigger baseDir path (len > 1)
- Inline environment config to avoid base file relative path issues
- Verify both releases appear in output instead of comparing with parallel
(which may differ in ordering)
Signed-off-by: Aditya Menon <amenon@canarytechnologies.com>
* fix: reduce cognitive complexity and improve accuracy of sequential helmfiles
Replace inline visitSubHelmfiles closure with calls to the existing
processNestedHelmfiles() method, matching the parallel path. This
eliminates duplicated nested logic and reduces gocognit complexity
below the CI threshold of 110. Also fixes help text and docs to
accurately describe that single-file processing still uses within(),
and adds kubeContext verification to the integration test.
Signed-off-by: Aditya Menon <amenon@canarytechnologies.com>
* test: validate kubeContext resolution in sequential helmfiles integration test
Restructure the integration test to replicate the exact user scenario
from issue #2409:
- Multiple files in helmfile.d/ using bases: with relative paths
(../bases/) for environments and defaults
- Environment values set kubeContext via .Environment.Values
- helmDefaults.kubeContext rendered from gotmpl
- Local chart references (../../../../charts/raw) from helmfile.d/
- Run diff against the minikube cluster to exercise kubeContext
resolution, which would fail with "context does not exist" if
os.Chdir() broke relative path resolution
- Also verify template output for both releases and relative values
file (values/common.yaml) resolution
Fix normalizeChart() in util.go to be idempotent — skip re-prefixing
when the chart path already starts with basePath. This prevents
double-prefixing of local chart paths (e.g. helmfile.d/test/.../raw)
when normalizeChart is called multiple times (once during chart
preparation and again during diff/sync).
Signed-off-by: Aditya Menon <amenon@canarytechnologies.com>
---------
Signed-off-by: Aditya Menon <amenon@canarytechnologies.com>
toCLIError() panics on unhandled error types (e.g. helmexec.ExitError
from a failed helm plugin install). On Windows, plugin install hooks
often fail due to missing 'sh', causing helmfile init to crash even
when the plugin binary was placed correctly.
- Add helmexec.ExitError case to toCLIError and replace panic in the
default case with a graceful error return
- After AddPlugin/UpdatePlugin errors, verify whether the plugin is
actually present before failing; log a warning and continue if so
Fixes#1983
Signed-off-by: Aditya Menon <amenon@canarytechnologies.com>
Adds a new `helmfile unittest` command that integrates the helm-unittest
plugin, allowing users to define unit test paths per release and run them
via helmfile.
Closes#2376
Signed-off-by: Aditya Menon <amenon@canarytechnologies.com>
* Initial plan
* Fix helmfile.d parallel processing to respect alphabetical order
Changed multiple helmfile.d file processing from parallel to sequential
to match the documented behavior. Files are now processed in alphabetical
order as stated in the documentation, ensuring predictable deployment order.
This fixes issues where deployments would break because parallel processing
did not respect the intended ordering (e.g., databases before applications).
- Modified visitStatesWithContext to process files sequentially
- Removed unused processStateFileParallel function
- Updated test snapshots to reflect deterministic ordering
Co-authored-by: yxxhero <11087727+yxxhero@users.noreply.github.com>
* Add --sequential-helmfiles flag for opt-in sequential processing
Based on feedback, added a flag to control processing mode instead of
forcing sequential behavior. Parallel processing is now the default for
performance, with an opt-in --sequential-helmfiles flag for when order matters.
Changes:
- Added SequentialHelmfiles field to GlobalOptions and App
- Added --sequential-helmfiles CLI flag
- Modified visitStatesWithContext to check flag and choose parallel vs sequential
- Updated documentation to clarify parallel is default, sequential is opt-in
- Restored original parallel processing as default behavior
This gives users control over the behavior based on their needs.
Co-authored-by: yxxhero <11087727+yxxhero@users.noreply.github.com>
* Add documentation and improve code readability
- Added documentation to processStateFileParallel function
- Extracted shouldProcessInParallel variable for clarity
- All tests pass, security scan clean
Co-authored-by: yxxhero <11087727+yxxhero@users.noreply.github.com>
* Fix lint issue: remove trailing whitespace
Removed trailing whitespace from line 983 in pkg/app/app.go to fix
formatting lint error.
Co-authored-by: yxxhero <11087727+yxxhero@users.noreply.github.com>
---------
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: yxxhero <11087727+yxxhero@users.noreply.github.com>
This commit adds comprehensive support for Helm 4 while maintaining
full backward compatibility with Helm 3. The implementation includes:
- Updated helm version detection to support both Helm 3 and Helm 4
- Added HELMFILE_HELM4 environment variable to control Helm version
- Modified helm execution paths to handle version-specific binaries
- Updated helm plugin installation to support split architecture
- Helm 4: Uses split plugin architecture (3 separate .tgz files)
- helm-secrets.tgz
- helm-secrets-getter.tgz
- helm-secrets-post-renderer.tgz
- Helm 3: Continues using single plugin installation
- Updated Dockerfiles, CI workflows, and core installation code
- Helm 4 requires post-renderers to be plugins, not executable scripts
- Created Helm plugin structure for integration tests
- Updated helmfile.yaml templates to dynamically select renderer type
- Added test plugins: add-cm, add-cm1, add-cm2
- Updated integration tests for Helm 3/4 compatibility
- Created Helm 4 variant expected output files
- Fixed test determinism issues (repo cleanup between iterations)
- Added version-specific output filtering for warnings/messages
- Updated workflows to test both Helm 3 and Helm 4
- Matrix testing across Helm versions
- Updated helm-diff to v3.14.0 for compatibility
- Updated README and docs with Helm 4 information
- Added migration guidance
- Updated version requirements
All changes are backward compatible - existing Helm 3 users will
see no behavior changes.
fix: update Helm 4 lint expected output to match filtered output
The grep filter removes the semver warning, so the expected output
should not include it. Updated lint-helm4 files to match the filtered
output (warning removed, no extra blank line).
Signed-off-by: Aditya Menon <amenon@canarytechnologies.com>
This pr fixes auto-wrapping of booleans and integers into quotes when using --state-values-set by:
- Adding: --state-values-set-string flag for intentional string set of boolean or integer
- Changing: --state-values-set flag not wrapping now
- Removing -
Resolves https://github.com/roboll/helmfile/issues/1347
Signed-off-by: Tunahan Sezen <sezentunahan@outlook.com>
This is a successor to #442 rebuilt on top of #594 so that we can merge this while we are still at Hemlfile v0.x without worrying any backward-incompatibility.
Signed-off-by: Yusuke Kuoka <ykuoka@gmail.com>
Signed-off-by: Yusuke Kuoka <ykuoka@gmail.com>
* feat: Helmfile V1 mode
We add a new "V1 mode" to Helmfile so that you can seemlessly upgrade Helmfile from the current v0.x to the upcoming v1.0.
The idea is that we build both v0 and v1 binaries from the same tagged commit within the main branch, with different defaults for the "V1 mode"- the V1 mode is disabled by default for v0.x binaries, while it is enabled by default for v1.x binaries.
The V1 mode can be overrode at runtime via envvar. That is, even after upgrading the binary to v1, you will not see any backward-incompatible changes while you explicitly set an envvar, `HELMFILE_V1MODE=true`, at runtime.
Signed-off-by: Yusuke Kuoka <ykuoka@gmail.com>
Signed-off-by: yxxhero <aiopsclub@163.com>
* feat: show live output from the Helm binary
Signed-off-by: Rodrigo Fior Kuntzer <rodrigo@miro.com>
* fixup! Merge branch 'main' into enable-live-output
Signed-off-by: Yusuke Kuoka <ykuoka@gmail.com>
* add interactive in sync & remove --interactive in global options
Signed-off-by: yxxhero <aiopsclub@163.com>
* fix unittest
Signed-off-by: yxxhero <aiopsclub@163.com>
* same behave as apply when in interactive
Signed-off-by: yxxhero <aiopsclub@163.com>
Signed-off-by: yxxhero <aiopsclub@163.com>
