* fix: support go-getter URLs in ad-hoc dependencies to fix#821
Ad-hoc release dependencies (release.dependencies[].chart) that used
go-getter URLs like "git::https://host/repo.git@path?ref=tag" were passed
to chartify as-is. chartify then tried to resolve them via `helm repo
list`, which fails with "no helm list entry found for repository
\"git::https:\". please `helm repo add` it!".
The primary chart already fetched such URLs via downloadChartWithGoGetter,
but the ad-hoc dependency path in PrepareChartify only handled local
directories and OCI rewrites, so go-getter URLs fell through.
This adds a branch that detects remote go-getter URLs (remote.IsRemote)
and fetches them to a local cache directory via a new
downloadAdhocDepChartWithGoGetter helper, mirroring the primary-chart
fetch path. chartify then sees a local chart and takes its file:// branch.
Fixes#821
Signed-off-by: yxxhero <aiopsclub@163.com>
* test: add integration test for go-getter ad-hoc dependencies (#821)
Adds an integration test that commits a chart to a throwaway local git repo
and references it via a "git::file://..." go-getter URL as a release
ad-hoc dependency, then asserts `helmfile template` renders both the main
chart and the fetched dependency.
Using file:// (rather than https://) keeps the test deterministic and
network-free while exercising the exact fix path (remote.IsRemote +
downloadAdhocDepChartWithGoGetter in PrepareChartify). Verified to fail on
the unfixed code with the original "no helm list entry found for repository
\"git::file:\"" error and pass on the fixed code.
Signed-off-by: yxxhero <aiopsclub@163.com>
* test: surface helmfile error in issue #821 integration test
The integration runner (run.sh) enables `set -e`, so a non-zero helmfile
exit aborted the script before the captured output could be printed,
hiding the real failure in CI. Disable `set -e` around the helmfile
invocation and report the exit code plus full output on failure.
Signed-off-by: yxxhero <aiopsclub@163.com>
* test: fix issue-821 integration test chart path and errexit handling
Two issues caused the integration test to fail in CI (while passing in my
local smoke test, which used an absolute path):
1. The main chart path was relative to the integration CWD, but the
helmfile.yaml is generated in a temp directory and helmfile resolves
`chart:` relative to that directory (its basePath). The relative path
was interpreted as a named-repo chart and failed instantly with
`Error: repo test not found`. Resolve the case dir to an absolute path
with `$(cd ... && pwd)`.
2. run.sh runs under `set -e`, so the unguarded helmfile invocation exited
the whole script on failure before the captured output could be printed,
hiding the real error. Capture the exit code via `cmd || rc=$?` so
failures are reported.
Signed-off-by: yxxhero <aiopsclub@163.com>
---------
Signed-off-by: yxxhero <aiopsclub@163.com>
* feat: parallel kubedog tracking with progress printer and safety valves
Rework kubedog integration so resource tracking runs in parallel with
helm upgrade/install, giving live progress output and recovering from
known helm/kubedog wedge conditions.
Core:
- kubedogTrackingHandle runs tracking in a background goroutine alongside
the helm subprocess (startBackgroundKubedogTracking); helm output is
buffered and replayed as a single block so it no longer interleaves
with progress ticks.
- Capture UID+generation baselines before handing off to helm so each
tracker waits until the resource actually changes (freshness gate).
Progress printer (pkg/kubedog/printer.go):
- Styled, auto-sized progress table with a heartbeat flusher, child (pod)
status roll-up, pre-ready pod-phase handling, multi-namespace support,
and optional color. PreviewBreakdown summarizes kept/filtered resources.
Safety valves (verify cluster state via the live API):
- Tracker-race valve (always on): when helm succeeds but a dyntracker
goroutine is wedged, poll the API and cancel the tracker so wait()
returns success instead of blocking until --track-timeout.
- Helm-stuck killer (opt-in via helmStuckGrace): if the cluster stays
converged while helm v4's hook waiter is wedged, SIGINT the helm
subprocess to recover.
- Failure watchdog (pkg/kubedog/watchdog.go): surface failing pods that
never made it into dyntracker's resource graph.
Options: trackFailedLogs, helmStuckGrace, trackTimeout, color
(Color/NoColor), and resource filtering (trackKinds/skipKinds/
trackResources). PersistentVolumeClaim support in resource classification.
Signed-off-by: Roman Mykhailiuk <romanm@cybellum.com>
* test: add unit tests for kubedog tracking
Cover the progress printer, resource classification, the failure
watchdog, helm-output trimming/dedup, the release hard-timeout helper,
and the color/track option plumbing.
Signed-off-by: Roman Mykhailiuk <romanm@cybellum.com>
* test: update golden logs, e2e snapshots, and values-id fixtures
Refresh pkg/app testapply/testdestroy golden logs, e2e template
snapshots, and the TestGenerateID values-id golden hashes for the new
kubedog progress output and the merged release struct layout.
Signed-off-by: Roman Mykhailiuk <romanm@cybellum.com>
* refactor: remove dead kubedog display code and dedupe tracker setup
Deep-review pass on the parallel kubedog tracking changes:
- Remove pkg/kubedog/display.go (308 lines) and display_test.go (453 lines).
These rendered progress for the legacy per-kind trackers that this PR
replaces; in the merged tree every function is unreferenced outside its
own tests. The new progressPrinter (printer.go) supersedes them.
TestMain (color.ForceColor for deterministic ANSI in tests) is preserved
in a new main_test.go.
- Dedupe trackWithKubedog: the post-helm fallback rebuilt the exact same
tracker options as buildReleaseTracker. Reuse buildReleaseTracker instead,
dropping ~40 lines of duplicated timeout/log/filter/tracker construction.
No behavior change; build, go vet, golangci-lint, and the kubedog/state
unit tests all pass.
Signed-off-by: yxxhero <aiopsclub@163.com>
* fix: stop waitForFreshness busy-looping the API after helm finishes
Once upstreamDoneCh closes it is always ready, so the select in
waitForFreshness stopped blocking on the ticker and re-ran probe() (a
live GET) as fast as the round-trip allowed for the whole 3s grace
window — hammering the API server once per tracked resource.
Track a local view of the channel and nil it out on first delivery so
the first hit records the timestamp (one fast retry, as intended) and
all subsequent polls are ticker-throttled. Functional behavior is
unchanged: return nil when fresh, errUpstreamDoneNoChange after grace.
Signed-off-by: yxxhero <aiopsclub@163.com>
* test: drop trailing blank line from helm4 OCI pull snapshots
The trailing-newline trim in helmexec.info() removes the blank line helm
prints after the OCI chart "Digest:" line. The helm3 (output.yaml)
snapshots never captured that line, but the helm4 (output-helm4.yaml)
snapshots for oci_chart_pull{,_direct,_once,_once2} and
issue_473_oci_chart_url_fetch still expected it, so they failed under
helm 4. Remove the blank line so the snapshots match the trimmed output.
Signed-off-by: yxxhero <aiopsclub@163.com>
* test: refresh diff-args integration goldens for styled release headers
DisplayAffectedReleases now emits a styled "========== Updated Releases
==========" header (matching the app/e2e goldens already updated by this
PR) and helmexec.info() trims the trailing blank after helm's install
status. Update the diff-args apply-stderr{,-helm4} and apply-live-
stderr{,-helm4} goldens accordingly so they match the actual stderr.
Signed-off-by: yxxhero <aiopsclub@163.com>
* test: drop trimmed blank line from v1-subhelmfile template golden
The trailing-newline trim in helmexec.info() removes the blank line helm
prints after '"incubator" has been added to your repositories'. Update
the v1-subhelmfile-multi-bases-with-array-values result and result-live
goldens so the template stdout comparison matches.
Signed-off-by: yxxhero <aiopsclub@163.com>
---------
Signed-off-by: Roman Mykhailiuk <romanm@cybellum.com>
Signed-off-by: yxxhero <aiopsclub@163.com>
Co-authored-by: Roman Mykhailiuk <romanm@cybellum.com>
Add appendServerSideFlagsForDiff to validate helm-diff plugin version
(v3.15.10+) before passing the --server-side flag to helm diff upgrade.
Extract resolveServerSideValue helper to share precedence logic between
upgrade and diff paths. Bump helm-diff recommended version to v3.15.10
across Dockerfiles, CI, and integration scripts.
Signed-off-by: yxxhero <aiopsclub@163.com>
* feat: add support for helm 4 --server-side upgrade flag
Add support for the helm 4 upgrade flag --server-side which accepts
"true", "false", or "auto" (default "auto"). This allows users to
explicitly control server-side apply behavior, which is needed for
releases originally installed with Helm 3 and being managed with Helm 4.
The flag can be configured via:
- CLI: --server-side flag on sync, apply, and diff commands
- helmDefaults.serverSide in helmfile.yaml
- releases[].serverSide per-release override
Precedence: release-level > CLI flag > helmDefaults.
Errors are returned when serverSide is set but running Helm 3, or when
an invalid value is provided.
Closes#2640
Signed-off-by: yxxhero <aiopsclub@163.com>
* test: update TestGenerateID expected hashes for new ServerSide field
Adding ServerSide *string to ReleaseSpec changes spew's %#v output and
shifts the FNV hash used by generateValuesID. Update the hard-coded want
values to the new deterministic hashes.
Signed-off-by: yxxhero <aiopsclub@163.com>
---------
Signed-off-by: yxxhero <aiopsclub@163.com>
When helmfile.d contains multiple release files and one release has a
local chart dependency (e.g. chart: ../chart), the dependency path was
passed to DirectoryExistsAt without normalizing against basePath. This
caused the path to be resolved against CWD instead of the helmfile
directory, so the local chart was not detected and helmfile tried to
resolve it as a remote repo, failing with:
'failed reading adhoc dependencies: no helm list entry found for repository'
Fixes#2596
Signed-off-by: yxxhero <aiopsclub@163.com>
PR #1839 introduced template rendering for postRendererArgs, but PR #2510
reverted it while fixing a separate regression. This left helmDefaults-level
postRendererArgs containing template expressions (e.g. {{ .Release.Name }})
passed to helm as literal strings instead of being resolved per-release.
Add renderPostRendererArgs() that templates helmDefaults.postRendererArgs
at flag-generation time using the release's template data, reusing the
existing createReleaseTemplateData() helper. Release-level args are already
templated by ExecuteTemplateExpressions and CLI args are static, so only
the helmDefaults path needs rendering.
Fixes#2580
Signed-off-by: opencode <opencode@users.noreply.github.com>
Signed-off-by: yxxhero <aiopsclub@163.com>
When a release `dependencies[].chart` is given as `<repoName>/<chart>`
and the matching `repositories:` entry has `oci: true`, helmfile now
rewrites it to `oci://<repoURL>/<chart>` before passing it to chartify.
Without this, chartify's lookup falls into its `helm repo list` branch,
which never finds OCI repos because helm 3+ does not register OCI
registries as named repos (they live in the `helm registry login`
state instead). The user-visible failure was:
failed reading adhoc dependencies: no helm list entry found for
repository "<name>". please `helm repo add` it!
Explicit `oci://` URLs already worked through chartify's OCI branch;
this change makes the `<repoName>/<chart>` form behave the same way.
Non-OCI repo prefixes, unknown prefixes, single-segment names, and
explicit `oci://` URLs all pass through unchanged. A debug log records
each rewrite at the call site for easier troubleshooting.
Fixes#1756.
Signed-off-by: Dominik Schmidt <dev@dominik-schmidt.de>
* fix: add trackFailOnError option to control kubedog exit code behavior
When kubedog release tracking fails (e.g. pod ImagePullBackOff), helmfile
exits with code 0 instead of a non-zero exit code. Add a trackFailOnError
configuration option (default: false) that when set to true, propagates
kubedog tracking failures to the exit code.
The option is available as:
- Per-release YAML: trackFailOnError: true
- CLI flag: --track-fail-on-error (sync and apply commands)
Extract trackReleaseIfEnabled helper to consolidate kubedog tracking logic
from two duplicated call sites into a single maintainable method.
Fixes#2507
Signed-off-by: yxxhero <aiopsclub@163.com>
* fix: add //go:build ignore to server.go to fix go test CI failure
The test/integration/test-cases/issue-2103/input/server.go is a
package main helper binary used by the issue-2103 integration test.
When go test -coverprofile runs on this package, it fails with
"go: no such tool covdata" in the CI environment.
Adding //go:build ignore excludes the file from go list ./... (and
therefore from PKGS in the Makefile), while still allowing the
integration test to build it explicitly via file path:
go build -o server ./path/to/server.go
Agent-Logs-Url: https://github.com/helmfile/helmfile/sessions/8a7000af-72b7-48f8-8a82-24813b5df341
Co-authored-by: yxxhero <11087727+yxxhero@users.noreply.github.com>
* fix: update TestGenerateID expected hashes after adding TrackFailOnError field
Adding TrackFailOnError *bool to ReleaseSpec changed the spew
serialization of the struct, which changed the FNV-32a hash values
produced by generateValuesID. Update temp_test.go with the new
expected hash strings.
Agent-Logs-Url: https://github.com/helmfile/helmfile/sessions/caa86cd9-73d1-4894-b745-fd70c0811fd6
Co-authored-by: yxxhero <11087727+yxxhero@users.noreply.github.com>
---------
Signed-off-by: yxxhero <aiopsclub@163.com>
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
* fix: use --post-renderer-args=VALUE format to prevent Helm flag parsing failure
When postRendererArgs contains values like short flags (e.g. -v), passing
--post-renderer-args and the value as separate arguments causes Helm to
interpret the value as its own flag. Using the --post-renderer-args=VALUE
format unambiguously binds the value to the flag.
Fixes#2563
Signed-off-by: yxxhero <aiopsclub@163.com>
* fix: update hasFlagWithValue doc/errors and add -v short-flag test cases
- Update hasFlagWithValue doc comment to describe both '--flag value' and
'--flag=value' forms
- Update t.Errorf messages in app_test.go to reflect both accepted formats
- Add 'post-renderer-args-short-flag-value' test case (-v) to both
TestHelmState_flagsForUpgrade and TestHelmState_flagsForTemplate to
verify --post-renderer-args=-v emission (core regression from #2563)
Agent-Logs-Url: https://github.com/helmfile/helmfile/sessions/dd95f046-358b-4867-9069-9432c1b5318e
Co-authored-by: yxxhero <11087727+yxxhero@users.noreply.github.com>
---------
Signed-off-by: yxxhero <aiopsclub@163.com>
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
* fix: helmfile fetch fails for kustomization directories
Fixes#2503
When running `helmfile fetch` on a release that points to a local
kustomization directory (without Chart.yaml), the command failed with
"Chart.yaml is missing".
The issue was that the condition `helmfileCommand != "pull"` in
prepareChartForRelease skipped chartification for ALL cases during
fetch, including local kustomization directories that NEED chartify
to convert them to Helm charts.
Solution:
- Added `NeedsChartifyForLocalDir` field to the Chartify struct to
track when chartification is needed because the local directory
is not a Helm chart (no Chart.yaml)
- Modified the condition to skip chartification for "pull" ONLY when
it's not a local directory without Chart.yaml
This preserves the original fix (commit 1f134d93) for remote charts
with transformers while fixing local kustomization directories.
Signed-off-by: yxxhero <aiopsclub@163.com>
* test: add integration test for helmfile fetch with kustomization
Add test case for issue #2503 to verify helmfile fetch works correctly
with local kustomization directories (without Chart.yaml).
Signed-off-by: yxxhero <aiopsclub@163.com>
---------
Signed-off-by: yxxhero <aiopsclub@163.com>
* fix: use --force-replace flag for Helm 4 instead of deprecated --force
Helm 4 deprecated the --force flag in favor of --force-replace.
This fix detects the Helm version and uses the appropriate flag:
- Helm 4: --force-replace
- Helm 3: --force
Also fixed a nil pointer panic in appendHideNotesFlags when called
with nil SyncOpts.
Fixes#2476
Signed-off-by: yxxhero <aiopsclub@163.com>
* fix(ci): pin semver to v2.12.0 for Go 1.25 compatibility
semver@latest requires Go 1.26.1 but the project uses Go 1.25.4.
Pinning to v2.12.0 which is compatible with Go 1.25.
Signed-off-by: yxxhero <aiopsclub@163.com>
* test: add test cases for force flag from defaults with nil release
Add test cases to cover the scenario where release.Force is nil and
HelmDefaults.Force enables force for both Helm 3 and Helm 4.
Signed-off-by: yxxhero <aiopsclub@163.com>
* test: add nil ops test and rename misleading test names
- Add test case for appendHideNotesFlags with ops=nil to prevent
regression
- Rename force-from-default-nil-release-* to
force-from-default-nil-force-* for clarity (release.Force is nil,
not the release itself)
Signed-off-by: yxxhero <aiopsclub@163.com>
* refactor: add explicit parentheses for force condition
Add explicit parentheses around the two disjuncts in the force
condition to make the intended grouping unambiguous and easier
to read.
Signed-off-by: yxxhero <aiopsclub@163.com>
* refactor: check ops nil before Helm version in appendHideNotesFlags
- Swap the order to check ops == nil first to avoid unnecessary
IsVersionAtLeast call
- Restore the "see Helm release" comment for consistency with other
flag helpers
Signed-off-by: yxxhero <aiopsclub@163.com>
---------
Signed-off-by: yxxhero <aiopsclub@163.com>
Add support for trackMode: helm-legacy to use Helm v4's --wait=legacy flag,
which maintains compatibility with Helm v3's wait behavior during migration.
Helm v4 changed the default --wait behavior from polling to a watcher-based
approach. This can cause issues with charts that have broken livenessProbe
configurations without startupProbe. The --wait=legacy flag preserves the
Helm v3 polling behavior for smoother migration.
Changes:
- Add TrackModeHelmLegacy constant in pkg/kubedog/options.go
- Use kubedog.TrackMode constants instead of raw strings in helmx.go
- Enhance appendWaitFlags to use --wait=legacy for Helm v4 when trackMode
is helm-legacy
- Add nil check for logger before logging warning
- Add version check with warning when helm-legacy is used with Helm v3
- Update validation in pkg/config to accept helm-legacy track mode
- Update command-line flags in cmd/apply.go and cmd/sync.go
- Add comprehensive documentation in docs/advanced-features.md
- Add thorough test coverage including warning message verification
Behavior:
- Helm v4 + helm-legacy: Uses --wait=legacy
- Helm v3 + helm-legacy: Falls back to --wait with warning
- Helm v4 + helm: Uses --wait (watcher mode)
- Any + kubedog: Skips --wait flag
Fixes#2464
Signed-off-by: yxxhero <aiopsclub@163.com>
Co-authored-by: Copilot <copilot@github.com>
This commit adds comprehensive support for Helm 4 while maintaining
full backward compatibility with Helm 3. The implementation includes:
- Updated helm version detection to support both Helm 3 and Helm 4
- Added HELMFILE_HELM4 environment variable to control Helm version
- Modified helm execution paths to handle version-specific binaries
- Updated helm plugin installation to support split architecture
- Helm 4: Uses split plugin architecture (3 separate .tgz files)
- helm-secrets.tgz
- helm-secrets-getter.tgz
- helm-secrets-post-renderer.tgz
- Helm 3: Continues using single plugin installation
- Updated Dockerfiles, CI workflows, and core installation code
- Helm 4 requires post-renderers to be plugins, not executable scripts
- Created Helm plugin structure for integration tests
- Updated helmfile.yaml templates to dynamically select renderer type
- Added test plugins: add-cm, add-cm1, add-cm2
- Updated integration tests for Helm 3/4 compatibility
- Created Helm 4 variant expected output files
- Fixed test determinism issues (repo cleanup between iterations)
- Added version-specific output filtering for warnings/messages
- Updated workflows to test both Helm 3 and Helm 4
- Matrix testing across Helm versions
- Updated helm-diff to v3.14.0 for compatibility
- Updated README and docs with Helm 4 information
- Added migration guidance
- Updated version requirements
All changes are backward compatible - existing Helm 3 users will
see no behavior changes.
fix: update Helm 4 lint expected output to match filtered output
The grep filter removes the semver warning, so the expected output
should not include it. Updated lint-helm4 files to match the filtered
output (warning removed, no extra blank line).
Signed-off-by: Aditya Menon <amenon@canarytechnologies.com>
Add a `--show-only` parameter to the `helmfile template` command to pass
on to the `helm template` command.
Signed-off-by: Jim Barber <jim.barber@healthengine.com.au>
* Add "PostRendererArgs" option to be passed to helm
This allows using PowerShell scripts on Windows as Post Renderer.
Signed-off-by: Maarten Boekhold <maarten.boekhold@finastra.com>
Helmfile has been providing a feature called "adhoc chart dependency" that basially enabled you to add Chart.yaml `dependencies` entry adhocly without forking or modifying the chart.
It was missing the support for using a local chart as the adhoc dependency. This patch adds that.
Usage:
`releases[].dependencies[].chart` is enhanced to accept the fs path to the local chart:
```
releases:
- name: foo
chart: ./path/to/foo
dependencies:
- chart: ./path/to/bar
```
Resolves#1762
Related to #494
This feature is mostly a built-in alternative to the `incubator/raw` chart without external dependency and has
access to helmfile's own template functions and template data.
The expected use-case of this feature is to add arbitrary K8s resources to your deployment.
Unlike the original issue raised in #494 this doesn't enable you to add arbitary resources to a release. That's another story. But this would be a good foundation for that, too.
yxxhero