* feat: parallel kubedog tracking with progress printer and safety valves
Rework kubedog integration so resource tracking runs in parallel with
helm upgrade/install, giving live progress output and recovering from
known helm/kubedog wedge conditions.
Core:
- kubedogTrackingHandle runs tracking in a background goroutine alongside
the helm subprocess (startBackgroundKubedogTracking); helm output is
buffered and replayed as a single block so it no longer interleaves
with progress ticks.
- Capture UID+generation baselines before handing off to helm so each
tracker waits until the resource actually changes (freshness gate).
Progress printer (pkg/kubedog/printer.go):
- Styled, auto-sized progress table with a heartbeat flusher, child (pod)
status roll-up, pre-ready pod-phase handling, multi-namespace support,
and optional color. PreviewBreakdown summarizes kept/filtered resources.
Safety valves (verify cluster state via the live API):
- Tracker-race valve (always on): when helm succeeds but a dyntracker
goroutine is wedged, poll the API and cancel the tracker so wait()
returns success instead of blocking until --track-timeout.
- Helm-stuck killer (opt-in via helmStuckGrace): if the cluster stays
converged while helm v4's hook waiter is wedged, SIGINT the helm
subprocess to recover.
- Failure watchdog (pkg/kubedog/watchdog.go): surface failing pods that
never made it into dyntracker's resource graph.
Options: trackFailedLogs, helmStuckGrace, trackTimeout, color
(Color/NoColor), and resource filtering (trackKinds/skipKinds/
trackResources). PersistentVolumeClaim support in resource classification.
Signed-off-by: Roman Mykhailiuk <romanm@cybellum.com>
* test: add unit tests for kubedog tracking
Cover the progress printer, resource classification, the failure
watchdog, helm-output trimming/dedup, the release hard-timeout helper,
and the color/track option plumbing.
Signed-off-by: Roman Mykhailiuk <romanm@cybellum.com>
* test: update golden logs, e2e snapshots, and values-id fixtures
Refresh pkg/app testapply/testdestroy golden logs, e2e template
snapshots, and the TestGenerateID values-id golden hashes for the new
kubedog progress output and the merged release struct layout.
Signed-off-by: Roman Mykhailiuk <romanm@cybellum.com>
* refactor: remove dead kubedog display code and dedupe tracker setup
Deep-review pass on the parallel kubedog tracking changes:
- Remove pkg/kubedog/display.go (308 lines) and display_test.go (453 lines).
These rendered progress for the legacy per-kind trackers that this PR
replaces; in the merged tree every function is unreferenced outside its
own tests. The new progressPrinter (printer.go) supersedes them.
TestMain (color.ForceColor for deterministic ANSI in tests) is preserved
in a new main_test.go.
- Dedupe trackWithKubedog: the post-helm fallback rebuilt the exact same
tracker options as buildReleaseTracker. Reuse buildReleaseTracker instead,
dropping ~40 lines of duplicated timeout/log/filter/tracker construction.
No behavior change; build, go vet, golangci-lint, and the kubedog/state
unit tests all pass.
Signed-off-by: yxxhero <aiopsclub@163.com>
* fix: stop waitForFreshness busy-looping the API after helm finishes
Once upstreamDoneCh closes it is always ready, so the select in
waitForFreshness stopped blocking on the ticker and re-ran probe() (a
live GET) as fast as the round-trip allowed for the whole 3s grace
window — hammering the API server once per tracked resource.
Track a local view of the channel and nil it out on first delivery so
the first hit records the timestamp (one fast retry, as intended) and
all subsequent polls are ticker-throttled. Functional behavior is
unchanged: return nil when fresh, errUpstreamDoneNoChange after grace.
Signed-off-by: yxxhero <aiopsclub@163.com>
* test: drop trailing blank line from helm4 OCI pull snapshots
The trailing-newline trim in helmexec.info() removes the blank line helm
prints after the OCI chart "Digest:" line. The helm3 (output.yaml)
snapshots never captured that line, but the helm4 (output-helm4.yaml)
snapshots for oci_chart_pull{,_direct,_once,_once2} and
issue_473_oci_chart_url_fetch still expected it, so they failed under
helm 4. Remove the blank line so the snapshots match the trimmed output.
Signed-off-by: yxxhero <aiopsclub@163.com>
* test: refresh diff-args integration goldens for styled release headers
DisplayAffectedReleases now emits a styled "========== Updated Releases
==========" header (matching the app/e2e goldens already updated by this
PR) and helmexec.info() trims the trailing blank after helm's install
status. Update the diff-args apply-stderr{,-helm4} and apply-live-
stderr{,-helm4} goldens accordingly so they match the actual stderr.
Signed-off-by: yxxhero <aiopsclub@163.com>
* test: drop trimmed blank line from v1-subhelmfile template golden
The trailing-newline trim in helmexec.info() removes the blank line helm
prints after '"incubator" has been added to your repositories'. Update
the v1-subhelmfile-multi-bases-with-array-values result and result-live
goldens so the template stdout comparison matches.
Signed-off-by: yxxhero <aiopsclub@163.com>
---------
Signed-off-by: Roman Mykhailiuk <romanm@cybellum.com>
Signed-off-by: yxxhero <aiopsclub@163.com>
Co-authored-by: Roman Mykhailiuk <romanm@cybellum.com>
* feat: add support for helm 4 --server-side upgrade flag
Add support for the helm 4 upgrade flag --server-side which accepts
"true", "false", or "auto" (default "auto"). This allows users to
explicitly control server-side apply behavior, which is needed for
releases originally installed with Helm 3 and being managed with Helm 4.
The flag can be configured via:
- CLI: --server-side flag on sync, apply, and diff commands
- helmDefaults.serverSide in helmfile.yaml
- releases[].serverSide per-release override
Precedence: release-level > CLI flag > helmDefaults.
Errors are returned when serverSide is set but running Helm 3, or when
an invalid value is provided.
Closes#2640
Signed-off-by: yxxhero <aiopsclub@163.com>
* test: update TestGenerateID expected hashes for new ServerSide field
Adding ServerSide *string to ReleaseSpec changes spew's %#v output and
shifts the FNV hash used by generateValuesID. Update the hard-coded want
values to the new deterministic hashes.
Signed-off-by: yxxhero <aiopsclub@163.com>
---------
Signed-off-by: yxxhero <aiopsclub@163.com>
* fix: add trackFailOnError option to control kubedog exit code behavior
When kubedog release tracking fails (e.g. pod ImagePullBackOff), helmfile
exits with code 0 instead of a non-zero exit code. Add a trackFailOnError
configuration option (default: false) that when set to true, propagates
kubedog tracking failures to the exit code.
The option is available as:
- Per-release YAML: trackFailOnError: true
- CLI flag: --track-fail-on-error (sync and apply commands)
Extract trackReleaseIfEnabled helper to consolidate kubedog tracking logic
from two duplicated call sites into a single maintainable method.
Fixes#2507
Signed-off-by: yxxhero <aiopsclub@163.com>
* fix: add //go:build ignore to server.go to fix go test CI failure
The test/integration/test-cases/issue-2103/input/server.go is a
package main helper binary used by the issue-2103 integration test.
When go test -coverprofile runs on this package, it fails with
"go: no such tool covdata" in the CI environment.
Adding //go:build ignore excludes the file from go list ./... (and
therefore from PKGS in the Makefile), while still allowing the
integration test to build it explicitly via file path:
go build -o server ./path/to/server.go
Agent-Logs-Url: https://github.com/helmfile/helmfile/sessions/8a7000af-72b7-48f8-8a82-24813b5df341
Co-authored-by: yxxhero <11087727+yxxhero@users.noreply.github.com>
* fix: update TestGenerateID expected hashes after adding TrackFailOnError field
Adding TrackFailOnError *bool to ReleaseSpec changed the spew
serialization of the struct, which changed the FNV-32a hash values
produced by generateValuesID. Update temp_test.go with the new
expected hash strings.
Agent-Logs-Url: https://github.com/helmfile/helmfile/sessions/caa86cd9-73d1-4894-b745-fd70c0811fd6
Co-authored-by: yxxhero <11087727+yxxhero@users.noreply.github.com>
---------
Signed-off-by: yxxhero <aiopsclub@163.com>
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Add support for trackMode: helm-legacy to use Helm v4's --wait=legacy flag,
which maintains compatibility with Helm v3's wait behavior during migration.
Helm v4 changed the default --wait behavior from polling to a watcher-based
approach. This can cause issues with charts that have broken livenessProbe
configurations without startupProbe. The --wait=legacy flag preserves the
Helm v3 polling behavior for smoother migration.
Changes:
- Add TrackModeHelmLegacy constant in pkg/kubedog/options.go
- Use kubedog.TrackMode constants instead of raw strings in helmx.go
- Enhance appendWaitFlags to use --wait=legacy for Helm v4 when trackMode
is helm-legacy
- Add nil check for logger before logging warning
- Add version check with warning when helm-legacy is used with Helm v3
- Update validation in pkg/config to accept helm-legacy track mode
- Update command-line flags in cmd/apply.go and cmd/sync.go
- Add comprehensive documentation in docs/advanced-features.md
- Add thorough test coverage including warning message verification
Behavior:
- Helm v4 + helm-legacy: Uses --wait=legacy
- Helm v3 + helm-legacy: Falls back to --wait with warning
- Helm v4 + helm: Uses --wait (watcher mode)
- Any + kubedog: Skips --wait flag
Fixes#2464
Signed-off-by: yxxhero <aiopsclub@163.com>
Co-authored-by: Copilot <copilot@github.com>
* Initial plan
* Implement --timeout flag for helmfile sync command
- Add Timeout field to SyncOptions struct in pkg/config/sync.go
- Add --timeout flag to sync command in cmd/sync.go
- Add Timeout field to SyncOpts struct in pkg/state/state.go
- Modify timeoutFlags() function to prioritize CLI timeout over release and default configs
- Add test case to verify CLI timeout overrides other timeout settings
- Follow same pattern as existing --wait and --wait-for-jobs flags
Co-authored-by: yxxhero <11087727+yxxhero@users.noreply.github.com>
* Fix lint issues: format test struct fields properly
Co-authored-by: yxxhero <11087727+yxxhero@users.noreply.github.com>
* Update docs: Add --timeout flag documentation for helmfile sync command
Co-authored-by: yxxhero <11087727+yxxhero@users.noreply.github.com>
---------
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: yxxhero <11087727+yxxhero@users.noreply.github.com>
* Add "PostRendererArgs" option to be passed to helm
This allows using PowerShell scripts on Windows as Post Renderer.
Signed-off-by: Maarten Boekhold <maarten.boekhold@finastra.com>
Ref https://github.com/helmfile/helmfile/discussions/593
* implemented --reset-values flag overriding .helmDefaults.reuseValues=true
* tests for --reset-values flag
* fixed pkg/app tests
* added tests for both reset & reuse flags given
* added appendValuesControlMode method for --reuse/reset-values flag processing
* updated code comments & docs
Signed-off-by: Karol Ossowski <k@koralsky.pl>
1. only implement post-renderer flags this patch
2. As mumoshu advise, add helmfile flags `--post-render` and add the
postRenderer config in helmDefaults and release. the priority is
helmfile flags > release > helmDefaults.
3. fix the test case in state_test.go and some other tests.
Signed-off-by: guofutan <guofutan@tencent.com>
Signed-off-by: yxxhero <aiopsclub@163.com>
* add interactive in sync & remove --interactive in global options
Signed-off-by: yxxhero <aiopsclub@163.com>
* fix unittest
Signed-off-by: yxxhero <aiopsclub@163.com>
* same behave as apply when in interactive
Signed-off-by: yxxhero <aiopsclub@163.com>
Signed-off-by: yxxhero <aiopsclub@163.com>
