Files
helmfile/pkg/config/apply.go
T
yxxheroandRoman Mykhailiuk 9d3a9ad6c7 feat: parallel kubedog tracking with progress printer and safety valves (#2654)
* feat: parallel kubedog tracking with progress printer and safety valves

Rework kubedog integration so resource tracking runs in parallel with
helm upgrade/install, giving live progress output and recovering from
known helm/kubedog wedge conditions.

Core:
- kubedogTrackingHandle runs tracking in a background goroutine alongside
  the helm subprocess (startBackgroundKubedogTracking); helm output is
  buffered and replayed as a single block so it no longer interleaves
  with progress ticks.
- Capture UID+generation baselines before handing off to helm so each
  tracker waits until the resource actually changes (freshness gate).

Progress printer (pkg/kubedog/printer.go):
- Styled, auto-sized progress table with a heartbeat flusher, child (pod)
  status roll-up, pre-ready pod-phase handling, multi-namespace support,
  and optional color. PreviewBreakdown summarizes kept/filtered resources.

Safety valves (verify cluster state via the live API):
- Tracker-race valve (always on): when helm succeeds but a dyntracker
  goroutine is wedged, poll the API and cancel the tracker so wait()
  returns success instead of blocking until --track-timeout.
- Helm-stuck killer (opt-in via helmStuckGrace): if the cluster stays
  converged while helm v4's hook waiter is wedged, SIGINT the helm
  subprocess to recover.
- Failure watchdog (pkg/kubedog/watchdog.go): surface failing pods that
  never made it into dyntracker's resource graph.

Options: trackFailedLogs, helmStuckGrace, trackTimeout, color
(Color/NoColor), and resource filtering (trackKinds/skipKinds/
trackResources). PersistentVolumeClaim support in resource classification.

Signed-off-by: Roman Mykhailiuk <romanm@cybellum.com>

* test: add unit tests for kubedog tracking

Cover the progress printer, resource classification, the failure
watchdog, helm-output trimming/dedup, the release hard-timeout helper,
and the color/track option plumbing.

Signed-off-by: Roman Mykhailiuk <romanm@cybellum.com>

* test: update golden logs, e2e snapshots, and values-id fixtures

Refresh pkg/app testapply/testdestroy golden logs, e2e template
snapshots, and the TestGenerateID values-id golden hashes for the new
kubedog progress output and the merged release struct layout.

Signed-off-by: Roman Mykhailiuk <romanm@cybellum.com>

* refactor: remove dead kubedog display code and dedupe tracker setup

Deep-review pass on the parallel kubedog tracking changes:

- Remove pkg/kubedog/display.go (308 lines) and display_test.go (453 lines).
  These rendered progress for the legacy per-kind trackers that this PR
  replaces; in the merged tree every function is unreferenced outside its
  own tests. The new progressPrinter (printer.go) supersedes them.
  TestMain (color.ForceColor for deterministic ANSI in tests) is preserved
  in a new main_test.go.
- Dedupe trackWithKubedog: the post-helm fallback rebuilt the exact same
  tracker options as buildReleaseTracker. Reuse buildReleaseTracker instead,
  dropping ~40 lines of duplicated timeout/log/filter/tracker construction.

No behavior change; build, go vet, golangci-lint, and the kubedog/state
unit tests all pass.

Signed-off-by: yxxhero <aiopsclub@163.com>

* fix: stop waitForFreshness busy-looping the API after helm finishes

Once upstreamDoneCh closes it is always ready, so the select in
waitForFreshness stopped blocking on the ticker and re-ran probe() (a
live GET) as fast as the round-trip allowed for the whole 3s grace
window — hammering the API server once per tracked resource.

Track a local view of the channel and nil it out on first delivery so
the first hit records the timestamp (one fast retry, as intended) and
all subsequent polls are ticker-throttled. Functional behavior is
unchanged: return nil when fresh, errUpstreamDoneNoChange after grace.

Signed-off-by: yxxhero <aiopsclub@163.com>

* test: drop trailing blank line from helm4 OCI pull snapshots

The trailing-newline trim in helmexec.info() removes the blank line helm
prints after the OCI chart "Digest:" line. The helm3 (output.yaml)
snapshots never captured that line, but the helm4 (output-helm4.yaml)
snapshots for oci_chart_pull{,_direct,_once,_once2} and
issue_473_oci_chart_url_fetch still expected it, so they failed under
helm 4. Remove the blank line so the snapshots match the trimmed output.

Signed-off-by: yxxhero <aiopsclub@163.com>

* test: refresh diff-args integration goldens for styled release headers

DisplayAffectedReleases now emits a styled "========== Updated Releases
==========" header (matching the app/e2e goldens already updated by this
PR) and helmexec.info() trims the trailing blank after helm's install
status. Update the diff-args apply-stderr{,-helm4} and apply-live-
stderr{,-helm4} goldens accordingly so they match the actual stderr.

Signed-off-by: yxxhero <aiopsclub@163.com>

* test: drop trimmed blank line from v1-subhelmfile template golden

The trailing-newline trim in helmexec.info() removes the blank line helm
prints after '"incubator" has been added to your repositories'. Update
the v1-subhelmfile-multi-bases-with-array-values result and result-live
goldens so the template stdout comparison matches.

Signed-off-by: yxxhero <aiopsclub@163.com>

---------

Signed-off-by: Roman Mykhailiuk <romanm@cybellum.com>
Signed-off-by: yxxhero <aiopsclub@163.com>
Co-authored-by: Roman Mykhailiuk <romanm@cybellum.com>
2026-06-21 13:43:17 +08:00

361 lines
11 KiB
Go

package config
import (
"fmt"
"slices"
)
// ApplyOptoons is the options for the apply command
type ApplyOptions struct {
// Set is a list of key value pairs to be merged into the command
Set []string
// Values is a list of value files to be merged into the command
Values []string
// Concurrency is the maximum number of concurrent helm processes to run
Concurrency int
// Validate is validate your manifests against the Kubernetes cluster you are currently pointing at. Note that this requires access to a Kubernetes cluster to obtain information necessary for validating, like the list of available API versions
Validate bool
// Context is the number of lines of context to show around changes
Context int
// Output is the output format for the diff plugin
Output string
// DetailedExitcode is true if the exit code should be 2 instead of 0 if there were changes detected and the changes were synced successfully
DetailedExitcode bool
// StripTrailingCR is true if trailing carriage returns should be stripped during diffing
StripTrailingCR bool
// SkipCleanup is true if the cleanup of temporary values files should be skipped
SkipCleanup bool
// SkipCRDs is true if the CRDs should be skipped
SkipCRDs bool
// SkipNeeds is true if the needs should be skipped
SkipNeeds bool
// IncludeNeeds is true if the needs should be included
IncludeNeeds bool
// IncludeTransitiveNeeds is true if the transitive needs should be included
IncludeTransitiveNeeds bool
// EnforceNeedsAreInstalled is true if we should error if/when there are unmeetable dependencies
EnforceNeedsAreInstalled bool
// SkipDiffOnInstall is true if the diff should be skipped on install
SkipDiffOnInstall bool
// DiffArgs is the list of arguments to pass to the helm-diff.
DiffArgs string
// IncludeTests is true if the tests should be included
IncludeTests bool
// Suppress is true if the output should be suppressed
Suppress []string
// SuppressSecrets is true if the secrets should be suppressed
SuppressSecrets bool
// ShowSecrets is true if the secrets should be shown
ShowSecrets bool
// NoHooks skips checking for hooks
NoHooks bool
// SuppressDiff is true if the diff should be suppressed
SuppressDiff bool
// Wait is true if the helm command should wait for the release to be deployed
Wait bool
// WaitRetries is the number of times to retry waiting for the release to be deployed
WaitRetries int
// WaitForJobs is true if the helm command should wait for the jobs to be completed
WaitForJobs bool
// Timeout is the timeout for helm operations in seconds
Timeout int
// Propagate '--skip-schema-validation' to helmv3 template and helm install
SkipSchemaValidation bool
// ReuseValues is true if the helm command should reuse the values
ReuseValues bool
// ResetValues is true if helm command should reset values to charts' default
ResetValues bool
// Propagate '--post-renderer' to helmv3 template and helm install
PostRenderer string
// Propagate '--post-renderer-args' to helmv3 template and helm install
PostRendererArgs []string
// Cascade '--cascade' to helmv3 delete, available values: background, foreground, or orphan, default: background
Cascade string
// SuppressOutputLineRegex is a list of regexes to suppress output lines
SuppressOutputLineRegex []string
// SyncArgs is the list of arguments to pass to helm upgrade.
SyncArgs string
// HideNotes is the hide notes flag
HideNotes bool
// TakeOwnership is true if the ownership should be taken
TakeOwnership bool
// ServerSide controls the helm 4 --server-side flag. Must be "true", "false", or "auto".
ServerSide string
SyncReleaseLabels bool
// TrackMode specifies whether to use 'helm' or 'kubedog' for tracking resources
TrackMode string
// TrackTimeout specifies timeout for kubedog tracking (in seconds)
TrackTimeout int
// TrackLogs enables log streaming with kubedog
TrackLogs bool
// TrackFailedLogs streams logs only for pods that enter a failed state.
TrackFailedLogs bool
// HelmStuckGrace, when > 0, enables the helm-killer safety valve. See
// ReleaseSpec.HelmStuckGrace for details. Value is in seconds.
HelmStuckGrace int
// TrackFailOnError controls whether kubedog tracking failures cause a non-zero exit code
TrackFailOnError bool
// Description is the description that will be passed to helm upgrade --description
Description string
}
// NewApply creates a new Apply
func NewApplyOptions() *ApplyOptions {
return &ApplyOptions{}
}
// ApplyImpl is impl for applyOptions
type ApplyImpl struct {
*GlobalImpl
*ApplyOptions
}
// NewApplyImpl creates a new ApplyImpl
func NewApplyImpl(g *GlobalImpl, a *ApplyOptions) *ApplyImpl {
return &ApplyImpl{
GlobalImpl: g,
ApplyOptions: a,
}
}
// Set returns the set.
func (a *ApplyImpl) Set() []string {
return a.ApplyOptions.Set
}
// Concurrency returns the concurrency.
func (a *ApplyImpl) Concurrency() int {
return a.ApplyOptions.Concurrency
}
// Context returns the context.
func (a *ApplyImpl) Context() int {
return a.ApplyOptions.Context
}
// DetailedExitcode returns the detailed exitcode.
func (a *ApplyImpl) DetailedExitcode() bool {
return a.ApplyOptions.DetailedExitcode
}
// StripTrailingCR is true if trailing carriage returns should be stripped during diffing
func (a *ApplyImpl) StripTrailingCR() bool {
return a.ApplyOptions.StripTrailingCR
}
// DiffOutput returns the diff output.
func (a *ApplyImpl) DiffOutput() string {
return a.Output
}
// IncludeNeeds returns the include needs.
func (a *ApplyImpl) IncludeNeeds() bool {
return a.ApplyOptions.IncludeNeeds || a.IncludeTransitiveNeeds()
}
// IncludeTests returns the include tests.
func (a *ApplyImpl) IncludeTests() bool {
return a.ApplyOptions.IncludeTests
}
// IncludeTransitiveNeeds returns the include transitive needs.
func (a *ApplyImpl) IncludeTransitiveNeeds() bool {
return a.ApplyOptions.IncludeTransitiveNeeds
}
// EnforceNeedsAreInstalled is true we should error if/when there are unmeetable dependencies
func (a *ApplyImpl) EnforceNeedsAreInstalled() bool {
return a.ApplyOptions.EnforceNeedsAreInstalled
}
// ShowSecrets returns the show secrets.
func (a *ApplyImpl) ShowSecrets() bool {
return a.ApplyOptions.ShowSecrets
}
// NoHooks skips hooks.
func (a *ApplyImpl) NoHooks() bool {
return a.ApplyOptions.NoHooks
}
// SkipCRDs returns the skip crds.
func (a *ApplyImpl) SkipCRDs() bool {
return a.ApplyOptions.SkipCRDs
}
// SkipCleanup returns the skip cleanup.
func (a *ApplyImpl) SkipCleanup() bool {
return a.ApplyOptions.SkipCleanup
}
// SkipDiffOnInstall returns the skip diff on install.
func (a *ApplyImpl) SkipDiffOnInstall() bool {
return a.ApplyOptions.SkipDiffOnInstall
}
// DiffArgs is the list of arguments to pass to helm-diff.
func (a *ApplyImpl) DiffArgs() string {
return a.ApplyOptions.DiffArgs
}
// SkipNeeds returns the skip needs.
func (a *ApplyImpl) SkipNeeds() bool {
if !a.IncludeNeeds() {
return a.ApplyOptions.SkipNeeds
}
return false
}
// Suppress returns the suppress.
func (a *ApplyImpl) Suppress() []string {
return a.ApplyOptions.Suppress
}
// SuppressDiff returns the suppress diff.
func (a *ApplyImpl) SuppressDiff() bool {
return a.ApplyOptions.SuppressDiff
}
// SuppressSecrets returns the suppress secrets.
func (a *ApplyImpl) SuppressSecrets() bool {
return a.ApplyOptions.SuppressSecrets
}
// Validate returns the validate.
func (a *ApplyImpl) Validate() bool {
return a.ApplyOptions.Validate
}
// Values returns the values.
func (a *ApplyImpl) Values() []string {
return a.ApplyOptions.Values
}
// Wait returns the wait.
func (a *ApplyImpl) Wait() bool {
return a.ApplyOptions.Wait
}
// WaitRetries returns the wait retries.
func (a *ApplyImpl) WaitRetries() int {
return a.ApplyOptions.WaitRetries
}
// WaitForJobs returns the wait for jobs.
func (a *ApplyImpl) WaitForJobs() bool {
return a.ApplyOptions.WaitForJobs
}
// Timeout returns the timeout.
func (a *ApplyImpl) Timeout() int {
return a.ApplyOptions.Timeout
}
// ReuseValues returns the ReuseValues.
func (a *ApplyImpl) ReuseValues() bool {
if !a.ResetValues() {
return a.ApplyOptions.ReuseValues
}
return false
}
func (a *ApplyImpl) ResetValues() bool {
return a.ApplyOptions.ResetValues
}
// PostRenderer returns the PostRenderer.
func (a *ApplyImpl) PostRenderer() string {
return a.ApplyOptions.PostRenderer
}
// PostRendererArgs returns the PostRendererArgs.
func (a *ApplyImpl) PostRendererArgs() []string {
return a.ApplyOptions.PostRendererArgs
}
// SkipSchemaValidation returns the SkipSchemaValidation.
func (a *ApplyImpl) SkipSchemaValidation() bool {
return a.ApplyOptions.SkipSchemaValidation
}
// Cascade returns cascade flag
func (a *ApplyImpl) Cascade() string {
return a.ApplyOptions.Cascade
}
// SuppressOutputLineRegex returns the SuppressOutputLineRegex.
func (a *ApplyImpl) SuppressOutputLineRegex() []string {
return a.ApplyOptions.SuppressOutputLineRegex
}
// SyncArgs returns the SyncArgs.
func (a *ApplyImpl) SyncArgs() string {
return a.ApplyOptions.SyncArgs
}
// HideNotes returns the HideNotes.
func (a *ApplyImpl) HideNotes() bool {
return a.ApplyOptions.HideNotes
}
// TakeOwnership returns the TakeOwnership.
func (a *ApplyImpl) TakeOwnership() bool {
return a.ApplyOptions.TakeOwnership
}
// ServerSide returns the ServerSide.
func (a *ApplyImpl) ServerSide() string {
return a.ApplyOptions.ServerSide
}
// SyncReleaseLabels returns the SyncReleaseLabels.
func (a *ApplyImpl) SyncReleaseLabels() bool {
return a.ApplyOptions.SyncReleaseLabels
}
// TrackMode returns the track mode.
func (a *ApplyImpl) TrackMode() string {
return a.ApplyOptions.TrackMode
}
// TrackTimeout returns the track timeout.
func (a *ApplyImpl) TrackTimeout() int {
return a.ApplyOptions.TrackTimeout
}
// TrackLogs returns the track logs flag.
func (a *ApplyImpl) TrackLogs() bool {
return a.ApplyOptions.TrackLogs
}
// TrackFailedLogs returns the track-failed-logs flag.
func (a *ApplyImpl) TrackFailedLogs() bool {
return a.ApplyOptions.TrackFailedLogs
}
// HelmStuckGrace returns the helm-stuck-grace value (seconds, 0 = disabled).
func (a *ApplyImpl) HelmStuckGrace() int {
return a.ApplyOptions.HelmStuckGrace
}
// TrackFailOnError returns whether kubedog tracking failures should cause a non-zero exit code.
func (a *ApplyImpl) TrackFailOnError() bool {
return a.ApplyOptions.TrackFailOnError
}
// Description returns the description.
func (a *ApplyImpl) Description() string {
return a.ApplyOptions.Description
}
func (a *ApplyImpl) ValidateConfig() error {
validTrackModes := []string{"helm", "helm-legacy", "kubedog"}
if a.ApplyOptions.TrackMode != "" && !slices.Contains(validTrackModes, a.ApplyOptions.TrackMode) {
return fmt.Errorf("--track-mode must be 'helm', 'helm-legacy', or 'kubedog', got: %s", a.ApplyOptions.TrackMode)
}
return a.GlobalImpl.ValidateConfig()
}