public_git/helmfile
1
0
Fork 0
mirror of https://github.com/helmfile/helmfile.git synced 2026-09-30 14:01:46 +02:00
Code Issues Projects Releases Wiki Activity
Files
913ebfdfa478bcebdc3d0970d9e95ddaafd90e3a
helmfile/pkg/app/destroy_test.go
T
yxxheroandRoman Mykhailiuk 9d3a9ad6c7 feat: parallel kubedog tracking with progress printer and safety valves (#2654) 
* feat: parallel kubedog tracking with progress printer and safety valves

Rework kubedog integration so resource tracking runs in parallel with
helm upgrade/install, giving live progress output and recovering from
known helm/kubedog wedge conditions.

Core:
- kubedogTrackingHandle runs tracking in a background goroutine alongside
  the helm subprocess (startBackgroundKubedogTracking); helm output is
  buffered and replayed as a single block so it no longer interleaves
  with progress ticks.
- Capture UID+generation baselines before handing off to helm so each
  tracker waits until the resource actually changes (freshness gate).

Progress printer (pkg/kubedog/printer.go):
- Styled, auto-sized progress table with a heartbeat flusher, child (pod)
  status roll-up, pre-ready pod-phase handling, multi-namespace support,
  and optional color. PreviewBreakdown summarizes kept/filtered resources.

Safety valves (verify cluster state via the live API):
- Tracker-race valve (always on): when helm succeeds but a dyntracker
  goroutine is wedged, poll the API and cancel the tracker so wait()
  returns success instead of blocking until --track-timeout.
- Helm-stuck killer (opt-in via helmStuckGrace): if the cluster stays
  converged while helm v4's hook waiter is wedged, SIGINT the helm
  subprocess to recover.
- Failure watchdog (pkg/kubedog/watchdog.go): surface failing pods that
  never made it into dyntracker's resource graph.

Options: trackFailedLogs, helmStuckGrace, trackTimeout, color
(Color/NoColor), and resource filtering (trackKinds/skipKinds/
trackResources). PersistentVolumeClaim support in resource classification.

Signed-off-by: Roman Mykhailiuk <romanm@cybellum.com>

* test: add unit tests for kubedog tracking

Cover the progress printer, resource classification, the failure
watchdog, helm-output trimming/dedup, the release hard-timeout helper,
and the color/track option plumbing.

Signed-off-by: Roman Mykhailiuk <romanm@cybellum.com>

* test: update golden logs, e2e snapshots, and values-id fixtures

Refresh pkg/app testapply/testdestroy golden logs, e2e template
snapshots, and the TestGenerateID values-id golden hashes for the new
kubedog progress output and the merged release struct layout.

Signed-off-by: Roman Mykhailiuk <romanm@cybellum.com>

* refactor: remove dead kubedog display code and dedupe tracker setup

Deep-review pass on the parallel kubedog tracking changes:

- Remove pkg/kubedog/display.go (308 lines) and display_test.go (453 lines).
  These rendered progress for the legacy per-kind trackers that this PR
  replaces; in the merged tree every function is unreferenced outside its
  own tests. The new progressPrinter (printer.go) supersedes them.
  TestMain (color.ForceColor for deterministic ANSI in tests) is preserved
  in a new main_test.go.
- Dedupe trackWithKubedog: the post-helm fallback rebuilt the exact same
  tracker options as buildReleaseTracker. Reuse buildReleaseTracker instead,
  dropping ~40 lines of duplicated timeout/log/filter/tracker construction.

No behavior change; build, go vet, golangci-lint, and the kubedog/state
unit tests all pass.

Signed-off-by: yxxhero <aiopsclub@163.com>

* fix: stop waitForFreshness busy-looping the API after helm finishes

Once upstreamDoneCh closes it is always ready, so the select in
waitForFreshness stopped blocking on the ticker and re-ran probe() (a
live GET) as fast as the round-trip allowed for the whole 3s grace
window — hammering the API server once per tracked resource.

Track a local view of the channel and nil it out on first delivery so
the first hit records the timestamp (one fast retry, as intended) and
all subsequent polls are ticker-throttled. Functional behavior is
unchanged: return nil when fresh, errUpstreamDoneNoChange after grace.

Signed-off-by: yxxhero <aiopsclub@163.com>

* test: drop trailing blank line from helm4 OCI pull snapshots

The trailing-newline trim in helmexec.info() removes the blank line helm
prints after the OCI chart "Digest:" line. The helm3 (output.yaml)
snapshots never captured that line, but the helm4 (output-helm4.yaml)
snapshots for oci_chart_pull{,_direct,_once,_once2} and
issue_473_oci_chart_url_fetch still expected it, so they failed under
helm 4. Remove the blank line so the snapshots match the trimmed output.

Signed-off-by: yxxhero <aiopsclub@163.com>

* test: refresh diff-args integration goldens for styled release headers

DisplayAffectedReleases now emits a styled "========== Updated Releases
==========" header (matching the app/e2e goldens already updated by this
PR) and helmexec.info() trims the trailing blank after helm's install
status. Update the diff-args apply-stderr{,-helm4} and apply-live-
stderr{,-helm4} goldens accordingly so they match the actual stderr.

Signed-off-by: yxxhero <aiopsclub@163.com>

* test: drop trimmed blank line from v1-subhelmfile template golden

The trailing-newline trim in helmexec.info() removes the blank line helm
prints after '"incubator" has been added to your repositories'. Update
the v1-subhelmfile-multi-bases-with-array-values result and result-live
goldens so the template stdout comparison matches.

Signed-off-by: yxxhero <aiopsclub@163.com>

---------

Signed-off-by: Roman Mykhailiuk <romanm@cybellum.com>
Signed-off-by: yxxhero <aiopsclub@163.com>
Co-authored-by: Roman Mykhailiuk <romanm@cybellum.com>
2026-06-21 13:43:17 +08:00

427 lines
14 KiB
Go
Raw Blame History

package app
import (
"strings"
"sync"
"testing"
"github.com/helmfile/vals"
"github.com/stretchr/testify/assert"
"go.uber.org/zap"
"github.com/helmfile/helmfile/pkg/exectest"
ffs "github.com/helmfile/helmfile/pkg/filesystem"
"github.com/helmfile/helmfile/pkg/helmexec"
)
const (
helmV3ListFlags = "--kube-context default --uninstalling --deployed --failed --pending"
helmV3ListFlagsWithoutKubeContext = "--uninstalling --deployed --failed --pending"
)
func listFlags(namespace, kubeContext string) string {
var flags []string
if kubeContext != "" {
flags = append(flags, "--kube-context", kubeContext)
}
if namespace != "" {
flags = append(flags, "--namespace", namespace)
}
flags = append(flags, "--uninstalling --deployed --failed --pending")
return strings.Join(flags, " ")
}
type destroyConfig struct {
args string
cascade string
concurrency int
interactive bool
skipDeps bool
skipRefresh bool
logger *zap.SugaredLogger
includeTransitiveNeeds bool
skipCharts bool
deleteWait bool
deleteTimeout int
}
func (d destroyConfig) Args() string {
return d.args
}
func (d destroyConfig) Cascade() string {
return d.cascade
}
func (d destroyConfig) SkipCharts() bool {
return d.skipCharts
}
func (d destroyConfig) Interactive() bool {
return d.interactive
}
func (d destroyConfig) Logger() *zap.SugaredLogger {
return d.logger
}
func (d destroyConfig) Concurrency() int {
return d.concurrency
}
func (d destroyConfig) SkipDeps() bool {
return d.skipDeps
}
func (d destroyConfig) SkipRefresh() bool {
return d.skipRefresh
}
func (d destroyConfig) IncludeTransitiveNeeds() bool {
return d.includeTransitiveNeeds
}
func (d destroyConfig) DeleteWait() bool {
return d.deleteWait
}
func (d destroyConfig) DeleteTimeout() int {
return d.deleteTimeout
}
func (d destroyConfig) NoColor() bool {
return true
}
func TestDestroy(t *testing.T) {
type testcase struct {
ns string
concurrency int
error string
files map[string]string
selectors []string
lists map[exectest.ListKey]string
diffs map[exectest.DiffKey]error
upgraded []exectest.Release
deleted []exectest.Release
log string
deleteWait bool
deleteTimeout int
}
check := func(t *testing.T, tc testcase) {
t.Helper()
wantUpgrades := tc.upgraded
wantDeletes := tc.deleted
var helm = &exectest.Helm{
FailOnUnexpectedList: true,
FailOnUnexpectedDiff: true,
Lists: tc.lists,
Diffs: tc.diffs,
DiffMutex: &sync.Mutex{},
ChartsMutex: &sync.Mutex{},
ReleasesMutex: &sync.Mutex{},
}
bs := runWithLogCapture(t, "debug", func(t *testing.T, logger *zap.SugaredLogger) {
t.Helper()
valsRuntime, err := vals.New(vals.Options{CacheSize: 32})
if err != nil {
t.Errorf("unexpected error creating vals runtime: %v", err)
}
app := appWithFs(&App{
OverrideHelmBinary: DefaultHelmBinary,
fs: ffs.DefaultFileSystem(),
OverrideKubeContext: "default",
DisableKubeVersionAutoDetection: true,
Env: "default",
Logger: logger,
helms: map[helmKey]helmexec.Interface{
createHelmKey("helm", "default"): helm,
},
valsRuntime: valsRuntime,
}, tc.files)
if tc.ns != "" {
app.Namespace = tc.ns
}
if tc.selectors != nil {
app.Selectors = tc.selectors
}
destroyErr := app.Destroy(destroyConfig{
// if we check log output, concurrency must be 1. otherwise the test becomes non-deterministic.
concurrency: tc.concurrency,
logger: logger,
})
switch {
case tc.error == "" && destroyErr != nil:
t.Fatalf("unexpected error: %v", destroyErr)
case tc.error != "" && destroyErr == nil:
t.Fatal("expected error did not occur")
case tc.error != "" && destroyErr != nil && tc.error != destroyErr.Error():
t.Fatalf("invalid error: expected %q, got %q", tc.error, destroyErr.Error())
}
if len(wantUpgrades) > len(helm.Releases) {
t.Fatalf("insufficient number of upgrades: got %d, want %d", len(helm.Releases), len(wantUpgrades))
}
for relIdx := range wantUpgrades {
if wantUpgrades[relIdx].Name != helm.Releases[relIdx].Name {
t.Errorf("releases[%d].name: got %q, want %q", relIdx, helm.Releases[relIdx].Name, wantUpgrades[relIdx].Name)
}
for flagIdx := range wantUpgrades[relIdx].Flags {
if wantUpgrades[relIdx].Flags[flagIdx] != helm.Releases[relIdx].Flags[flagIdx] {
t.Errorf("releaes[%d].flags[%d]: got %v, want %v", relIdx, flagIdx, helm.Releases[relIdx].Flags[flagIdx], wantUpgrades[relIdx].Flags[flagIdx])
}
}
}
if len(wantDeletes) > len(helm.Deleted) {
t.Fatalf("insufficient number of deletes: got %d, want %d", len(helm.Deleted), len(wantDeletes))
}
for relIdx := range wantDeletes {
if wantDeletes[relIdx].Name != helm.Deleted[relIdx].Name {
t.Errorf("releases[%d].name: got %q, want %q", relIdx, helm.Deleted[relIdx].Name, wantDeletes[relIdx].Name)
}
for flagIdx := range wantDeletes[relIdx].Flags {
if wantDeletes[relIdx].Flags[flagIdx] != helm.Deleted[relIdx].Flags[flagIdx] {
t.Errorf("releaes[%d].flags[%d]: got %v, want %v", relIdx, flagIdx, helm.Deleted[relIdx].Flags[flagIdx], wantDeletes[relIdx].Flags[flagIdx])
}
}
}
})
if tc.log != "" {
actual := bs.String()
assert.Equal(t, tc.log, actual)
} else {
assertLogEqualsToSnapshot(t, bs.String())
}
}
files := map[string]string{
"/path/to/helmfile.yaml": `
releases:
- name: database
chart: charts/mysql
needs:
- logging
- name: frontend-v1
chart: charts/frontend
installed: false
needs:
- servicemesh
- logging
- backend-v1
- name: frontend-v2
chart: charts/frontend
needs:
- servicemesh
- logging
- backend-v2
- name: frontend-v3
chart: charts/frontend
needs:
- servicemesh
- logging
- backend-v2
- name: backend-v1
chart: charts/backend
installed: false
needs:
- servicemesh
- logging
- database
- anotherbackend
- name: backend-v2
chart: charts/backend
needs:
- servicemesh
- logging
- database
- anotherbackend
- name: anotherbackend
chart: charts/anotherbackend
needs:
- servicemesh
- logging
- database
- name: servicemesh
chart: charts/istio
needs:
- logging
- name: logging
chart: charts/fluent-bit
- name: front-proxy
chart: stable/envoy
`,
}
filesForTwoReleases := map[string]string{
"/path/to/helmfile.yaml": `
releases:
- name: backend-v1
chart: charts/backend
installed: false
- name: frontend-v1
chart: charts/frontend
needs:
- backend-v1
`,
}
t.Run("smoke", func(t *testing.T) {
//
// complex test cases for smoke testing
//
check(t, testcase{
files: files,
diffs: map[exectest.DiffKey]error{},
lists: map[exectest.ListKey]string{
{Filter: "^frontend-v1$", Flags: listFlags("", "default")}: `NAME REVISION UPDATED STATUS CHART APP VERSION NAMESPACE
`,
{Filter: "^frontend-v2$", Flags: listFlags("", "default")}: `NAME REVISION UPDATED STATUS CHART APP VERSION NAMESPACE
frontend-v2 4 Fri Nov 1 08:40:07 2019 DEPLOYED frontend-3.1.0 3.1.0 default
`,
{Filter: "^frontend-v3$", Flags: listFlags("", "default")}: `NAME REVISION UPDATED STATUS CHART APP VERSION NAMESPACE
frontend-v3 4 Fri Nov 1 08:40:07 2019 DEPLOYED frontend-3.1.0 3.1.0 default
`,
{Filter: "^backend-v1$", Flags: listFlags("", "default")}: `NAME REVISION UPDATED STATUS CHART APP VERSION NAMESPACE
`,
{Filter: "^backend-v2$", Flags: listFlags("", "default")}: `NAME REVISION UPDATED STATUS CHART APP VERSION NAMESPACE
backend-v2 4 Fri Nov 1 08:40:07 2019 DEPLOYED backend-3.1.0 3.1.0 default
`,
{Filter: "^logging$", Flags: listFlags("", "default")}: `NAME REVISION UPDATED STATUS CHART APP VERSION NAMESPACE
logging 4 Fri Nov 1 08:40:07 2019 DEPLOYED fluent-bit-3.1.0 3.1.0 default
`,
{Filter: "^front-proxy$", Flags: listFlags("", "default")}: `NAME REVISION UPDATED STATUS CHART APP VERSION NAMESPACE
front-proxy 4 Fri Nov 1 08:40:07 2019 DEPLOYED envoy-3.1.0 3.1.0 default
`,
{Filter: "^servicemesh$", Flags: listFlags("", "default")}: `NAME REVISION UPDATED STATUS CHART APP VERSION NAMESPACE
servicemesh 4 Fri Nov 1 08:40:07 2019 DEPLOYED istio-3.1.0 3.1.0 default
`,
{Filter: "^database$", Flags: listFlags("", "default")}: `NAME REVISION UPDATED STATUS CHART APP VERSION NAMESPACE
database 4 Fri Nov 1 08:40:07 2019 DEPLOYED mysql-3.1.0 3.1.0 default
`,
{Filter: "^anotherbackend$", Flags: listFlags("", "default")}: `NAME REVISION UPDATED STATUS CHART APP VERSION NAMESPACE
anotherbackend 4 Fri Nov 1 08:40:07 2019 DEPLOYED anotherbackend-3.1.0 3.1.0 default
`,
},
// Disable concurrency to avoid in-deterministic result
concurrency: 1,
// Enable wait and set timeout for destroy
deleteWait: true,
deleteTimeout: 300,
upgraded: []exectest.Release{},
deleted: []exectest.Release{
{Name: "frontend-v3", Flags: []string{}},
{Name: "frontend-v2", Flags: []string{}},
{Name: "frontend-v1", Flags: []string{}},
{Name: "backend-v2", Flags: []string{}},
{Name: "backend-v1", Flags: []string{}},
{Name: "anotherbackend", Flags: []string{}},
{Name: "servicemesh", Flags: []string{}},
{Name: "database", Flags: []string{}},
{Name: "front-proxy", Flags: []string{}},
{Name: "logging", Flags: []string{}},
},
})
})
t.Run("destroy only one release with selector", func(t *testing.T) {
check(t, testcase{
files: files,
selectors: []string{"name=logging"},
diffs: map[exectest.DiffKey]error{},
lists: map[exectest.ListKey]string{
{Filter: "^frontend-v1$", Flags: listFlags("", "default")}: `NAME REVISION UPDATED STATUS CHART APP VERSION NAMESPACE
`,
{Filter: "^frontend-v2$", Flags: listFlags("", "default")}: `NAME REVISION UPDATED STATUS CHART APP VERSION NAMESPACE
frontend-v2 4 Fri Nov 1 08:40:07 2019 DEPLOYED frontend-3.1.0 3.1.0 default
`,
{Filter: "^frontend-v3$", Flags: listFlags("", "default")}: `NAME REVISION UPDATED STATUS CHART APP VERSION NAMESPACE
frontend-v3 4 Fri Nov 1 08:40:07 2019 DEPLOYED frontend-3.1.0 3.1.0 default
`,
{Filter: "^backend-v1$", Flags: listFlags("", "default")}: `NAME REVISION UPDATED STATUS CHART APP VERSION NAMESPACE
`,
{Filter: "^backend-v2$", Flags: listFlags("", "default")}: `NAME REVISION UPDATED STATUS CHART APP VERSION NAMESPACE
backend-v2 4 Fri Nov 1 08:40:07 2019 DEPLOYED backend-3.1.0 3.1.0 default
`,
{Filter: "^logging$", Flags: listFlags("", "default")}: `NAME REVISION UPDATED STATUS CHART APP VERSION NAMESPACE
logging 4 Fri Nov 1 08:40:07 2019 DEPLOYED fluent-bit-3.1.0 3.1.0 default
`,
{Filter: "^front-proxy$", Flags: listFlags("", "default")}: `NAME REVISION UPDATED STATUS CHART APP VERSION NAMESPACE
front-proxy 4 Fri Nov 1 08:40:07 2019 DEPLOYED envoy-3.1.0 3.1.0 default
`,
{Filter: "^servicemesh$", Flags: listFlags("", "default")}: `NAME REVISION UPDATED STATUS CHART APP VERSION NAMESPACE
servicemesh 4 Fri Nov 1 08:40:07 2019 DEPLOYED istio-3.1.0 3.1.0 default
`,
{Filter: "^database$", Flags: listFlags("", "default")}: `NAME REVISION UPDATED STATUS CHART APP VERSION NAMESPACE
database 4 Fri Nov 1 08:40:07 2019 DEPLOYED mysql-3.1.0 3.1.0 default
`,
{Filter: "^anotherbackend$", Flags: listFlags("", "default")}: `NAME REVISION UPDATED STATUS CHART APP VERSION NAMESPACE
anotherbackend 4 Fri Nov 1 08:40:07 2019 DEPLOYED anotherbackend-3.1.0 3.1.0 default
`,
},
// Disable concurrency to avoid in-deterministic result
concurrency: 1,
upgraded: []exectest.Release{},
deleted: []exectest.Release{
{Name: "logging", Flags: []string{}},
},
})
})
t.Run("destroy installed but disabled release", func(t *testing.T) {
check(t, testcase{
files: filesForTwoReleases,
diffs: map[exectest.DiffKey]error{},
lists: map[exectest.ListKey]string{
{Filter: "^frontend-v1$", Flags: listFlags("", "default")}: `NAME REVISION UPDATED STATUS CHART APP VERSION NAMESPACE
`,
{Filter: "^backend-v1$", Flags: listFlags("", "default")}: `NAME REVISION UPDATED STATUS CHART APP VERSION NAMESPACE
`,
},
// Disable concurrency to avoid in-deterministic result
concurrency: 1,
upgraded: []exectest.Release{},
deleted: []exectest.Release{
{Name: "frontend-v1", Flags: []string{}},
},
})
})
t.Run("helm3", func(t *testing.T) {
check(t, testcase{
files: filesForTwoReleases,
diffs: map[exectest.DiffKey]error{},
lists: map[exectest.ListKey]string{
{Filter: "^frontend-v1$", Flags: listFlags("", "default")}: `NAME REVISION UPDATED STATUS CHART APP VERSION NAMESPACE
`,
{Filter: "^backend-v1$", Flags: listFlags("", "default")}: `NAME REVISION UPDATED STATUS CHART APP VERSION NAMESPACE
`,
},
// Disable concurrency to avoid in-deterministic result
concurrency: 1,
// Enable wait and set timeout for destroy
deleteWait: true,
deleteTimeout: 300,
upgraded: []exectest.Release{},
deleted: []exectest.Release{
{Name: "frontend-v1", Flags: []string{}},
},
})
})
}
Reference in New Issue View Git Blame Copy Permalink