Files
helmfile/pkg/app/app.go
T
Aditya Menon c8bcbcd629 🐛 Fix four critical issues: environment merging, kubeVersion detection, lookup() with kustomize, and Helm 4 color flags (#2276)
* fix: deep merge environments from multiple bases (#2273)

Problem:
When using multiple base helmfiles, environment values were being
completely replaced instead of deep-merged due to mergo.WithOverride
introduced in PR #2228.

Solution:
- Created mergeEnvironments() function for proper deep merging
- Manually merge environment Values and Secrets slices before struct merge
- Preserves all environment values from both base and current helmfile

Testing:
- Added TestEnvironmentMergingWithBases with two scenarios:
  1. Multiple bases with overlapping environment values
  2. Environment values with array merging

Fixes #2273

Signed-off-by: Aditya Menon <amenon@canarytechnologies.com>

* fix: auto-detect Kubernetes version for helm-diff (#2275)

Problem:
When helmfile runs helm-diff without specifying kubeVersion, helm-diff
falls back to v1.20.0. This causes chart compatibility checks to fail
for charts requiring newer Kubernetes versions (e.g., kubeVersion: ">=1.25.0").

Root Cause:
- flagsForDiff() was not passing kubeVersion to helm-diff plugin
- Without --kube-version flag, helm-diff uses default v1.20.0

Solution:
- Created pkg/cluster package with DetectServerVersion() function
- Auto-detect cluster version using k8s.io/client-go discovery API
- Pass detected version to helm-diff via --kube-version flag
- Priority: helmfile.yaml kubeVersion > auto-detected version
- Works with both Helm 3 and Helm 4

Implementation:
- pkg/cluster/version.go: Cluster version detection
- pkg/app/app.go: detectKubeVersion() helper used in diff() and apply()
- pkg/state/state.go: Added DetectedKubeVersion field to DiffOpts
- Integrated into flagsForDiff() with proper precedence

Testing:
- Unit tests for cluster version detection
- Unit tests for kubeVersion precedence logic
- Integration test with chart requiring Kubernetes >=1.25.0
- Tests verify upgrade scenario (critical failure case from issue)
- Validated with both Helm 3 and Helm 4

Fixes #2275

Signed-off-by: Aditya Menon <amenon@canarytechnologies.com>

* fix: enable lookup() function with strategicMergePatches (#2271)

Problem:
When using strategicMergePatches (kustomize), Helm's lookup() function
stops working. Charts like Grafana use lookup() to preserve existing
resource values (e.g., PVC volumeName), which get lost when using patches.

Root Cause:
- Chartify runs "helm template" to render charts before applying patches
- By default, "helm template" runs client-side without cluster access
- The lookup() function requires cluster connectivity to query resources
- Without cluster access, lookup() returns empty values

Solution:
- Pass --dry-run=server to helm template when using kustomize patches
- This enables cluster connectivity for lookup() while keeping client-side rendering
- Only applied to commands requiring cluster access (diff, apply, sync, etc.)
- Offline commands (template, lint, build) remain cluster-independent

Implementation:
- Modified processChartification() to accept helmfileCommand parameter
- Added switch-based logic to determine cluster requirement per command
- Conditionally set chartifyOpts.TemplateArgs = "--dry-run=server"
- Safe default: unknown commands assume cluster access

Command Behavior:
- helmfile diff/apply/sync: Uses --dry-run=server, lookup() works
- helmfile template/lint/build: No cluster requirement, works offline
- Charts without lookup(): Unaffected
- Charts with lookup() + cluster: Lookup values preserved correctly

Testing:
- Integration test with ConfigMap using lookup() to preserve values
- Verifies lookup works with strategicMergePatches
- Tests both with and without cluster access
- Validates offline template command still works

Fixes #2271

Signed-off-by: Aditya Menon <amenon@canarytechnologies.com>

* fix: remove unnecessary error return from mergeEnvironments

The mergeEnvironments function always returns nil, making the error
return value unnecessary. This fixes the unparam linter warning.

- Changed function signature to not return error
- Updated call site to not handle error
- All tests still pass

Signed-off-by: Aditya Menon <amenon@canarytechnologies.com>

* fix: handle nil Environments map in mergeEnvironments

Fixes panic when base helmfile has nil Environments map.
Initialize the destination map if nil before merging to prevent
"assignment to entry in nil map" panic.

- Added nil check in mergeEnvironments to return early
- Initialize layers[0].Environments before merge if nil
- Fixes TestVisitDesiredStatesWithReleasesFiltered_Issue1008_MissingNonDefaultEnvInBase

The panic occurred when a base helmfile didn't define any environments
but a subsequent layer did. Now we properly initialize an empty map
to merge into.

Signed-off-by: Aditya Menon <amenon@canarytechnologies.com>

* test: disable kubeVersion auto-detection in unit tests

Add DisableKubeVersionAutoDetection field to App struct to prevent
unit tests from connecting to real Kubernetes clusters during testing.

The kubeVersion auto-detection feature (issue #2275) was causing
unit tests to fail because:
1. Tests use mock helm implementations without real cluster access
2. Auto-detection was connecting to local minikube cluster (v1.34.0)
3. Test expectations didn't include --kube-version flag in diff keys

Solution:
- Add DisableKubeVersionAutoDetection bool field to App struct
- Check this flag in detectKubeVersion() before attempting detection
- Set flag to true in all pkg/app/*_test.go files

This ensures unit tests remain isolated and don't depend on
external cluster state while preserving auto-detection for
production use.

Signed-off-by: Aditya Menon <amenon@canarytechnologies.com>

* chore: upgrade helm-diff plugin to v3.14.1

Update helm-diff plugin from v3.14.0 to v3.14.1 across all environments:
- Dockerfiles (main, debian-stable-slim, ubuntu)
- CI workflow matrix configurations
- Integration test default version

This ensures consistency across development, testing, and production
environments.

Signed-off-by: Aditya Menon <amenon@canarytechnologies.com>

* test: fix table formatting and improve E2E test infrastructure

This commit addresses multiple test failures and improves the testing
infrastructure for better reliability and maintainability.

Table Formatting Fixes:
- Added trimTrailingWhitespace() helper function to remove trailing
  whitespace from table output in both FormatAsTable() and printDAG()
- Fixes TestList and TestDAG failures caused by tabwriter padding
  empty columns with trailing spaces
- Updated golden file for table output test to match new behavior

E2E Test Infrastructure Improvements:
- Implemented dynamic port allocation for Docker registry tests to
  prevent port conflicts (replaced hardcoded port 5000/5001)
- Added getFreePort() function using kernel-allocated unused ports
- Added waitForRegistry() function with proper health check polling
  of Docker Registry /v2/ endpoint (replaces sleep hack)
- Added prepareInputFile() function to handle port substitution and
  path resolution when copying helmfile configs to temp directories
- Extracted setupLocalDockerRegistry() helper to reduce cognitive
  complexity from 111 to ≤110 (gocognit threshold)
- Added port normalization in test output to replace dynamic ports
  with $REGISTRY_PORT placeholder for deterministic comparisons

Test Configuration Updates:
- Updated OCI chart tests to use dynamic port allocation via
  $REGISTRY_PORT placeholder in helmfile configs
- Converted relative chart paths to absolute paths when input files
  are copied to temp directories (fixes path resolution issues)
- Left postrenderer paths as relative since they're resolved from
  working directory (works for both Helm 3 and Helm 4)

Golden File Updates:
- Updated all OCI-related test expected outputs to use $REGISTRY_PORT
  placeholder instead of hardcoded ports
- Removed trailing whitespace from issue_493 test expected output
- Updated postrenderer test outputs to reflect chart path normalization

Test Cleanup:
- Removed unused fakeInit struct and CheckHelmPlugins() call from
  snapshot tests (not needed for template/fetch/list commands)
- Removed unused imports (app, helmexec packages)

Technical Details:
- Port allocation uses net.Listen with port 0 for kernel assignment
- Registry health check polls with 500ms intervals and 30s timeout
- Chart paths: ../../charts/* → absolute paths (input file moves to temp)
- Postrenderer paths: remain relative (resolved from working directory)
- OCI cache paths normalized: oci__localhost_PORT → oci__localhost_$REGISTRY_PORT

All originally failing tests now pass:
- TestList ✓
- TestDAG ✓
- TestHelmfileTemplateWithBuildCommand (all OCI tests) ✓
- TestFormatAsTable ✓

Fixes three test failures reported in issue.

Signed-off-by: Aditya Menon <amenon@canarytechnologies.com>

* fix(test): convert postrenderer paths to absolute for Helm 3

Helm 3 resolves postrenderer script paths relative to the helmfile
location. When the input file is copied to a temp directory for
port substitution, relative postrenderer paths break.

Solution:
- Added postrenderersDir parameter to prepareInputFile()
- Convert ../../postrenderers/* to absolute paths for Helm 3 only
- Use existing isHelm4() function to detect Helm version
- Helm 4 extracts plugin names from paths, so works with relative

This fixes the postrenderer test failure in CI where Helm 3 could
not find the postrenderer script at the relative path.

Fixes: Error: unable to find binary at ../../postrenderers/add-cm2.bash
Signed-off-by: Aditya Menon <amenon@canarytechnologies.com>

* fix(test): remove remaining hardcoded port 5001 in OCI tests

Updated 4 remaining OCI chart tests that still had hardcoded port 5001:
- oci_chart_pull
- oci_chart_pull_once
- oci_chart_pull_once2
- oci_chart_pull_direct

Changes:
- config.yaml: Removed hardcoded port, use dynamic allocation
- input.yaml.gotmpl: Replaced localhost:5001 with localhost:$REGISTRY_PORT

This ensures all OCI chart tests use dynamic port allocation to
prevent port conflicts during parallel test execution.

Signed-off-by: Aditya Menon <amenon@canarytechnologies.com>

* fix: prevent helm-diff from normalizing server-side defaults

Problem:
The suppress-output-line-regex integration test was failing because
helm-diff was reporting "has changed, but diff is empty after suppression"
for Service resources when it should have shown ipFamilyPolicy and ipFamilies
fields being removed.

Root Cause:
When auto-detected kubeVersion (e.g., 1.34.0) is passed to helm-diff via
--kube-version flag, helm-diff normalizes server-side defaults. This makes
fields like ipFamilyPolicy and ipFamilies appear unchanged, even though they
don't exist in the chart template and will be removed by the upgrade.

After applying suppressOutputLineRegex patterns, only label changes remained
(helm.sh/chart and app.kubernetes.io/version). These were correctly suppressed,
leaving an empty diff - hence the "diff is empty after suppression" message.

Solution:
Added a new configuration option 'disableAutoDetectedKubeVersionForDiff' to allow
disabling auto-detected kubeVersion being passed to helm-diff. This prevents
helm-diff from normalizing server-side defaults when needed.

Default behavior: Pass auto-detected kubeVersion (fixes issue #2275, existing behavior)
Opt-out behavior: Set flag to true to only use explicit kubeVersion from helmfile.yaml

helmDefaults:
  disableAutoDetectedKubeVersionForDiff: true  # false by default

releases:
- name: myrelease
  disableAutoDetectedKubeVersionForDiff: true  # override per-release

Implementation:
- Added DisableAutoDetectedKubeVersionForDiff field to HelmSpec and ReleaseSpec
- Updated flagsForDiff() to check this flag before passing kubeVersion
- Default (false): pass auto-detected kubeVersion (fixes issue #2275)
- Opt-out (true): only pass explicit kubeVersion from helmfile.yaml
- Updated suppress-output-line-regex test to disable auto-detected kubeVersion

This approach:
- Maintains backward compatibility (default passes auto-detected kubeVersion)
- Fixes issue #2275 for charts requiring newer Kubernetes versions
- Allows users to opt-out when server-side normalization causes issues
- Fixes suppress-output-line-regex test regression

Signed-off-by: Aditya Menon <amenon@canarytechnologies.com>

* test: update hash values in TestGenerateID after adding DisableAutoDetectedKubeVersionForDiff field

The hash values in TestGenerateID needed to be updated because adding the
DisableAutoDetectedKubeVersionForDiff field to ReleaseSpec changed the structure's
hash representation. This is expected behavior as generateValuesID() hashes the
entire ReleaseSpec structure.

Updated all expected hash values to match the new values:
- baseline: foo-values-66f7fd6f7b
- different bytes content: foo-values-6664979cd7
- different map content: foo-values-78897dfd49
- different chart: foo-values-64b7846cb7
- different name: bar-values-576cb7ddc7
- specific ns: myns-foo-values-6c567f54c

Signed-off-by: Aditya Menon <amenon@canarytechnologies.com>

* fix: address PR review comments and resolve issue #2280

This commit addresses all review comments from GitHub Copilot and
resolves issue #2280 regarding --color flag conflict with Helm 4.

Changes:
1. Fixed documentation in pkg/cluster/version.go
   - Updated function comment to reflect error return behavior
   - Corrected version format example and comment

2. Added complete command categorization in pkg/state/state.go
   - Added all helmfile commands to cluster access switch statement
   - Properly categorized 15+ commands based on cluster requirements
   - Added clarifying comments for command groups

3. Resolved issue #2280: --color flag conflict with Helm 4
   - In Helm 4, --color expects a value (never/auto/always)
   - Converts --color to --color=always for Helm 4
   - Converts --no-color to --color=never for Helm 4
   - Prevents Helm from consuming next argument as color value
   - Added comprehensive unit tests
   - Added integration test (Helm 4 only)

Issue #2280 Details:
When running helmfile diff with --color and --context flags on Helm 4,
the --color flag would consume --context as its value, resulting in:
"invalid color mode '--context': must be one of: never, auto, always"

The fix detects Helm 4 and converts boolean color flags to the format
Helm 4 expects, preventing the argument consumption issue.

Fixes #2280

Signed-off-by: Aditya Menon <amenon@canarytechnologies.com>

* fix: correct kubeVersion precedence comment in test

The comment incorrectly stated that state.KubeVersion takes precedence
over paramKubeVersion, but the actual implementation (getKubeVersion in
state.go:3354-3364) shows the correct order is:

1. paramKubeVersion (auto-detected from cluster)
2. release.KubeVersion (per-release override)
3. state.KubeVersion (helmfile.yaml global setting)

Updated the comment to match the implementation and the test cases.

Signed-off-by: Aditya Menon <amenon@canarytechnologies.com>

* fix: resolve Helm 4 --color flag conflict (issue #2280)

This commit resolves issue #2280 where the --color flag causes Helm 4
to consume the next argument, resulting in errors like:
"invalid color mode '--context': must be one of: never, auto, always"

Root Cause:
In Helm 4, the --color flag is parsed by the Helm binary before being
passed to plugins like helm-diff. This causes Helm to interpret the
next argument (e.g., --context) as the value for --color.

Solution:
Remove --color and --no-color flags from helm-diff commands when using
Helm 4, and instead use the HELM_DIFF_COLOR environment variable.
The helm-diff plugin supports HELM_DIFF_COLOR=[true|false] as an
alternative to the --color/--no-color flags.

Changes:
1. Added filterColorFlagsForHelm4() function in pkg/helmexec/exec.go
   - Removes --color and --no-color flags from flags slice
   - Sets HELM_DIFF_COLOR=true for --color
   - Sets HELM_DIFF_COLOR=false for --no-color

2. Modified DiffRelease() to call filterColorFlagsForHelm4() on Helm 4

3. Added comprehensive unit tests in pkg/helmexec/exec_test.go
   - Test_DiffRelease_ColorFlagHelm4: Verifies flags are filtered
   - Test_FilterColorFlagsForHelm4: Tests all flag combinations

4. Added integration test in test/integration/test-cases/issue-2280.sh
   - Tests the exact scenario from issue #2280
   - Verifies --color and --context flags work together
   - Helm 4 only test (skipped on Helm 3)

Fixes #2280

Signed-off-by: Aditya Menon <amenon@canarytechnologies.com>

* refactor: apply Copilot code review nitpicks

This commit addresses minor code quality improvements suggested by
GitHub Copilot's automated review.

Changes:
1. pkg/app/formatters.go - Optimize trimTrailingWhitespace()
   - Only modify lines that actually have trailing whitespace
   - Avoids unnecessary string allocations for clean lines
   - Performance optimization for table formatting

2. test/e2e/template/helmfile/snapshot_test.go
   - Use 0600 permissions for temporary input files (was 0644)
   - Improves security by making temp files owner-only read/write
   - Prevents potential exposure of sensitive test data

   - Improve error messages in getFreePort()
   - Wrap errors with context using fmt.Errorf("%w")
   - Better error debugging when port allocation fails

   - Add retry logic to setupLocalDockerRegistry()
   - Handles race condition where port gets taken between allocation and use
   - Retries up to 3 times with new ports on "address already in use" errors
   - Fails fast on other Docker errors for better test diagnostics

All tests passing. These are non-functional improvements that enhance
code quality, performance, security, and test reliability.

Signed-off-by: Aditya Menon <amenon@canarytechnologies.com>

* docs: improve code comments based on Copilot feedback

This commit addresses documentation nitpicks from GitHub Copilot's
automated review to improve code clarity and maintainability.

Changes:
1. pkg/app/app.go - Clarify detectKubeVersion() return conditions
   - Updated comment to explicitly list all three cases when empty
     string is returned: kubeVersion already set, auto-detection
     disabled, or detection fails
   - Improves function documentation clarity

2. test/e2e/template/helmfile/snapshot_test.go
   - Added reference to retry logic in getFreePort() comment
   - Points callers to setupLocalDockerRegistry() for proper race
     condition handling example
   - Better guidance for future code maintainers

3. pkg/state/state.go - Explain patches check rationale
   - Added comment explaining why --dry-run=server is only enabled
     when patches are used
   - Clarifies that this is a conservative approach to minimize
     unnecessary cluster connections
   - Documents primary use case (Grafana chart with PVC preservation)

All changes are documentation-only with no functional impact.
All tests passing.

Signed-off-by: Aditya Menon <amenon@canarytechnologies.com>

* refactor: enable lookup() for all cluster commands and add defensive check

This commit addresses two Copilot review suggestions to improve code
robustness and functionality.

Changes:
1. pkg/state/state.go - Remove patches requirement for lookup()
   - Previously only enabled --dry-run=server when patches were present
   - Now enables it for ALL cluster-requiring commands
   - Rationale: lookup() function can be used without patches
   - Improves compatibility with charts using lookup() standalone
   - Trade-off: Slightly more cluster connections vs broader support

2. pkg/helmexec/exec.go - Add defensive check for HELM_DIFF_COLOR
   - Only set environment variable if not already present
   - Makes code more defensive for future implementation changes
   - Note: Changes behavior from "last wins" to "first wins"
   - In practice, env map is freshly created so check is precautionary

3. pkg/helmexec/exec_test.go - Update test expectations
   - Changed test case to reflect "first wins" behavior
   - Updated test name and comment for clarity

Breaking behavior change:
- When both --color and --no-color are present, the FIRST flag now
  wins instead of the LAST flag
- This deviates from standard CLI conventions where later flags
  override earlier ones
- However, this is unlikely to affect real usage as users rarely
  specify conflicting flags

All tests passing.

Signed-off-by: Aditya Menon <amenon@canarytechnologies.com>

---------

Signed-off-by: Aditya Menon <amenon@canarytechnologies.com>
2025-11-21 08:32:54 +08:00

2501 lines
66 KiB
Go

package app
import (
"bytes"
goContext "context"
"fmt"
"os"
"path/filepath"
"sort"
"strings"
"sync"
"text/tabwriter"
"github.com/helmfile/vals"
"go.uber.org/zap"
"github.com/helmfile/helmfile/pkg/argparser"
"github.com/helmfile/helmfile/pkg/cluster"
"github.com/helmfile/helmfile/pkg/envvar"
"github.com/helmfile/helmfile/pkg/filesystem"
"github.com/helmfile/helmfile/pkg/helmexec"
"github.com/helmfile/helmfile/pkg/plugins"
"github.com/helmfile/helmfile/pkg/remote"
"github.com/helmfile/helmfile/pkg/state"
)
var CleanWaitGroup sync.WaitGroup
var Cancel goContext.CancelFunc
// App is the main application object.
type App struct {
OverrideKubeContext string
OverrideHelmBinary string
OverrideKustomizeBinary string
EnableLiveOutput bool
StripArgsValuesOnExitError bool
DisableForceUpdate bool
EnforcePluginVerification bool
HelmOCIPlainHTTP bool
DisableKubeVersionAutoDetection bool
Logger *zap.SugaredLogger
Kubeconfig string
Env string
Namespace string
Chart string
Selectors []string
Args string
ValuesFiles []string
Set map[string]any
FileOrDir string
fs *filesystem.FileSystem
remote *remote.Remote
valsRuntime vals.Evaluator
helms map[helmKey]helmexec.Interface
helmsMutex sync.Mutex
ctx goContext.Context
}
type HelmRelease struct {
Name string `json:"name"`
Namespace string `json:"namespace"`
Enabled bool `json:"enabled"`
Installed bool `json:"installed"`
Labels string `json:"labels"`
Chart string `json:"chart"`
Version string `json:"version"`
}
func New(conf ConfigProvider) *App {
ctx := goContext.Background()
ctx, Cancel = goContext.WithCancel(ctx)
return Init(&App{
OverrideKubeContext: conf.KubeContext(),
OverrideHelmBinary: conf.HelmBinary(),
OverrideKustomizeBinary: conf.KustomizeBinary(),
EnableLiveOutput: conf.EnableLiveOutput(),
StripArgsValuesOnExitError: conf.StripArgsValuesOnExitError(),
DisableForceUpdate: conf.DisableForceUpdate(),
EnforcePluginVerification: conf.EnforcePluginVerification(),
HelmOCIPlainHTTP: conf.HelmOCIPlainHTTP(),
Logger: conf.Logger(),
Kubeconfig: conf.Kubeconfig(),
Env: conf.Env(),
Namespace: conf.Namespace(),
Chart: conf.Chart(),
Selectors: conf.Selectors(),
Args: conf.Args(),
FileOrDir: conf.FileOrDir(),
ValuesFiles: conf.StateValuesFiles(),
Set: conf.StateValuesSet(),
fs: filesystem.DefaultFileSystem(),
ctx: ctx,
})
}
func Init(app *App) *App {
var err error
app.valsRuntime, err = plugins.ValsInstance()
if err != nil {
panic(fmt.Sprintf("Failed to initialize vals runtime: %v", err))
}
if app.EnableLiveOutput {
app.Logger.Info("Live output is enabled")
}
return app
}
func (a *App) Init(c InitConfigProvider) error {
runner := &helmexec.ShellRunner{
Logger: a.Logger,
Ctx: a.ctx,
StripArgsValuesOnExitError: a.StripArgsValuesOnExitError,
}
helmfileInit := NewHelmfileInit(a.OverrideHelmBinary, c, a.Logger, runner)
return helmfileInit.Initialize()
}
func (a *App) Deps(c DepsConfigProvider) error {
return a.ForEachState(func(run *Run) (_ bool, errs []error) {
errs = run.Deps(c)
return
}, c.IncludeTransitiveNeeds(), SetFilter(true))
}
func (a *App) Repos(c ReposConfigProvider) error {
return a.ForEachState(func(run *Run) (_ bool, errs []error) {
reposErr := run.Repos(c)
if reposErr != nil {
errs = append(errs, reposErr)
}
return
}, c.IncludeTransitiveNeeds(), SetFilter(true))
}
func (a *App) Diff(c DiffConfigProvider) error {
var allDiffDetectedErrs []error
var affectedAny bool
err := a.ForEachState(func(run *Run) (bool, []error) {
var criticalErrs []error
var msg *string
var matched, affected bool
var errs []error
includeCRDs := !c.SkipCRDs()
prepErr := run.withPreparedCharts("diff", state.ChartPrepareOptions{
SkipRepos: c.SkipRefresh() || c.SkipDeps(),
SkipRefresh: c.SkipRefresh(),
SkipDeps: c.SkipDeps(),
IncludeCRDs: &includeCRDs,
Validate: c.Validate(),
Concurrency: c.Concurrency(),
IncludeTransitiveNeeds: c.IncludeNeeds(),
}, func() {
msg, matched, affected, errs = a.diff(run, c)
})
if msg != nil {
a.Logger.Info(*msg)
}
if prepErr != nil {
errs = append(errs, prepErr)
}
affectedAny = affectedAny || affected
for i := range errs {
switch e := errs[i].(type) {
case *state.ReleaseError:
switch e.Code {
case 2:
// See https://github.com/roboll/helmfile/issues/874
allDiffDetectedErrs = append(allDiffDetectedErrs, e)
default:
criticalErrs = append(criticalErrs, e)
}
default:
criticalErrs = append(criticalErrs, e)
}
}
return matched, criticalErrs
}, c.IncludeTransitiveNeeds())
if err != nil {
return err
}
if c.DetailedExitcode() && (len(allDiffDetectedErrs) > 0 || affectedAny) {
// We take the first release error w/ exit status 2 (although all the deferred errs should have exit status 2)
// to just let helmfile itself to exit with 2
// See https://github.com/roboll/helmfile/issues/749
code := 2
e := &Error{
msg: "Identified at least one change",
code: &code,
}
return e
}
return nil
}
func (a *App) Template(c TemplateConfigProvider) error {
return a.ForEachState(func(run *Run) (ok bool, errs []error) {
includeCRDs := c.IncludeCRDs()
// Live output should never be enabled for the "template" subcommand to avoid breaking `helmfile template | kubectl apply -f -`
run.helm.SetEnableLiveOutput(false)
// Reset helm extra args to not pollute BuildDeps() and AddRepo() on subsequent helmfiles
// https://github.com/helmfile/helmfile/issues/1749
run.helm.SetExtraArgs()
prepErr := run.withPreparedCharts("template", state.ChartPrepareOptions{
SkipRepos: c.SkipRefresh() || c.SkipDeps(),
SkipRefresh: c.SkipRefresh(),
SkipDeps: c.SkipDeps(),
IncludeCRDs: &includeCRDs,
SkipCleanup: c.SkipCleanup(),
Validate: c.Validate(),
Concurrency: c.Concurrency(),
IncludeTransitiveNeeds: c.IncludeNeeds(),
Set: c.Set(),
Values: c.Values(),
KubeVersion: c.KubeVersion(),
HelmOCIPlainHTTP: a.HelmOCIPlainHTTP,
}, func() {
ok, errs = a.template(run, c)
})
if prepErr != nil {
errs = append(errs, prepErr)
}
return
}, c.IncludeTransitiveNeeds())
}
func (a *App) WriteValues(c WriteValuesConfigProvider) error {
return a.ForEachState(func(run *Run) (ok bool, errs []error) {
prepErr := run.withPreparedCharts("write-values", state.ChartPrepareOptions{
SkipRepos: c.SkipRefresh() || c.SkipDeps(),
SkipRefresh: c.SkipRefresh(),
SkipDeps: c.SkipDeps(),
SkipCleanup: c.SkipCleanup(),
Concurrency: c.Concurrency(),
}, func() {
ok, errs = a.writeValues(run, c)
})
if prepErr != nil {
errs = append(errs, prepErr)
}
return
}, c.IncludeTransitiveNeeds(), SetFilter(true))
}
type MultiError struct {
Errors []error
}
func (e *MultiError) Error() string {
indent := func(text string, indent string) string {
lines := strings.Split(text, "\n")
var buf bytes.Buffer
for _, l := range lines {
buf.WriteString(indent)
buf.WriteString(l)
buf.WriteString("\n")
}
return buf.String()
}
lines := []string{fmt.Sprintf("Failed with %d errors:", len(e.Errors))}
for i, err := range e.Errors {
lines = append(lines, fmt.Sprintf("Error %d:\n\n%v", i+1, indent(err.Error(), " ")))
}
return strings.Join(lines, "\n\n")
}
func (a *App) Lint(c LintConfigProvider) error {
var deferredLintErrors []error
err := a.ForEachState(func(run *Run) (ok bool, errs []error) {
var lintErrs []error
// `helm lint` on helm v2 and v3 does not support remote charts, that we need to set `forceDownload=true` here
prepErr := run.withPreparedCharts("lint", state.ChartPrepareOptions{
ForceDownload: true,
SkipRepos: c.SkipRefresh() || c.SkipDeps(),
SkipRefresh: c.SkipRefresh(),
SkipDeps: c.SkipDeps(),
SkipCleanup: c.SkipCleanup(),
Concurrency: c.Concurrency(),
IncludeTransitiveNeeds: c.IncludeNeeds(),
}, func() {
ok, lintErrs, errs = a.lint(run, c)
})
if prepErr != nil {
errs = append(errs, prepErr)
}
if len(lintErrs) > 0 {
deferredLintErrors = append(deferredLintErrors, lintErrs...)
}
return
}, c.IncludeTransitiveNeeds())
if err != nil {
return err
}
if len(deferredLintErrors) > 0 {
return &MultiError{Errors: deferredLintErrors}
}
return nil
}
func (a *App) Fetch(c FetchConfigProvider) error {
return a.ForEachState(func(run *Run) (ok bool, errs []error) {
prepErr := run.withPreparedCharts("pull", state.ChartPrepareOptions{
ForceDownload: true,
SkipRefresh: c.SkipRefresh(),
SkipRepos: c.SkipRefresh() || c.SkipDeps(),
SkipDeps: c.SkipDeps(),
OutputDir: c.OutputDir(),
OutputDirTemplate: c.OutputDirTemplate(),
Concurrency: c.Concurrency(),
}, func() {})
if prepErr != nil {
errs = append(errs, prepErr)
}
return
}, false, SetFilter(true))
}
func (a *App) Sync(c SyncConfigProvider) error {
return a.ForEachState(func(run *Run) (ok bool, errs []error) {
includeCRDs := !c.SkipCRDs()
prepErr := run.withPreparedCharts("sync", state.ChartPrepareOptions{
SkipRepos: c.SkipRefresh() || c.SkipDeps(),
SkipRefresh: c.SkipRefresh(),
SkipDeps: c.SkipDeps(),
Wait: c.Wait(),
WaitRetries: c.WaitRetries(),
WaitForJobs: c.WaitForJobs(),
IncludeCRDs: &includeCRDs,
IncludeTransitiveNeeds: c.IncludeNeeds(),
Validate: c.Validate(),
Concurrency: c.Concurrency(),
}, func() {
ok, errs = a.sync(run, c)
})
if prepErr != nil {
errs = append(errs, prepErr)
}
return
}, c.IncludeTransitiveNeeds())
}
func (a *App) Apply(c ApplyConfigProvider) error {
var any bool
mut := &sync.Mutex{}
var opts []LoadOption
opts = append(opts, SetRetainValuesFiles(c.SkipCleanup()))
err := a.ForEachState(func(run *Run) (ok bool, errs []error) {
includeCRDs := !c.SkipCRDs()
prepErr := run.withPreparedCharts("apply", state.ChartPrepareOptions{
SkipRepos: c.SkipRefresh() || c.SkipDeps(),
SkipRefresh: c.SkipRefresh(),
SkipDeps: c.SkipDeps(),
Wait: c.Wait(),
WaitRetries: c.WaitRetries(),
WaitForJobs: c.WaitForJobs(),
IncludeCRDs: &includeCRDs,
SkipCleanup: c.SkipCleanup(),
Validate: c.Validate(),
Concurrency: c.Concurrency(),
IncludeTransitiveNeeds: c.IncludeNeeds(),
}, func() {
matched, updated, es := a.apply(run, c)
mut.Lock()
any = any || updated
mut.Unlock()
ok, errs = matched, es
})
if prepErr != nil {
errs = append(errs, prepErr)
}
return
}, c.IncludeTransitiveNeeds(), opts...)
if err != nil {
return err
}
if c.DetailedExitcode() && any {
code := 2
return &Error{msg: "", Errors: nil, code: &code}
}
return nil
}
func (a *App) Status(c StatusesConfigProvider) error {
return a.ForEachState(func(run *Run) (ok bool, errs []error) {
err := run.withPreparedCharts("status", state.ChartPrepareOptions{
SkipRepos: true,
SkipDeps: true,
Concurrency: c.Concurrency(),
}, func() {
ok, errs = a.status(run, c)
})
if err != nil {
errs = append(errs, err)
}
return
}, false, SetFilter(true))
}
func (a *App) Destroy(c DestroyConfigProvider) error {
return a.ForEachState(func(run *Run) (ok bool, errs []error) {
if !c.SkipCharts() {
err := run.withPreparedCharts("destroy", state.ChartPrepareOptions{
SkipRepos: c.SkipRefresh() || c.SkipDeps(),
SkipRefresh: c.SkipRefresh(),
SkipDeps: c.SkipDeps(),
Concurrency: c.Concurrency(),
DeleteWait: c.DeleteWait(),
DeleteTimeout: c.DeleteTimeout(),
}, func() {
ok, errs = a.delete(run, true, c)
})
if err != nil {
errs = append(errs, err)
}
} else {
ok, errs = a.delete(run, true, c)
}
return
}, false, SetReverse(true))
}
func (a *App) Test(c TestConfigProvider) error {
return a.ForEachState(func(run *Run) (_ bool, errs []error) {
if c.Cleanup() {
a.Logger.Warnf("warn: requested cleanup will not be applied. " +
"To clean up test resources with Helm 3, you have to remove them manually " +
"or set helm.sh/hook-delete-policy\n")
}
err := run.withPreparedCharts("test", state.ChartPrepareOptions{
SkipRepos: c.SkipRefresh() || c.SkipDeps(),
SkipRefresh: c.SkipRefresh(),
SkipDeps: c.SkipDeps(),
Concurrency: c.Concurrency(),
}, func() {
errs = a.test(run, c)
})
if err != nil {
errs = append(errs, err)
}
return
}, false, SetFilter(true))
}
func (a *App) PrintDAGState(c DAGConfigProvider) error {
var err error
return a.ForEachState(func(run *Run) (ok bool, errs []error) {
err = run.withPreparedCharts("show-dag", state.ChartPrepareOptions{
SkipRepos: true,
SkipDeps: true,
Concurrency: 2,
}, func() {
err = a.dag(run)
if err != nil {
errs = append(errs, err)
}
})
return ok, errs
}, false, SetFilter(true))
}
func (a *App) PrintState(c StateConfigProvider) error {
return a.ForEachState(func(run *Run) (_ bool, errs []error) {
err := run.withPreparedCharts("build", state.ChartPrepareOptions{
SkipRepos: true,
SkipDeps: true,
Concurrency: 2,
}, func() {
if c.EmbedValues() {
for i := range run.state.Releases {
r := run.state.Releases[i]
values, err := run.state.LoadYAMLForEmbedding(&r, r.Values, r.MissingFileHandler, r.ValuesPathPrefix)
if err != nil {
errs = []error{err}
return
}
run.state.Releases[i].Values = values
secrets, err := run.state.LoadYAMLForEmbedding(&r, r.Secrets, r.MissingFileHandler, r.ValuesPathPrefix)
if err != nil {
errs = []error{err}
return
}
run.state.Releases[i].Secrets = secrets
}
}
stateYaml, err := run.state.ToYaml()
if err != nil {
errs = []error{err}
return
}
sourceFile, err := run.state.FullFilePath()
if err != nil {
errs = []error{err}
return
}
fmt.Printf("---\n# Source: %s\n\n%+v", sourceFile, stateYaml)
errs = []error{}
})
if err != nil {
errs = append(errs, err)
}
return
}, false, SetFilter(true))
}
func (a *App) dag(r *Run) error {
st := r.state
batches, err := st.PlanReleases(state.PlanOptions{SelectedReleases: st.Releases, Reverse: false, SkipNeeds: false, IncludeNeeds: true, IncludeTransitiveNeeds: true})
if err != nil {
return err
}
fmt.Print(printDAG(batches))
return nil
}
func (a *App) ListReleases(c ListConfigProvider) error {
releasesChan := make(chan []*HelmRelease, 100)
err := a.ForEachState(func(run *Run) (_ bool, errs []error) {
var stateReleases []*HelmRelease
var err error
if !c.SkipCharts() {
err = run.withPreparedCharts("list", state.ChartPrepareOptions{
SkipRepos: true,
SkipDeps: true,
Concurrency: 2,
}, func() {
rel, err := a.list(run)
if err != nil {
panic(err)
}
stateReleases = rel
})
} else {
stateReleases, err = a.list(run)
}
if err != nil {
errs = append(errs, err)
}
if len(stateReleases) > 0 {
releasesChan <- stateReleases
}
return
}, false, SetFilter(true))
close(releasesChan)
// Collect all releases from channel
var releases []*HelmRelease
for rels := range releasesChan {
releases = append(releases, rels...)
}
if err != nil {
return err
}
// Sort releases to ensure deterministic output order regardless of parallel execution
sort.Slice(releases, func(i, j int) bool {
if releases[i].Namespace != releases[j].Namespace {
return releases[i].Namespace < releases[j].Namespace
}
return releases[i].Name < releases[j].Name
})
if c.Output() == "json" {
err = FormatAsJson(releases)
} else {
err = FormatAsTable(releases)
}
return err
}
func (a *App) list(run *Run) ([]*HelmRelease, error) {
var releases []*HelmRelease
for _, r := range run.state.Releases {
labels := ""
if r.Labels == nil {
r.Labels = map[string]string{}
}
for k, v := range run.state.CommonLabels {
r.Labels[k] = v
}
var keys []string
for k := range r.Labels {
keys = append(keys, k)
}
sort.Strings(keys)
for _, k := range keys {
v := r.Labels[k]
labels = fmt.Sprintf("%s,%s:%s", labels, k, v)
}
labels = strings.Trim(labels, ",")
enabled, err := state.ConditionEnabled(r, run.state.Values())
if err != nil {
return nil, err
}
releases = append(releases, &HelmRelease{
Name: r.Name,
Namespace: r.Namespace,
Installed: r.Desired(),
Enabled: enabled,
Labels: labels,
Chart: r.Chart,
Version: r.Version,
})
}
return releases, nil
}
func (a *App) within(dir string, do func() error) error {
if dir == "." {
return do()
}
prev, err := a.fs.Getwd()
if err != nil {
return fmt.Errorf("failed getting current working directory: %v", err)
}
absDir, err := a.fs.Abs(dir)
if err != nil {
return err
}
a.Logger.Debugf("changing working directory to \"%s\"", absDir)
if err := a.fs.Chdir(absDir); err != nil {
return fmt.Errorf("failed changing working directory to \"%s\": %v", absDir, err)
}
appErr := do()
a.Logger.Debugf("changing working directory back to \"%s\"", prev)
if chdirBackErr := a.fs.Chdir(prev); chdirBackErr != nil {
if appErr != nil {
a.Logger.Warnf("%v", appErr)
}
return fmt.Errorf("failed chaging working directory back to \"%s\": %v", prev, chdirBackErr)
}
return appErr
}
func (a *App) visitStateFiles(fileOrDir string, opts LoadOpts, do func(string, string) error) error {
desiredStateFiles, err := a.findDesiredStateFiles(fileOrDir, opts)
if err != nil {
return appError("", err)
}
for _, relPath := range desiredStateFiles {
var file string
var dir string
if a.fs.DirectoryExistsAt(relPath) {
file = relPath
dir = relPath
} else {
file = filepath.Base(relPath)
dir = filepath.Dir(relPath)
}
a.Logger.Debugf("processing file \"%s\" in directory \"%s\"", file, dir)
absd, errAbsDir := a.fs.Abs(dir)
if errAbsDir != nil {
return errAbsDir
}
err := a.within(absd, func() error {
return do(file, absd)
})
if err != nil {
return appError(fmt.Sprintf("in %s/%s", dir, file), err)
}
}
return nil
}
func (a *App) loadDesiredStateFromYaml(file string, opts ...LoadOpts) (*state.HelmState, error) {
return a.loadDesiredStateFromYamlWithBaseDir(file, "", opts...)
}
func (a *App) loadDesiredStateFromYamlWithBaseDir(file string, baseDir string, opts ...LoadOpts) (*state.HelmState, error) {
var op LoadOpts
if len(opts) > 0 {
op = opts[0]
}
ld := &desiredStateLoader{
fs: a.fs,
env: a.Env,
namespace: a.Namespace,
chart: a.Chart,
logger: a.Logger,
remote: a.remote,
baseDir: baseDir,
overrideKubeContext: a.OverrideKubeContext,
overrideHelmBinary: a.OverrideHelmBinary,
overrideKustomizeBinary: a.OverrideKustomizeBinary,
enableLiveOutput: a.EnableLiveOutput,
getHelm: a.getHelm,
valsRuntime: a.valsRuntime,
}
return ld.Load(file, op)
}
type helmKey struct {
Binary string
Context string
}
func createHelmKey(bin, kubectx string) helmKey {
return helmKey{
Binary: bin,
Context: kubectx,
}
}
// GetHelm returns the global helm exec instance for the specified state that is used for helmfile-wise operation
// like decrypting environment secrets.
//
// This is currently used for running all the helm commands for reconciling releases. But this may change in the future
// once we enable each release to have its own helm binary/version.
func (a *App) getHelm(st *state.HelmState) (helmexec.Interface, error) {
a.helmsMutex.Lock()
defer a.helmsMutex.Unlock()
if a.helms == nil {
a.helms = map[helmKey]helmexec.Interface{}
}
bin := st.DefaultHelmBinary
if bin == "" {
bin = state.DefaultHelmBinary
}
kubeconfig := a.Kubeconfig
kubectx := st.HelmDefaults.KubeContext
key := createHelmKey(bin, kubectx)
if _, ok := a.helms[key]; !ok {
exec, err := helmexec.New(bin, helmexec.HelmExecOptions{
EnableLiveOutput: a.EnableLiveOutput,
DisableForceUpdate: a.DisableForceUpdate,
EnforcePluginVerification: a.EnforcePluginVerification,
HelmOCIPlainHTTP: a.HelmOCIPlainHTTP,
}, a.Logger, kubeconfig, kubectx, &helmexec.ShellRunner{
Logger: a.Logger,
Ctx: a.ctx,
StripArgsValuesOnExitError: a.StripArgsValuesOnExitError,
})
if err != nil {
return nil, err
}
a.helms[key] = exec
}
return a.helms[key], nil
}
func (a *App) visitStates(fileOrDir string, defOpts LoadOpts, converge func(*state.HelmState) (bool, []error)) error {
return a.visitStatesWithContext(fileOrDir, defOpts, converge, nil)
}
func (a *App) processStateFileParallel(relPath string, defOpts LoadOpts, converge func(*state.HelmState) (bool, []error), sharedCtx *Context, errChan chan error, matchChan chan bool) {
var file string
var dir string
if a.fs.DirectoryExistsAt(relPath) {
file = relPath
dir = relPath
} else {
file = filepath.Base(relPath)
dir = filepath.Dir(relPath)
}
absd, errAbsDir := a.fs.Abs(dir)
if errAbsDir != nil {
errChan <- errAbsDir
return
}
opts := defOpts.DeepCopy()
if opts.CalleePath == "" {
opts.CalleePath = file
}
st, err := a.loadDesiredStateFromYamlWithBaseDir(file, absd, opts)
if err != nil {
switch stateLoadErr := err.(type) {
case *state.StateLoadError:
switch stateLoadErr.Cause.(type) {
case *state.UndefinedEnvError:
return
default:
errChan <- appError(fmt.Sprintf("in %s/%s", dir, file), err)
return
}
default:
errChan <- appError(fmt.Sprintf("in %s/%s", dir, file), err)
return
}
}
if st == nil {
return
}
st.Selectors = opts.Selectors
if len(st.Helmfiles) > 0 && !opts.Reverse {
if err := a.processNestedHelmfiles(st, absd, file, defOpts, opts, converge, sharedCtx); err != nil {
errChan <- err
return
}
}
templated, err := st.ExecuteTemplates()
if err != nil {
errChan <- appError(fmt.Sprintf("in %s/%s: failed executing release templates in \"%s\"", dir, file, file), err)
return
}
var errs []error
CleanWaitGroup.Add(1)
var cleanErr error
defer func() {
defer CleanWaitGroup.Done()
cleanErr = context{app: a, st: templated, retainValues: defOpts.RetainValuesFiles}.clean(errs)
}()
processed, errs := converge(templated)
if len(errs) > 0 {
errChan <- errs[0]
return
}
if cleanErr != nil {
errChan <- cleanErr
return
}
// Report if this file had matching releases
if processed {
matchChan <- true
}
if opts.Reverse && len(st.Helmfiles) > 0 {
if err := a.processNestedHelmfiles(st, absd, file, defOpts, opts, converge, sharedCtx); err != nil {
errChan <- err
}
}
}
func (a *App) processNestedHelmfiles(st *state.HelmState, absd, file string, defOpts, opts LoadOpts, converge func(*state.HelmState) (bool, []error), sharedCtx *Context) error {
for i, m := range st.Helmfiles {
optsForNestedState := LoadOpts{
CalleePath: filepath.Join(absd, file),
Environment: m.Environment,
Reverse: defOpts.Reverse,
RetainValuesFiles: defOpts.RetainValuesFiles,
}
if (m.Selectors == nil && !isExplicitSelectorInheritanceEnabled()) || m.SelectorsInherited {
optsForNestedState.Selectors = opts.Selectors
} else {
optsForNestedState.Selectors = m.Selectors
}
if err := a.visitStatesWithContext(m.Path, optsForNestedState, converge, sharedCtx); err != nil {
switch err.(type) {
case *NoMatchingHelmfileError:
default:
return appError(fmt.Sprintf("in .helmfiles[%d]", i), err)
}
}
}
return nil
}
func (a *App) visitStatesWithContext(fileOrDir string, defOpts LoadOpts, converge func(*state.HelmState) (bool, []error), sharedCtx *Context) error {
noMatchInHelmfiles := true
desiredStateFiles, err := a.findDesiredStateFiles(fileOrDir, defOpts)
if len(desiredStateFiles) > 1 {
var wg sync.WaitGroup
errChan := make(chan error, len(desiredStateFiles))
matchChan := make(chan bool, len(desiredStateFiles))
for _, relPath := range desiredStateFiles {
wg.Add(1)
go func(relPath string) {
defer wg.Done()
a.processStateFileParallel(relPath, defOpts, converge, sharedCtx, errChan, matchChan)
}(relPath)
}
wg.Wait()
close(errChan)
close(matchChan)
for err := range errChan {
if err != nil {
return err
}
}
// Check if any files had matching releases
for range matchChan {
noMatchInHelmfiles = false
}
} else {
// Sequential processing for single file
err = a.visitStateFiles(fileOrDir, defOpts, func(f, d string) (retErr error) {
opts := defOpts.DeepCopy()
if opts.CalleePath == "" {
opts.CalleePath = f
}
st, err := a.loadDesiredStateFromYaml(f, opts)
ctx := context{app: a, st: st, retainValues: defOpts.RetainValuesFiles}
if err != nil {
switch stateLoadErr := err.(type) {
case *state.StateLoadError:
switch stateLoadErr.Cause.(type) {
case *state.UndefinedEnvError:
return nil
default:
return ctx.wrapErrs(err)
}
default:
return ctx.wrapErrs(err)
}
}
st.Selectors = opts.Selectors
visitSubHelmfiles := func() error {
if len(st.Helmfiles) > 0 {
noMatchInSubHelmfiles := true
for i, m := range st.Helmfiles {
optsForNestedState := LoadOpts{
CalleePath: filepath.Join(d, f),
Environment: m.Environment,
Reverse: defOpts.Reverse,
RetainValuesFiles: defOpts.RetainValuesFiles,
}
if (m.Selectors == nil && !isExplicitSelectorInheritanceEnabled()) || m.SelectorsInherited {
optsForNestedState.Selectors = opts.Selectors
} else {
optsForNestedState.Selectors = m.Selectors
}
if err := a.visitStates(m.Path, optsForNestedState, converge); err != nil {
switch err.(type) {
case *NoMatchingHelmfileError:
default:
return appError(fmt.Sprintf("in .helmfiles[%d]", i), err)
}
} else {
noMatchInSubHelmfiles = false
}
}
noMatchInHelmfiles = noMatchInHelmfiles && noMatchInSubHelmfiles
}
return nil
}
if !opts.Reverse {
err = visitSubHelmfiles()
if err != nil {
return err
}
}
templated, tmplErr := st.ExecuteTemplates()
if tmplErr != nil {
return appError(fmt.Sprintf("failed executing release templates in \"%s\"", f), tmplErr)
}
var (
processed bool
errs []error
)
CleanWaitGroup.Add(1)
defer func() {
defer CleanWaitGroup.Done()
cleanErr := context{app: a, st: templated, retainValues: defOpts.RetainValuesFiles}.clean(errs)
if retErr == nil {
retErr = cleanErr
} else if cleanErr != nil {
a.Logger.Debugf("Failed to clean up temporary files generated while processing %q: %v", templated.FilePath, cleanErr)
}
}()
processed, errs = converge(templated)
noMatchInHelmfiles = noMatchInHelmfiles && !processed
if opts.Reverse {
err = visitSubHelmfiles()
if err != nil {
return err
}
}
return nil
})
}
if err != nil {
return err
}
if noMatchInHelmfiles {
return &NoMatchingHelmfileError{selectors: a.Selectors, env: a.Env}
}
return nil
}
type LoadOption func(o *LoadOpts)
var (
SetReverse = func(r bool) func(o *LoadOpts) {
return func(o *LoadOpts) {
o.Reverse = r
}
}
SetRetainValuesFiles = func(r bool) func(o *LoadOpts) {
return func(o *LoadOpts) {
o.RetainValuesFiles = r
}
}
SetFilter = func(f bool) func(o *LoadOpts) {
return func(o *LoadOpts) {
o.Filter = f
}
}
)
func (a *App) ForEachState(do func(*Run) (bool, []error), includeTransitiveNeeds bool, o ...LoadOption) error {
ctx := NewContext()
err := a.visitStatesWithSelectorsAndRemoteSupportWithContext(a.FileOrDir, func(st *state.HelmState) (bool, []error) {
helm, err := a.getHelm(st)
if err != nil {
return false, []error{err}
}
run, err := NewRun(st, helm, &ctx)
if err != nil {
return false, []error{err}
}
return do(run)
}, includeTransitiveNeeds, &ctx, o...)
return err
}
func printBatches(batches [][]state.Release) string {
buf := &bytes.Buffer{}
w := new(tabwriter.Writer)
w.Init(buf, 0, 1, 1, ' ', 0)
_, _ = fmt.Fprintln(w, "GROUP\tRELEASES")
for i, batch := range batches {
ids := []string{}
for _, r := range batch {
ids = append(ids, state.ReleaseToID(&r.ReleaseSpec))
}
_, _ = fmt.Fprintf(w, "%d\t%s\n", i+1, strings.Join(ids, ", "))
}
_ = w.Flush()
return buf.String()
}
func printDAG(batches [][]state.Release) string {
buf := &bytes.Buffer{}
w := new(tabwriter.Writer)
w.Init(buf, 0, 1, 1, ' ', 0)
_, _ = fmt.Fprintln(w, "GROUP\tRELEASE\tDEPENDENCIES")
for i, batch := range batches {
for _, r := range batch {
id := state.ReleaseToID(&r.ReleaseSpec)
needs := r.Needs
_, _ = fmt.Fprintf(w, "%d\t%s\t%s\n", i+1, id, strings.Join(needs, ", "))
}
}
_ = w.Flush()
return trimTrailingWhitespace(buf.String())
}
// nolint: unparam
func withDAG(templated *state.HelmState, helm helmexec.Interface, logger *zap.SugaredLogger, opts state.PlanOptions, converge func(*state.HelmState, helmexec.Interface) (bool, []error)) (bool, []error) {
batches, err := templated.PlanReleases(opts)
if err != nil {
return false, []error{err}
}
return withBatches(opts.Purpose, templated, batches, helm, logger, converge)
}
func withBatches(purpose string, templated *state.HelmState, batches [][]state.Release, helm helmexec.Interface, logger *zap.SugaredLogger, converge func(*state.HelmState, helmexec.Interface) (bool, []error)) (bool, []error) {
numBatches := len(batches)
if purpose == "" {
purpose = "processing"
}
logger.Debugf("%s %d groups of releases in this order:\n%s", purpose, numBatches, printBatches(batches))
any := false
for i, batch := range batches {
var targets []state.ReleaseSpec
for _, marked := range batch {
targets = append(targets, marked.ReleaseSpec)
}
var releaseIds []string
for _, r := range targets {
release := r
releaseIds = append(releaseIds, state.ReleaseToID(&release))
}
logger.Debugf("%s releases in group %d/%d: %s", purpose, i+1, numBatches, strings.Join(releaseIds, ", "))
batchSt := *templated
batchSt.Releases = targets
processed, errs := converge(&batchSt, helm)
if len(errs) > 0 {
return false, errs
}
any = any || processed
}
return any, nil
}
type Opts struct {
DAGEnabled bool
}
func (a *App) visitStatesWithSelectorsAndRemoteSupportWithContext(fileOrDir string, converge func(*state.HelmState) (bool, []error), includeTransitiveNeeds bool, sharedCtx *Context, opt ...LoadOption) error {
opts := LoadOpts{
Selectors: a.Selectors,
}
for _, o := range opt {
o(&opts)
}
envvals := []any{}
for _, v := range a.ValuesFiles {
envvals = append(envvals, v)
}
if len(a.Set) > 0 {
envvals = append(envvals, a.Set)
}
if len(envvals) > 0 {
opts.Environment.OverrideValues = envvals
}
a.remote = remote.NewRemote(a.Logger, "", a.fs)
f := converge
if opts.Filter {
f = func(st *state.HelmState) (bool, []error) {
return processFilteredReleases(st, func(st *state.HelmState) []error {
_, err := converge(st)
return err
},
includeTransitiveNeeds)
}
}
// pre-handles HelmState
fHelmStatsWithOverrides := func(st *state.HelmState) (bool, []error) {
var err error
// override release settings
st.Releases, err = st.GetReleasesWithOverrides()
if err != nil {
return false, []error{err}
}
// override release labels
st.Releases = st.GetReleasesWithLabels()
return f(st)
}
return a.visitStatesWithContext(fileOrDir, opts, fHelmStatsWithOverrides, sharedCtx)
}
func processFilteredReleases(st *state.HelmState, converge func(st *state.HelmState) []error, includeTransitiveNeeds bool) (bool, []error) {
if len(st.Selectors) > 0 {
err := st.FilterReleases(includeTransitiveNeeds)
if err != nil {
return false, []error{err}
}
}
if err := checkDuplicates(st.Releases); err != nil {
return false, []error{err}
}
errs := converge(st)
processed := len(st.Releases) != 0 && len(errs) == 0
return processed, errs
}
func checkDuplicates(releases []state.ReleaseSpec) error {
type Key struct {
Namespace, Name, KubeContext string
}
releaseNameCounts := map[Key]int{}
for _, r := range releases {
namespace := r.Namespace
releaseNameCounts[Key{namespace, r.Name, r.KubeContext}]++
}
for name, c := range releaseNameCounts {
if c > 1 {
var msg string
if name.Namespace != "" {
msg += fmt.Sprintf(" in namespace %q", name.Namespace)
}
if name.KubeContext != "" {
msg += fmt.Sprintf(" in kubecontext %q", name.KubeContext)
}
return fmt.Errorf("duplicate release %q found%s: there were %d releases named \"%s\" matching specified selector", name.Name, msg, c, name.Name)
}
}
return nil
}
func (a *App) Wrap(converge func(*state.HelmState, helmexec.Interface) []error) func(st *state.HelmState, helm helmexec.Interface, includeTransitiveNeeds bool) (bool, []error) {
return func(st *state.HelmState, helm helmexec.Interface, includeTransitiveNeeds bool) (bool, []error) {
return processFilteredReleases(st, func(st *state.HelmState) []error {
return converge(st, helm)
}, includeTransitiveNeeds)
}
}
func (a *App) WrapWithoutSelector(converge func(*state.HelmState, helmexec.Interface) []error) func(st *state.HelmState, helm helmexec.Interface) (bool, []error) {
return func(st *state.HelmState, helm helmexec.Interface) (bool, []error) {
errs := converge(st, helm)
processed := len(st.Releases) != 0 && len(errs) == 0
return processed, errs
}
}
func (a *App) findDesiredStateFiles(specifiedPath string, opts LoadOpts) ([]string, error) {
path, err := a.remote.Locate(specifiedPath, "states")
if err != nil {
return nil, fmt.Errorf("locate: %v", err)
}
if specifiedPath != path {
a.Logger.Debugf("fetched remote \"%s\" to local cache \"%s\" and loading the latter...", specifiedPath, path)
}
specifiedPath = path
var helmfileDir string
if specifiedPath != "" {
switch {
case a.fs.FileExistsAt(specifiedPath):
return []string{specifiedPath}, nil
case a.fs.DirectoryExistsAt(specifiedPath):
helmfileDir = specifiedPath
default:
return []string{}, fmt.Errorf("specified state file %s is not found", specifiedPath)
}
} else {
var defaultFile string
DefaultGotmplHelmfile := DefaultHelmfile + ".gotmpl"
if a.fs.FileExistsAt(DefaultHelmfile) && a.fs.FileExistsAt(DefaultGotmplHelmfile) {
return []string{}, fmt.Errorf("both %s and %s.gotmpl exist. Please remove one of them", DefaultHelmfile, DefaultHelmfile)
}
switch {
case a.fs.FileExistsAt(DefaultHelmfile):
defaultFile = DefaultHelmfile
case a.fs.FileExistsAt(DefaultGotmplHelmfile):
defaultFile = DefaultGotmplHelmfile
}
switch {
case a.fs.DirectoryExistsAt(DefaultHelmfileDirectory):
if defaultFile != "" {
return []string{}, fmt.Errorf("configuration conlict error: you can have either %s or %s, but not both", defaultFile, DefaultHelmfileDirectory)
}
helmfileDir = DefaultHelmfileDirectory
case defaultFile != "":
return []string{defaultFile}, nil
default:
return []string{}, fmt.Errorf("no state file found. It must be named %s/*.{yaml,yml,yaml.gotmpl,yml.gotmpl}, %s, or %s, otherwise specified with the --file flag or %s environment variable", DefaultHelmfileDirectory, DefaultHelmfile, DefaultGotmplHelmfile, envvar.FilePath)
}
}
files := []string{}
ymlFiles, err := a.fs.Glob(filepath.Join(helmfileDir, "*.y*ml"))
if err != nil {
return []string{}, err
}
gotmplFiles, err := a.fs.Glob(filepath.Join(helmfileDir, "*.y*ml.gotmpl"))
if err != nil {
return []string{}, err
}
files = append(files, ymlFiles...)
files = append(files, gotmplFiles...)
if opts.Reverse {
sort.Slice(files, func(i, j int) bool {
return files[j] < files[i]
})
} else {
sort.Slice(files, func(i, j int) bool {
return files[i] < files[j]
})
}
a.Logger.Debugf("found %d helmfile state files in %s: %s", len(ymlFiles)+len(gotmplFiles), helmfileDir, strings.Join(files, ", "))
return files, nil
}
func (a *App) getSelectedReleases(r *Run, includeTransitiveNeeds bool) ([]state.ReleaseSpec, []state.ReleaseSpec, error) {
selected, err := r.state.GetSelectedReleases(includeTransitiveNeeds)
if err != nil {
return nil, nil, err
}
selectedIds := map[string]state.ReleaseSpec{}
selectedCounts := map[string]int{}
for _, r := range selected {
id := state.ReleaseToID(&r)
selectedIds[id] = r
selectedCounts[id]++
if dupCount := selectedCounts[id]; dupCount > 1 {
return nil, nil, fmt.Errorf("found %d duplicate releases with ID %q", dupCount, id)
}
}
allReleases := r.state.Releases
groupsByID := map[string][]*state.ReleaseSpec{}
for _, r := range allReleases {
groupsByID[state.ReleaseToID(&r)] = append(groupsByID[state.ReleaseToID(&r)], &r)
}
var deduplicated []state.ReleaseSpec
dedupedBefore := map[string]struct{}{}
// We iterate over allReleases rather than groupsByID
// to preserve the order of releases
for _, seq := range allReleases {
release := seq
id := state.ReleaseToID(&release)
rs := groupsByID[id]
if len(rs) == 1 {
deduplicated = append(deduplicated, *rs[0])
continue
}
if _, ok := dedupedBefore[id]; ok {
continue
}
// We keep the selected one only when there were two or more duplicate
// releases in the helmfile config.
// Otherwise we can't compute the DAG of releases correctly.
r, deduped := selectedIds[id]
if deduped {
deduplicated = append(deduplicated, r)
dedupedBefore[id] = struct{}{}
}
}
if err := checkDuplicates(deduplicated); err != nil {
return nil, nil, err
}
var extra string
if len(r.state.Selectors) > 0 {
extra = " matching " + strings.Join(r.state.Selectors, ",")
}
a.Logger.Debugf("%d release(s)%s found in %s\n", len(selected), extra, r.state.FilePath)
return selected, deduplicated, nil
}
func (a *App) apply(r *Run, c ApplyConfigProvider) (bool, bool, []error) {
st := r.state
helm := r.helm
helm.SetExtraArgs(GetArgs(c.Args(), r.state)...)
selectedReleases, selectedAndNeededReleases, err := a.getSelectedReleases(r, c.IncludeTransitiveNeeds())
if err != nil {
return false, false, []error{err}
}
if len(selectedReleases) == 0 {
return false, false, nil
}
// This is required when you're trying to deduplicate releases by the selector.
// Without this, `PlanReleases` conflates duplicates and return both in `batches`,
// even if we provided `SelectedReleases: selectedReleases`.
// See https://github.com/roboll/helmfile/issues/1818 for more context.
st.Releases = selectedAndNeededReleases
plan, err := st.PlanReleases(state.PlanOptions{Reverse: false, SelectedReleases: selectedReleases, SkipNeeds: c.SkipNeeds(), IncludeNeeds: c.IncludeNeeds(), IncludeTransitiveNeeds: c.IncludeTransitiveNeeds()})
if err != nil {
return false, false, []error{err}
}
var toApplyWithNeeds []state.ReleaseSpec
for _, rs := range plan {
for _, r := range rs {
toApplyWithNeeds = append(toApplyWithNeeds, r.ReleaseSpec)
}
}
// Do build deps and prepare only on selected releases so that we won't waste time
// on running various helm commands on unnecessary releases
st.Releases = toApplyWithNeeds
// helm must be 2.11+ and helm-diff should be provided `--detailed-exitcode` in order for `helmfile apply` to work properly
detailedExitCode := true
detectedKubeVersion := a.detectKubeVersion(st)
diffOpts := &state.DiffOpts{
Color: c.Color(),
NoColor: c.NoColor(),
Context: c.Context(),
Output: c.DiffOutput(),
Set: c.Set(),
SkipCleanup: c.SkipCleanup(),
SkipDiffOnInstall: c.SkipDiffOnInstall(),
ReuseValues: c.ReuseValues(),
ResetValues: c.ResetValues(),
DiffArgs: c.DiffArgs(),
PostRenderer: c.PostRenderer(),
PostRendererArgs: c.PostRendererArgs(),
SkipSchemaValidation: c.SkipSchemaValidation(),
SuppressOutputLineRegex: c.SuppressOutputLineRegex(),
TakeOwnership: c.TakeOwnership(),
DetectedKubeVersion: detectedKubeVersion,
}
infoMsg, releasesToBeUpdated, releasesToBeDeleted, errs := r.diff(false, detailedExitCode, c, diffOpts)
if len(errs) > 0 {
return false, false, errs
}
var toDelete []state.ReleaseSpec
for _, r := range releasesToBeDeleted {
toDelete = append(toDelete, r)
}
var toUpdate []state.ReleaseSpec
for _, r := range releasesToBeUpdated {
toUpdate = append(toUpdate, r)
}
releasesWithNoChange := map[string]state.ReleaseSpec{}
for _, r := range toApplyWithNeeds {
release := r
id := state.ReleaseToID(&release)
_, uninstalled := releasesToBeDeleted[id]
_, updated := releasesToBeUpdated[id]
if !uninstalled && !updated {
releasesWithNoChange[id] = release
}
}
infoMsgStr := ""
if infoMsg != nil {
infoMsgStr = *infoMsg
}
confMsg := fmt.Sprintf(`%s
Do you really want to apply?
Helmfile will apply all your changes, as shown above.
`, infoMsgStr)
interactive := c.Interactive()
if !interactive && infoMsgStr != "" {
a.Logger.Debug(infoMsgStr)
}
var applyErrs []error
affectedReleases := state.AffectedReleases{}
// Traverse DAG of all the releases so that we don't suffer from false-positive missing dependencies
st.Releases = selectedAndNeededReleases
if !interactive || interactive && r.askForConfirmation(confMsg) {
if _, preapplyErrors := withDAG(st, helm, a.Logger, state.PlanOptions{Purpose: "invoking preapply hooks for", Reverse: true, SelectedReleases: toApplyWithNeeds, SkipNeeds: true}, a.WrapWithoutSelector(func(subst *state.HelmState, helm helmexec.Interface) []error {
for _, r := range subst.Releases {
release := r
if _, err := st.TriggerPreapplyEvent(&release, "apply"); err != nil {
return []error{err}
}
}
return nil
})); len(preapplyErrors) > 0 {
return true, false, preapplyErrors
}
// We deleted releases by traversing the DAG in reverse order
if len(releasesToBeDeleted) > 0 {
_, deletionErrs := withDAG(st, helm, a.Logger, state.PlanOptions{Reverse: true, SelectedReleases: toDelete, SkipNeeds: true}, a.WrapWithoutSelector(func(subst *state.HelmState, helm helmexec.Interface) []error {
var rs []state.ReleaseSpec
for _, r := range subst.Releases {
release := r
if r2, ok := releasesToBeDeleted[state.ReleaseToID(&release)]; ok {
rs = append(rs, r2)
}
}
subst.Releases = rs
return subst.DeleteReleasesForSync(&affectedReleases, helm, c.Concurrency(), c.Cascade())
}))
if len(deletionErrs) > 0 {
applyErrs = append(applyErrs, deletionErrs...)
}
}
// We upgrade releases by traversing the DAG
if len(releasesToBeUpdated) > 0 {
_, updateErrs := withDAG(st, helm, a.Logger, state.PlanOptions{SelectedReleases: toUpdate, Reverse: false, SkipNeeds: true, IncludeTransitiveNeeds: c.IncludeTransitiveNeeds()}, a.WrapWithoutSelector(func(subst *state.HelmState, helm helmexec.Interface) []error {
var rs []state.ReleaseSpec
for _, r := range subst.Releases {
release := r
if r2, ok := releasesToBeUpdated[state.ReleaseToID(&release)]; ok {
rs = append(rs, r2)
}
}
subst.Releases = rs
syncOpts := &state.SyncOpts{
Set: c.Set(),
SkipCleanup: c.SkipCleanup(),
SkipCRDs: c.SkipCRDs(),
Wait: c.Wait(),
WaitRetries: c.WaitRetries(),
WaitForJobs: c.WaitForJobs(),
ReuseValues: c.ReuseValues(),
ResetValues: c.ResetValues(),
PostRenderer: c.PostRenderer(),
PostRendererArgs: c.PostRendererArgs(),
SkipSchemaValidation: c.SkipSchemaValidation(),
SyncArgs: c.SyncArgs(),
HideNotes: c.HideNotes(),
TakeOwnership: c.TakeOwnership(),
SyncReleaseLabels: c.SyncReleaseLabels(),
}
return subst.SyncReleases(&affectedReleases, helm, c.Values(), c.Concurrency(), syncOpts)
}))
if len(updateErrs) > 0 {
applyErrs = append(applyErrs, updateErrs...)
}
}
}
affectedReleases.DisplayAffectedReleases(c.Logger())
for id := range releasesWithNoChange {
r := releasesWithNoChange[id]
if _, err := st.TriggerCleanupEvent(&r, "apply"); err != nil {
a.Logger.Warnf("warn: %v\n", err)
}
}
if releasesToBeDeleted == nil && releasesToBeUpdated == nil {
return true, false, nil
}
return true, true, applyErrs
}
func (a *App) delete(r *Run, purge bool, c DestroyConfigProvider) (bool, []error) {
st := r.state
helm := r.helm
affectedReleases := state.AffectedReleases{}
toSync, _, err := a.getSelectedReleases(r, false)
if err != nil {
return false, []error{err}
}
if len(toSync) == 0 {
return false, nil
}
toDelete, err := st.DetectReleasesToBeDeleted(helm, toSync)
if err != nil {
return false, []error{err}
}
releasesToDelete := map[string]state.ReleaseSpec{}
for _, r := range toDelete {
release := r
id := state.ReleaseToID(&release)
releasesToDelete[id] = release
}
releasesWithNoChange := map[string]state.ReleaseSpec{}
for _, r := range toSync {
release := r
id := state.ReleaseToID(&release)
_, uninstalled := releasesToDelete[id]
if !uninstalled {
releasesWithNoChange[id] = release
}
}
for id := range releasesWithNoChange {
r := releasesWithNoChange[id]
if _, err := st.TriggerCleanupEvent(&r, "delete"); err != nil {
a.Logger.Warnf("warn: %v\n", err)
}
}
names := make([]string, len(toSync))
for i, r := range toSync {
names[i] = fmt.Sprintf(" %s (%s)", r.Name, r.Chart)
}
var errs []error
msg := fmt.Sprintf(`Affected releases are:
%s
Do you really want to delete?
Helmfile will delete all your releases, as shown above.
`, strings.Join(names, "\n"))
interactive := c.Interactive()
if !interactive || interactive && r.askForConfirmation(msg) {
r.helm.SetExtraArgs(GetArgs(c.Args(), r.state)...)
if len(releasesToDelete) > 0 {
_, deletionErrs := withDAG(st, helm, a.Logger, state.PlanOptions{SelectedReleases: toDelete, Reverse: true, SkipNeeds: true}, a.WrapWithoutSelector(func(subst *state.HelmState, helm helmexec.Interface) []error {
return subst.DeleteReleases(&affectedReleases, helm, c.Concurrency(), purge, c.Cascade())
}))
if len(deletionErrs) > 0 {
errs = append(errs, deletionErrs...)
}
}
}
affectedReleases.DisplayAffectedReleases(c.Logger())
return true, errs
}
// detectKubeVersion auto-detects the Kubernetes cluster version if not specified in helmfile.yaml.
// This prevents helm-diff from falling back to v1.20.0 (issue #2275).
// Returns empty string when kubeVersion is already set in helmfile.yaml (not needed),
// when auto-detection is disabled, or if detection fails.
func (a *App) detectKubeVersion(st *state.HelmState) string {
if st.KubeVersion != "" {
return ""
}
// Allow tests to disable auto-detection to avoid connecting to real clusters
if a.DisableKubeVersionAutoDetection {
return ""
}
version, err := cluster.DetectServerVersion(a.Kubeconfig, a.OverrideKubeContext)
if err != nil {
// If detection fails, we silently continue - helm-diff will handle it
return ""
}
return version
}
func (a *App) diff(r *Run, c DiffConfigProvider) (*string, bool, bool, []error) {
var (
infoMsg *string
updated, deleted map[string]state.ReleaseSpec
)
ok, errs := a.withNeeds(r, c, true, func(st *state.HelmState) []error {
helm := r.helm
var errs []error
helm.SetExtraArgs(GetArgs(c.Args(), r.state)...)
detectedKubeVersion := a.detectKubeVersion(st)
opts := &state.DiffOpts{
Context: c.Context(),
Output: c.DiffOutput(),
Color: c.Color(),
NoColor: c.NoColor(),
Set: c.Set(),
DiffArgs: c.DiffArgs(),
SkipDiffOnInstall: c.SkipDiffOnInstall(),
ReuseValues: c.ReuseValues(),
ResetValues: c.ResetValues(),
PostRenderer: c.PostRenderer(),
PostRendererArgs: c.PostRendererArgs(),
SkipSchemaValidation: c.SkipSchemaValidation(),
SuppressOutputLineRegex: c.SuppressOutputLineRegex(),
TakeOwnership: c.TakeOwnership(),
DetectedKubeVersion: detectedKubeVersion,
}
filtered := &Run{
state: st,
helm: helm,
ctx: r.ctx,
Ask: r.Ask,
}
infoMsg, updated, deleted, errs = filtered.diff(true, c.DetailedExitcode(), c, opts)
return errs
})
return infoMsg, ok, len(deleted) > 0 || len(updated) > 0, errs
}
func (a *App) lint(r *Run, c LintConfigProvider) (bool, []error, []error) {
var deferredLintErrs []error
ok, errs := a.withNeeds(r, c, false, func(st *state.HelmState) []error {
helm := r.helm
args := GetArgs(c.Args(), st)
// Reset the extra args if already set, not to break `helm fetch` by adding the args intended for `lint`
helm.SetExtraArgs()
if len(args) > 0 {
helm.SetExtraArgs(args...)
}
opts := &state.LintOpts{
Set: c.Set(),
SkipCleanup: c.SkipCleanup(),
}
lintErrs := st.LintReleases(helm, c.Values(), args, c.Concurrency(), opts)
if len(lintErrs) == 1 {
if err, ok := lintErrs[0].(helmexec.ExitError); ok {
if err.Code > 0 {
deferredLintErrs = append(deferredLintErrs, err)
return nil
}
}
}
return lintErrs
})
return ok, deferredLintErrs, errs
}
func (a *App) status(r *Run, c StatusesConfigProvider) (bool, []error) {
st := r.state
helm := r.helm
allReleases := st.Releases
selectedReleases, selectedAndNeededReleases, err := a.getSelectedReleases(r, false)
if err != nil {
return false, []error{err}
}
if len(selectedReleases) == 0 {
return false, nil
}
// Do build deps and prepare only on selected releases so that we won't waste time
// on running various helm commands on unnecessary releases
st.Releases = selectedAndNeededReleases
var toStatus []state.ReleaseSpec
for _, r := range selectedReleases {
if r.Desired() {
toStatus = append(toStatus, r)
}
}
var errs []error
// Traverse DAG of all the releases so that we don't suffer from false-positive missing dependencies
st.Releases = allReleases
args := GetArgs(c.Args(), st)
// Reset the extra args if already set, not to break `helm fetch` by adding the args intended for `lint`
helm.SetExtraArgs()
if len(args) > 0 {
helm.SetExtraArgs(args...)
}
if len(toStatus) > 0 {
_, templateErrs := withDAG(st, helm, a.Logger, state.PlanOptions{SelectedReleases: toStatus, Reverse: false, SkipNeeds: true}, a.WrapWithoutSelector(func(subst *state.HelmState, helm helmexec.Interface) []error {
return subst.ReleaseStatuses(helm, c.Concurrency())
}))
if len(templateErrs) > 0 {
errs = append(errs, templateErrs...)
}
}
return true, errs
}
func (a *App) sync(r *Run, c SyncConfigProvider) (bool, []error) {
st := r.state
helm := r.helm
selectedReleases, selectedAndNeededReleases, err := a.getSelectedReleases(r, c.IncludeTransitiveNeeds())
if err != nil {
return false, []error{err}
}
if len(selectedReleases) == 0 {
return false, nil
}
// This is required when you're trying to deduplicate releases by the selector.
// Without this, `PlanReleases` conflates duplicates and return both in `batches`,
// even if we provided `SelectedReleases: selectedReleases`.
// See https://github.com/roboll/helmfile/issues/1818 for more context.
st.Releases = selectedAndNeededReleases
batches, err := st.PlanReleases(state.PlanOptions{Reverse: false, SelectedReleases: selectedReleases, IncludeNeeds: c.IncludeNeeds(), IncludeTransitiveNeeds: c.IncludeTransitiveNeeds(), SkipNeeds: c.SkipNeeds()})
if err != nil {
return false, []error{err}
}
var toSyncWithNeeds []state.ReleaseSpec
for _, rs := range batches {
for _, r := range rs {
toSyncWithNeeds = append(toSyncWithNeeds, r.ReleaseSpec)
}
}
// Do build deps and prepare only on selected releases so that we won't waste time
// on running various helm commands on unnecessary releases
st.Releases = toSyncWithNeeds
toDelete, err := st.DetectReleasesToBeDeletedForSync(helm, toSyncWithNeeds)
if err != nil {
return false, []error{err}
}
releasesToDelete := map[string]state.ReleaseSpec{}
for _, r := range toDelete {
release := r
id := state.ReleaseToID(&release)
releasesToDelete[id] = release
}
var toUpdate []state.ReleaseSpec
for _, r := range toSyncWithNeeds {
release := r
if _, deleted := releasesToDelete[state.ReleaseToID(&release)]; !deleted {
if r.Desired() {
toUpdate = append(toUpdate, release)
}
// TODO Emit error when the user opted to fail when the needed release is disabled,
// instead of silently ignoring it.
// See https://github.com/roboll/helmfile/issues/1018
}
}
releasesToUpdate := map[string]state.ReleaseSpec{}
for _, r := range toUpdate {
release := r
id := state.ReleaseToID(&release)
releasesToUpdate[id] = release
}
releasesWithNoChange := map[string]state.ReleaseSpec{}
for _, r := range toSyncWithNeeds {
release := r
id := state.ReleaseToID(&release)
_, uninstalled := releasesToDelete[id]
_, updated := releasesToUpdate[id]
if !uninstalled && !updated {
releasesWithNoChange[id] = release
}
}
for id := range releasesWithNoChange {
r := releasesWithNoChange[id]
if _, err := st.TriggerCleanupEvent(&r, "sync"); err != nil {
a.Logger.Warnf("warn: %v\n", err)
}
}
names := []string{}
for _, r := range releasesToUpdate {
names = append(names, fmt.Sprintf(" %s (%s) UPDATED", r.Name, r.Chart))
}
for _, r := range releasesToDelete {
names = append(names, fmt.Sprintf(" %s (%s) DELETED", r.Name, r.Chart))
}
// Make the output deterministic for testing purpose
sort.Strings(names)
infoMsg := fmt.Sprintf(`Affected releases are:
%s
`, strings.Join(names, "\n"))
confMsg := fmt.Sprintf(`%s
Do you really want to sync?
Helmfile will sync all your releases, as shown above.
`, infoMsg)
interactive := c.Interactive()
if !interactive {
a.Logger.Debug(infoMsg)
}
var errs []error
r.helm.SetExtraArgs(GetArgs(c.Args(), r.state)...)
// Traverse DAG of all the releases so that we don't suffer from false-positive missing dependencies
st.Releases = selectedAndNeededReleases
affectedReleases := state.AffectedReleases{}
if !interactive || interactive && r.askForConfirmation(confMsg) {
if len(releasesToDelete) > 0 {
_, deletionErrs := withDAG(st, helm, a.Logger, state.PlanOptions{Reverse: true, SelectedReleases: toDelete, SkipNeeds: true}, a.WrapWithoutSelector(func(subst *state.HelmState, helm helmexec.Interface) []error {
var rs []state.ReleaseSpec
for _, r := range subst.Releases {
release := r
if r2, ok := releasesToDelete[state.ReleaseToID(&release)]; ok {
rs = append(rs, r2)
}
}
subst.Releases = rs
return subst.DeleteReleasesForSync(&affectedReleases, helm, c.Concurrency(), c.Cascade())
}))
if len(deletionErrs) > 0 {
errs = append(errs, deletionErrs...)
}
}
if len(releasesToUpdate) > 0 {
_, syncErrs := withDAG(st, helm, a.Logger, state.PlanOptions{SelectedReleases: toUpdate, SkipNeeds: true, IncludeTransitiveNeeds: c.IncludeTransitiveNeeds()}, a.WrapWithoutSelector(func(subst *state.HelmState, helm helmexec.Interface) []error {
var rs []state.ReleaseSpec
for _, r := range subst.Releases {
release := r
if _, ok := releasesToDelete[state.ReleaseToID(&release)]; !ok {
rs = append(rs, release)
}
}
subst.Releases = rs
opts := &state.SyncOpts{
Set: c.Set(),
SkipCRDs: c.SkipCRDs(),
Wait: c.Wait(),
WaitRetries: c.WaitRetries(),
WaitForJobs: c.WaitForJobs(),
ReuseValues: c.ReuseValues(),
ResetValues: c.ResetValues(),
PostRenderer: c.PostRenderer(),
PostRendererArgs: c.PostRendererArgs(),
SyncArgs: c.SyncArgs(),
HideNotes: c.HideNotes(),
TakeOwnership: c.TakeOwnership(),
SkipSchemaValidation: c.SkipSchemaValidation(),
SyncReleaseLabels: c.SyncReleaseLabels(),
}
return subst.SyncReleases(&affectedReleases, helm, c.Values(), c.Concurrency(), opts)
}))
if len(syncErrs) > 0 {
errs = append(errs, syncErrs...)
}
}
}
affectedReleases.DisplayAffectedReleases(c.Logger())
return true, errs
}
func (a *App) template(r *Run, c TemplateConfigProvider) (bool, []error) {
return a.withNeeds(r, c, false, func(st *state.HelmState) []error {
helm := r.helm
args := GetArgs(c.Args(), st)
// Reset the extra args if already set, not to break `helm fetch` by adding the args intended for `lint`
helm.SetExtraArgs()
if len(args) > 0 {
helm.SetExtraArgs(args...)
}
opts := &state.TemplateOpts{
Set: c.Set(),
IncludeCRDs: c.IncludeCRDs(),
NoHooks: c.NoHooks(),
OutputDirTemplate: c.OutputDirTemplate(),
SkipCleanup: c.SkipCleanup(),
SkipTests: c.SkipTests(),
PostRenderer: c.PostRenderer(),
PostRendererArgs: c.PostRendererArgs(),
KubeVersion: c.KubeVersion(),
ShowOnly: c.ShowOnly(),
SkipSchemaValidation: c.SkipSchemaValidation(),
}
return st.TemplateReleases(helm, c.OutputDir(), c.Values(), args, c.Concurrency(), c.Validate(), opts)
})
}
func (a *App) withNeeds(r *Run, c DAGConfig, includeDisabled bool, f func(*state.HelmState) []error) (bool, []error) {
st := r.state
selectedReleases, deduplicated, err := a.getSelectedReleases(r, false)
if err != nil {
return false, []error{err}
}
if len(selectedReleases) == 0 {
return false, nil
}
// This is required when you're trying to deduplicate releases by the selector.
// Without this, `PlanReleases` conflates duplicates and return both in `batches`,
// even if we provided `SelectedReleases: selectedReleases`.
// See https://github.com/roboll/helmfile/issues/1818 for more context.
st.Releases = deduplicated
includeNeeds := c.IncludeNeeds()
if c.IncludeTransitiveNeeds() {
includeNeeds = true
}
batches, err := st.PlanReleases(state.PlanOptions{
Reverse: false,
SelectedReleases: selectedReleases,
IncludeNeeds: includeNeeds,
IncludeTransitiveNeeds: c.IncludeTransitiveNeeds(),
SkipNeeds: c.SkipNeeds(),
})
if err != nil {
return false, []error{err}
}
var selectedReleasesWithNeeds []state.ReleaseSpec
for _, rs := range batches {
for _, r := range rs {
selectedReleasesWithNeeds = append(selectedReleasesWithNeeds, r.ReleaseSpec)
}
}
var toRender []state.ReleaseSpec
releasesToUninstall := map[string]state.ReleaseSpec{}
for _, r := range selectedReleasesWithNeeds {
release := r
id := state.ReleaseToID(&release)
if !release.Desired() {
releasesToUninstall[id] = release
} else {
toRender = append(toRender, release)
}
}
var rels []state.ReleaseSpec
if len(toRender) > 0 {
// toRender already contains the direct and transitive needs depending on the DAG options.
// That's why we don't pass in `IncludeNeeds: c.IncludeNeeds(), IncludeTransitiveNeeds: c.IncludeTransitiveNeeds()` here.
// Otherwise, in case include-needs=true, it will include the needs of needs, which results in unexpectedly introducing transitive needs,
// even if include-transitive-needs=true is unspecified.
if _, errs := withDAG(st, r.helm, a.Logger, state.PlanOptions{SelectedReleases: toRender, Reverse: false, SkipNeeds: c.SkipNeeds(), IncludeNeeds: includeNeeds}, a.WrapWithoutSelector(func(subst *state.HelmState, helm helmexec.Interface) []error {
rels = append(rels, subst.Releases...)
return nil
})); len(errs) > 0 {
return false, errs
}
}
if c.EnforceNeedsAreInstalled() {
for _, r := range toRender {
visited := make(map[string]bool)
if depErr := st.HasTransitiveDependencyWithInstalledFalse(r, visited); depErr != nil {
return false, []error{fmt.Errorf("Release %s has a transitive dependency %s marked as installed=false", depErr.Release.Name, depErr.Dependency.Name)}
}
}
}
if includeDisabled {
for _, d := range releasesToUninstall {
rels = append(rels, d)
}
}
// Traverse DAG of all the releases so that we don't suffer from false-positive missing dependencies
// and we don't fail on dependenciese on disabled releases.
// In diff, we need to diff on disabled releases to show to-be-uninstalled releases.
// In lint and template, we'd need to run respective helm commands only on enabled releases,
// without failing on disabled releases.
st.Releases = rels
errs := f(st)
return true, errs
}
func (a *App) test(r *Run, c TestConfigProvider) []error {
cleanup := c.Cleanup()
timeout := c.Timeout()
concurrency := c.Concurrency()
st := r.state
toTest, _, err := a.getSelectedReleases(r, false)
if err != nil {
return []error{err}
}
if len(toTest) == 0 {
return nil
}
// Do test only on selected releases, because that's what the user intended
// with conditions and selectors
st.Releases = toTest
r.helm.SetExtraArgs(GetArgs(c.Args(), r.state)...)
return st.TestReleases(r.helm, cleanup, timeout, concurrency, state.Logs(c.Logs()))
}
func (a *App) writeValues(r *Run, c WriteValuesConfigProvider) (bool, []error) {
st := r.state
helm := r.helm
toRender, _, err := a.getSelectedReleases(r, false)
if err != nil {
return false, []error{err}
}
if len(toRender) == 0 {
return false, nil
}
// Do build deps and prepare only on selected releases so that we won't waste time
// on running various helm commands on unnecessary releases
st.Releases = toRender
releasesToWrite := map[string]state.ReleaseSpec{}
for _, r := range toRender {
release := r
id := state.ReleaseToID(&release)
if release.Desired() {
releasesToWrite[id] = release
}
}
var errs []error
// Note: We don't calculate the DAG of releases here unlike other helmfile operations,
// because there's no need to do so for just writing values.
// See the first bullet in https://github.com/roboll/helmfile/issues/1460#issuecomment-691863465
if len(releasesToWrite) > 0 {
var rs []state.ReleaseSpec
for _, r := range releasesToWrite {
rs = append(rs, r)
}
st.Releases = rs
opts := &state.WriteValuesOpts{
Set: c.Set(),
OutputFileTemplate: c.OutputFileTemplate(),
SkipCleanup: c.SkipCleanup(),
}
errs = st.WriteReleasesValues(helm, c.Values(), opts)
}
return true, errs
}
// Error is a wrapper around an error that adds context to the error.
type Error struct {
msg string
Errors []error
code *int
}
func (e *Error) Error() string {
var cause string
if e.Errors == nil {
return e.msg
}
if len(e.Errors) == 1 {
if e.Errors[0] == nil {
panic(fmt.Sprintf("[bug] assertion error: unexpected state: e.Errors: %v", e.Errors))
}
cause = e.Errors[0].Error()
} else {
msgs := []string{}
for i, err := range e.Errors {
if err == nil {
continue
}
msgs = append(msgs, fmt.Sprintf("err %d: %v", i, err.Error()))
}
cause = fmt.Sprintf("%d errors:\n%s", len(e.Errors), strings.Join(msgs, "\n"))
}
msg := ""
if e.msg != "" {
msg = fmt.Sprintf("%s: %s", e.msg, cause)
} else {
msg = cause
}
return msg
}
func (e *Error) Code() int {
if e.code != nil {
return *e.code
}
allDiff := false
anyNonZero := false
for _, err := range e.Errors {
switch ee := err.(type) {
case *state.ReleaseError:
if anyNonZero {
allDiff = allDiff && ee.Code == 2
} else {
allDiff = ee.Code == 2
}
case *Error:
if anyNonZero {
allDiff = allDiff && ee.Code() == 2
} else {
allDiff = ee.Code() == 2
}
}
anyNonZero = true
}
if anyNonZero {
if allDiff {
return 2
}
return 1
}
panic(fmt.Sprintf("[bug] assertion error: unexpected state: unable to handle errors: %v", e.Errors))
}
func appError(msg string, err error) *Error {
return &Error{msg: msg, Errors: []error{err}}
}
func (c context) clean(errs []error) error {
if errs == nil {
errs = []error{}
}
if !c.retainValues {
cleanErrs := c.st.Clean()
if cleanErrs != nil {
errs = append(errs, cleanErrs...)
}
}
return c.wrapErrs(errs...)
}
type context struct {
app *App
st *state.HelmState
retainValues bool
}
func (c context) wrapErrs(errs ...error) error {
if len(errs) > 0 {
for _, err := range errs {
switch e := err.(type) {
case *state.ReleaseError:
c.app.Logger.Debugf("err: release \"%s\" in \"%s\" failed: %v", e.Name, c.st.FilePath, e)
default:
c.app.Logger.Debugf("err: %v", e)
}
}
return &Error{Errors: errs}
}
return nil
}
func (a *App) ShowCacheDir(c CacheConfigProvider) error {
fmt.Printf("Cache directory: %s\n", remote.CacheDir())
if !a.fs.DirectoryExistsAt(remote.CacheDir()) {
return nil
}
dirs, err := a.fs.ReadDir(remote.CacheDir())
if err != nil {
return err
}
for _, e := range dirs {
fmt.Printf("- %s\n", e.Name())
}
return nil
}
func (a *App) CleanCacheDir(c CacheConfigProvider) error {
if !a.fs.DirectoryExistsAt(remote.CacheDir()) {
return nil
}
fmt.Printf("Cleaning up cache directory: %s\n", remote.CacheDir())
dirs, err := os.ReadDir(remote.CacheDir())
if err != nil {
return err
}
for _, e := range dirs {
fmt.Printf("- %s\n", e.Name())
err := os.RemoveAll(filepath.Join(remote.CacheDir(), e.Name()))
if err != nil {
return err
}
}
return nil
}
func GetArgs(args string, state *state.HelmState) []string {
baseArgs := []string{}
stateArgs := []string{}
if len(args) > 0 {
baseArgs = argparser.CollectArgs(args)
}
if len(state.HelmDefaults.Args) > 0 {
stateArgs = argparser.CollectArgs(strings.Join(state.HelmDefaults.Args, " "))
}
state.HelmDefaults.Args = append(baseArgs, stateArgs...)
return state.HelmDefaults.Args
}