* Bump sprig to v3.1.0
test for mergeOverwrite
* Let mergo not (accidentally) try to merge unexported fields
This is also a good chance separate `HelmState` with the config loaded from YAML, which I had been wanting to do for a long time.
Co-authored-by: Johannes Alkjær <johannes.alkjaer@wunderman.com>
Co-authored-by: Yusuke Kuoka <ykuoka@gmail.com>
This adds `comonLabels` option to helmfile by:
- Adding `CommonLabels` to HelmState
- Changing `markExcludedReleases` and `ListReleases` functions to merge common labels into release labels
Resolves#1266
This reverts a part of #1383 so that repository updates are done in the pre-0.125.0 way, which tries to update any repositories only once regardless of they are referenced by selected releases or not.
Ref #1404
In #1172, we accidentally changed the meaning of prepare hook that is intended to be called BEFORE the pathExists check. It broke the scenario where one used a prepare hook for generating the local chart dynamically. This fixes Helmfile not to fetch local chart generated by prepare hook.
In addition to that, this patch results in the following fixes:
- Fix an issue that `helmfile template` without `--skip-deps` fails while trying to run `helm dep build` on `helm fetch`ed chart, when the remote chart has outdated dependencies in the Chart.lock file. It should be up to the chart maintainer to update Chart.lock and the user should not be blocked due to that. So, after this patch `helm dep build` is run only on the local chart, not on fetched remote chart.
- Skip fetching chart on `helmfile template` when using Helm v3. `helm template` in helm v3 does support rendering remote charts so we do not need to fetch beforehand.
Fixes#1328
May relate to #1341
Changes:
- Prevent Helmfile from unnecessarily running `helm repo add` and `helm repo up` against repositories for unused repositories(repositories of releases filtered out by selector)
- Fixes#1330
This, in combination with #1172, allows you to use `go-getter`-supported URL for K8s manifests on `chart`, so that Helmfile automatically fetches it and then turning it into a temporary local chart, which is then installed by Helmfile as similar as standard Helm charts.
An example usecase of this is to install cert-manager CRDs which is distributed separately from the chart:
```
releases:
- name: cert-manager-crds
chart: git::http://github.com/jetstack/cert-manager.git@deploy/crds?ref=v0.15.2
```
I'm adding this based on discussion with @lukasmrtvy. He was trying to install cert-manager and prometheus-opreator with Helmfile, and this combined with #1373 should do the job. Thanks for the input!
This is useful for e.g. removing state file names and their hash values out of output dirs so that it can be used easily in a gitops setup. For example, `--output-dir-template mybasedir/{{.Release.Name}}` produces `mybasedir/RELEASE/CHART/templates/*.yaml` for each release in your helmfile.yaml.
Changes:
* Add global hooks
* Add top level hooks field to yaml spec
* Add functions for global prepare and cleanup events
* Call global prepare and cleanup events in withPreparedCharts function
* Update README
* Add helmfileCommand variable to withPreparedCharts
Pass the information on what helmfileCommand has been run down from the
top level functions through withReposAndPreparedCharts and withPreparedCharts.
This is the GA version of the helm-x integration #673 developed last year.
You get all the following benefits without an extra helm plugin:
- Ability to add ad-hoc chart dependencies/aliases, without forking the chart (Fixes#876 )
- Ability to patch resulting K8s resources before installing the helm chart
- Ability to install a kustomization as a chart (Requires `kustomize` binary to be available in `$PATH`
- Ability to install a directory of K8s manifests as a chart
- etc.
Fixes https://github.com/roboll/helmfile/issues/1142
desired_state_file_loader.go
- Will now normalize the content before splitting it to parts
context:
Me & and a fellow dev have tried to figure out why helmfile didn't fill in certain values on his machine;
turns out, he'd mistakenly checked out our project w/ CRLF line endings, which had caused part splitting to not work (as it's hard coded to look for '\n').
The following was acted on as a single part, causing values from the bases not to be available in the next yaml part:
```
bases:\r\n
- base.yaml\r\n
---\r\n
releases:
- name: external-secrets-crd
... some templated yaml ...
```
I've thought about regex-ing it out instead of replace-all, but benchmarks had shown that a plain replace is faster.
I've also considered splitting by "\n---" instead of "\n---", but that would break if the dashes were to continue with some other text.
Resolves#1232
CHANGES:
* Disable repo update when installed is false
When install is false, we do not need to update the repositories and get
the chart.
Signed-off-by: Guillaume Perrin <guillaume28.perrin@gmail.com>
- createNamespace is a new attribute that can be added to helmDefaults
or an individual release to enforce the creation of a release namespace
during sync if the namespace does not exist. This leverages helm's
(3.2+) --create-namespace flag for the install/upgrade command. If
running helm < 3.2, the createNamespace attribute has no effect.
Resolves#891Resolves#1140
* feat(tmpl): added fetchSecretValue template function
This adds a tmpl `fetchSecretValue` and `expandSecretRefs` function by:
- Adding:
- `expandSecretRefs` function in tmpl package that uses vals
package to fetch secrets
- `fetchSecretValue` function in tmpl package like below but for
single string value
- gomock for tests purpose
- Changing:
- move init of vals package to function (so the same instance can be used for template values and rendering the whole template)
* doc(secret): added doc how to use new tmpl methods
Added example usage of `fetchSecretValue` and `expandSecretRefs`
* New output flag for list command
Support output as json
Add new formatters file to handle extrac formatting to its own concern
New config interface to support list command specification
* Fix usage message
* Add error handling for formatters
This commit proposes a potential solution for
https://github.com/roboll/helmfile/issues/1201
The gist is that, if prestate rendering fails, for any reason,
we do not populate the .Values in the second pass renderer.
I think that what have been expected in this case is to populate the
.Values irregardless.
pkg/app/two_pass_renderer.go
- Migrated to use finalEnv.GetMergedValues()
pkg/environment/environment.go
- Introduced GetMergedValues, which merges the environment's defaults
and current values, and then casts the keys to string;
This was previously defined in HelmState.Values() - however, as this
method is only concerned with the environment, I think it's more
appropriate for it to sit here.
pkg/state/state_exec_tmpl.go
- Extracted out HelmState.Values() to environment.go, see above
`--retain-values-files` prevents temporary values files that were passed to Helm commands run by Helmfile for debugging purpose.
With that, you can manually rerun helm commands that were logged when `--log-level=debug` is enabled.
Resolves ##1117
* Add option to suppress diff on apply
Add --supress-diff option on apply. Usable for fresh installs when a
lot of output is produces by diff.
Resolves#458
* fix tests for suppress-diff
This makes it possible to pass the API Capabilities to helmfile when executing a task that does not render against an actual cluster (diff, template, apply).
Resolves#1014
When helmfile is run with `--environment NAME` and there was a base hemlfile that misses `environments`, helmfile had been trying to load env values for NAME and failing.
A base helmfile is allowed to reference values from within itself, but that's optional. In other words, a base helmfile that misses the env is okay as long as it doesn't self-reference env values.
So, this change allows missing env and env values while loading base helmfile. After loading, a base helmfile can fail due to referencing missing env values, but that's okay.
Fixes#1008
Those are not actually random but would have looked like so. We use an external go pkg `variantdev/vals` to expand urls like `ref+vault://foo/bar` contained in release values into their respective secret values.
There was a bug in `vals` that it tries to expand unintended types of strings which resulted in confusing errors like reported in #973.
`vals` fixed the issue in ba4c7a2987. This commit upgrades `vals` to accomodate that.
Fixes#973
Runs `helm version` in helmexec.New, and exposes a method on Interface to allow other packages to use the detected version. Preserves compatibility with previous HELMFILE_HELM3 mechanism.
Resolves#923
* Fix regression since 0.90.0 that Helmfile becomes too slow when there are many releases
Fixes#959
* Ensure that the up-to-date helm-diff is installed and used in integration tests
The problem was that `--namespace NS` had been not taken into account while deleting releases, that resulted in releases that should be deleted are not deleted.
The recent addition of the DAG support(`needs`) and the fixes on it broke the delete-on-sync functionality. And there were two more bugs. One is that it was not correctly running `helm delete` when needed and the another is that it was failing when `--selector` is specified and the releases to delete by sync found, but nothing actually got deleted. This fixes all of them.
Fixes#941
This ports the fix for `helfmile apply` to `sync`, so that specifying `--selector` doesn't break `helmfile sync`.
Also make `helmfile template` DAG-aware, so that the manifests are rendered in the order of dependency.
Ref #919
This overrides the default helm command, if provided, as soon as possible.
This way it is already used in `visitStates`.
I ran into an issue using `HELM3` and `--helm-binary` together with helm-secrets.
I previously used tillerless, which i could now remove. This however caused `DecryptSecret` to fail, as it would still use the helm2 binary; because it runs before the first `helm.SetHelmBinary` call.
While helm-secrets is not fully helm3 compatible yet, its uses within helmfile are, and i was able to just install it as a helm3 plugin.
* fix: Fix `needs` to work for upgrades and when selectors are provided
Fixes#919
* Add test framework for `helmfile apply`
* Various enhancements and fixes to the DAG support
- Make the order of upgrades/deletes more deterministic for testability
- Fix the test framework so that we can validate log outputs and errors
- Add more test cases for `helmfile apply`, along with bug fixes.
- Make sure it fails with an intuitive error when you have non-existent releases referenced from witin "needs"
Set `HELMFILE_HELM3=1` and run `helmfile` like `HELMFILE_HELM3=1 helmfile ...`.
When `HELMFILE_HELM3` is set, `test`, `template`, `delete`, `destroy` behave differently so that it works with Helm 3.
Note that `helmfile diff` doesn't work as `helm-diff` called under the hood doesn't support Helm v3 yet.
Ref #668
Closes#444 and #782
This is the final PR to fully cache and parallelize helm secret decryption. It threads the shared helmexec.Interface into the StateCreator and HelmState structs to be used during environment secret decryption. This should effectively cache secrets for the duration of a helmfile run, regardless of where they are first decrypted.
Probably since #647 helmfile has been unable to merge nested maps in environment values if they were loaded from files. This fixes it.
The relevant test is also enhanced so that no further regression like this happens.
Fixes#677
Extends the remote-helmfile feature to also work when loading the first state file.
This should be useful for people who wants to give helmfile a try without ever opening `$EDITOR`.
* fix: persist original file path when using bases
Prior to this change, the resulting lock file was called `<bases[0]>.lock`,
instead of `<filename>.lock`.
This change ensures the final, merged state has the correct `.FilePath`.
* test: Assert proper FilePath in layered HelmState
This change enhances helmfile to accept terraform-module-like URLs in nested state files a.k.a sub-helmfiles.
```yaml
helmfiles:
- # Terraform-module-like URL for importing a remote directory and use a file in it as a nested-state file
# The nested-state file is locally checked-out along with the remote directory containing it.
# Therefore all the local paths in the file are resolved relative to the file
path: git::https://github.com/cloudposse/helmfiles.git@releases/kiam.yaml?ref=0.40.0
```
The URL isn't equivalent to terraform module sources. The difference is that we use `@` to distinguish between (1) the path to the repository and directory containing the state file and (2) the path to the state file being loaded. This distinction provides us enough fleibiity to instruct helmfile to check-out necessary and sufficient directory to make the state file works.
Under the hood, it uses [hashicorp/go-getter](https://github.com/hashicorp/go-getter), that is used for [terraform module sources](https://www.terraform.io/docs/modules/sources.html) as well.
Only the git provider without authentication like git-credentials helper is tested. But theoretically any go-getter providers should work. Please feel free to test the provider of your choice and contribute documentation or instruction to use it :)
Resolves#347
This adds `values` to state files as proposed in #640.
```yaml
values:
- key1: val1
- defaults.yaml
environments:
default:
- values:
- environments/default.yaml
production:
- values:
- environments/production.yaml
```
`{{ .Valuese.key1 }}` evaluates to `val1` if and only if it is not overrode via the production or the default env, or command-line args.
Resolves#640
Seems like we are affected by https://github.com/golang/go/issues/24963. That is, even though we internally use the template option `missingkey=zero`, in some cases it still prints `<no value>` instead of zero values, which has been confusing the state yaml parsing.
This fixes the issue by naively replacing all the remaining occurrences of `<no value>` in the rendered text, while printing debug logs to ease debugging in the future when there is unexpected side-effects introduced by this native method.
Fixes#553
The addition of `--set k1=v1,k2=v2` and `--values file1 --values file2` was originally planned in #361.
But it turned out we already had `--values` for existing helmfile commands like `sync`. Duplicated flags doesn't work, obviously.
So this actually add `--state-values-set k1=v1,k2=v2` and `--set-values-file file1 --set-values-file file2`.
They are called "state" values according to the discussion we had at #640Resolves#361
* feat: helmfile as a go library
This removes almost all the dependencies from the helmfile core logic to urfave/cli. `main.go` is now a thin wrapper around the core logic implemented in `pkg/app`.
We added envvals overrides in the state file via #622 two days ago:
```
helmfiles:
- name: sub.helmfile.yaml
environment:
values:
- mykey: myvalue
```
This change removes the `environment` level in the above cofig, so that it looks like:
```
helmfiles:
- name: sub.helmfile.yaml
values:
- mykey: myvalue
``
This is an inevitable breaking change towards #361. But I wanted to break it earlier so that less folks are affected.`
Ref https://github.com/roboll/helmfile/issues/361#issuecomment-497530819
* feat: specify env values from the parent to the nested state
Adds the `helmfiles[].environment.values` that accepts a mix of file pathes and inline dictes:
```yaml
helmfiles:
- path: path/to/nested/helmfile.yaml
environment:
values:
- key1: val1
- values.yaml
```
The values files are loaded in the context of the parent state file. For example, in case the above state file is located at `/path/to/helmfile.yaml`,
`values.yaml` is located at `/path/to/values.yaml` instead of `/path/to/nested/values.yaml`.
Resolves#523
* fix: multiple "bases" declarations yields duplicate releases
Fixes#615
* fix regression in double-rendering with env value overrides
The latest commit broke any state files like the below to NOT pass env value overrides at all:
```
helmfiles:
- path: nested/state.yaml
environment:
values:
- overrides.yaml
```
This fixes the issue.
```yaml
environments:
default:
missingFileHandler: Warn
values:
- path/to/values.yaml
secrets:
- path/to/secrets.yaml
```
`missingFileHandler` set to `Warn`, `Info`, or `Debug` results in helmfile NOT stop when `path/to/values.yaml` or `path/to/secrets.yaml` is missing.
Resolves#548
While implementing the above feature, I also found a bug that has been causing #559. This also fixes that.
To verify it is actually fixed, create an example helmfile.yaml that looks like the below, and run `helmfile diff`:
```
$ cat helmfile.yaml
environments:
default:
secrets:
- env-secrets.yaml
releases:
- name: myapp
chart: nginx
namespace: default
secrets: [secrets.yaml] # Notice this file does not exist
values:
- ingress:
enabled: true
$ helmfile diff
could not deduce `environment:` block, configuring only .Environment.Name. error: failed to read helmfile.yaml.part.0: environment values file matching "env-secrets.yaml" does not exist
in ./helmfile.yaml: failed to read helmfile.yaml: environment values file matching "env-secrets.yaml" does not exist
```
Fixes#559
This enhances helmfile's internal environment values files loader to expand glob patterns (#606)
Fixes the existing bug that helmfile was unable to load environment values file from absolute path (#549)
Resolves#606Fixes#549
* Various U/X improvements for `helmfile apply`
This improves the U/X of `helmfile apply`, by allowing you to selectively apply sub-helmfiles.
When you have two or more sub-helmfiles processed, typing `n` to cancel the first doesn't automatically stop the whole helmfile execution.
Instead, it proceeds by diffing the next sub-helmfile, and asks you to apply it, which should be what the user would expect.
To support this workflow, I have suppressed useless exec logs, correct exit status when diff exists in sub-helmfiles but not in the parent helmfile, and made the final error message emitted by helmfile better.
More concretely, this moves more output from `helm` to STDERR and the `debug` log-level.
The overall output from `helmfile` should be a bit more cleaner especially for `apply`, `sync`, `diff` and perhaps other `helmfile` sub-commands, too.
For example, when one of release failed, `helmfile`'s final error message now includes the error message from the failed `helm` execution, like seen in the last line:
```
List of updated releases :
RELEASE CHART VERSION
envoy stable/envoy 1.5.0
List of releases in error :
RELEASE
envoy2
in ./helmfile.yaml: in .helmfiles[0]: in /Users/c-ykuoka/helmfile/helmfile.1.yaml: failed processing release envoy2: helm exited with status 1:
Error: UPGRADE FAILED: "envoy2" has no deployed releases
```
This way you can better understand what caused helmfile to finally fail.
`helmfile` has been streaminig a lot of stdout and stderr contents from the `helm` commands regardless of the helmfile's log-level. It has been suppressed by default and moved to the `debug` log-level.
You will see that it helps you focus on what was the cause of a failure.
While working on the above, I found an another bug that made `--detailed-exitcode` useless in some case.
That is, `helmfile diff --detailed-exitcode`, when any diff existed only in sub-helmfiles, has been returning an exit code of `1`. It should return `2` when any release had diff and no release had an error, regardless of the target is a sub-helmfile or a parent helmfile. Why? Because that's what `--detailed-exitcode` meant for!
After this PR gets merged, `helmfile diff --detailed-exitcode` propery return exit code `2` in such cases.
Fixes#543Resolves#540
Fixes#344 by allowing explicit selectors to be specified for composed helmfiles using the following structure
```yaml
helmfiles:
- path: helmfile.d/a*.yaml
selectors:
- name=prometheus
- name!=zipkin
- helmfile.d/b*.yaml
- path: helmfile.d/c*.yaml
selectors: {}
```
2 modes here :
* legacy mode when no the env var HELMFILE_EXPERIMENTAL is not set to true
* no selector : inherit from the command line.
* selector: is specified then it is used (an emty means no inheritance from command line and take everything).
* experimental when the env var HELMFILE_EXPERIMENTAL=true
* no selector : nothing is inherited from the command line so use all releases.
* selector: is specified then it is used (an emty means no inheritance from command line and take everything).
This basically allows to define both `releases` and `helmfiles` within a helmfile.yaml, so that you can start using sub-helmfiles easily, by extracting only reused releases.
Resolves#445
* Improve code organization
To make sure it is still readable after upcoming changes to helmfile
* feat: `helmfile deps` to update dependencies of all the local charts
Resolves#450
* feat: helmfile updates repos and build deps by default
But not update deps. Use `helmfile deps` to update deps, and provide `--skip-deps` to skip updating repos and builds deps in sync/diff/apply/template
Resolves#415
* Improve integration test coverage
Yusuke Kuoka
Johannes Alkjær
Maksym Lushpenko
Max Audron
Kevin J. Qiu
Ilya Vassyutovich
Danny Shemesh
Guillaume Perrin
Victor Noël
nejtr0n
Craig Dunford
Marcin Kaciuba
chenrui
Rene Hernandez
RaymondKYLiu
Eric Bailey
Emil
Christoph Hösler
a-hat
刘相轩
Jeroen Demeyer
Andrew Drake
Jan Möller
Kyrylo Lebediev
Theo Meneau
eddycharly
Shane Starcher
astorath
Travis Groth
OlivierB
Andrey Afoninsky
sgandon
Patrick Valsecchi