This should fix#435 for Helmfile v0.x releases since the next v0.150.0.
We introduce a new envvar to opt-in to the new YAML library, so that you can give it a shot before upgrading your Helmfile to v1. The same envvar can be used to opt-out of the new YAML library after you upgrade to Helmfile v1, giving you a more flexible migration story.
Signed-off-by: Yusuke Kuoka <ykuoka@gmail.com>
This is a successor to #596. We need a smooth migration path from `gopkg.in/yaml.v2`, and this pull request moves it forward with `goccy/go-yaml` instead of `gopkg.in/yaml.v3`. Merging this unblocks users stuck in Helmfile v0.146.x or earlier due to #435, so that they can upgrade to 0.147.x or greater without updating their helmfile configs.
We previously tried to upgrade to `yaml.v3` (https://github.com/helmfile/helmfile/issues/394) in Helmfile v0.x, presuming it won't break anything. Apparently, it broke use-cases where you want to layer release's `values` field over three or more release templates and releases (#435).
We then tried to bring back `yaml.v2` for Helmfile v0.x and keep `yaml.v3` for the upcoming Helmfile v1. However, it failed due to incompatibility in the Unmarshaller interface between `yaml.v2` and `yaml.v3` (https://github.com/helmfile/helmfile/pull/596).
`goccy/go-yaml` is, from my observation, a well-maintained alternative to `yaml.v2`. One of its premises is that it enables us to swap the implementation from `gopkg.in/yaml.v2` to `goccy/go-yaml` just by replacing the import directive. It seems to use the same `Unmarshaller` interface as yaml.v2 too.
Once this PR gets merged, I'd like to follow-up with adding a new build-time variable and an envvar to set the proper default for the yaml parser Helmfile uses and the ability to switch the parser at runtime. All in all, the next Helmfile release, v0.150.0 will get reverted to use `gopkg.in/yaml.v2` by default which resolves#435.
New users who started using Helmfile since any of v0.148.0, v0.148.1, and v0.149.0 might be already relying on the new behavior, They might need to specify a new envvar to enable `goccy/go-yaml`.
Signed-off-by: yxxhero <aiopsclub@163.com>
Signed-off-by: yxxhero <aiopsclub@163.com>
Co-authored-by: yxxhero <aiopsclub@163.com>
This commit replaces `os.Setenv` with `t.Setenv` in tests. The
environment variable is automatically restored to its original value
when the test and all its subtests complete.
Reference: https://pkg.go.dev/testing#T.Setenv
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
* introduce DISABLE_INSECURE_FEATURES to disable insecure executions
Signed-off-by: Quan TRAN <account@itscaro.me>
* disable remote sources when DISABLE_INSECURE_FEATURES is set to "true"
Signed-off-by: Quan TRAN <account@itscaro.me>
* refactor envvar package
Signed-off-by: Quan TRAN <account@itscaro.me>
* (test) fix test fixtures
Signed-off-by: Quan TRAN <account@itscaro.me>
* use absolute path to avoid unit test failure
Signed-off-by: Quan TRAN <account@itscaro.me>
* Fix conflicts
Co-authored-by: Yusuke Kuoka <ykuoka@gmail.com>
Adds a new templating function called `readDir`. With `readDir` users can read the contents of a specified directory and get a list of all contained files. This is useful when reading a bunch of files from a directory. The following example shows a snippet of a values file for configuring a Logstash deployment. Using only `readFile`, a user must specify each file by hand and adjust this list as the number of files to be read grows.
```yaml
logstash:
configs:
logstash.yml: |
{{- tpl (readFile "config/logstash.yml.gotmpl") . | nindent 6 }}
jvm.options: |
{{- readFile "config/jvm.options" | nindent 6 }}
pipelines.yml: |
{{- readFile "config/pipelines.yml" | nindent 6 }}
pipelines:
beats-log4j.conf: |
{{- readFile "config/pipelines/beats-log4j.conf" | nindent 6 }}
nginx-access.conf: |
{{- readFile "config/pipelines/nginx-access.conf" | nindent 6 }}
nginx-error.conf: |
{{- readFile "config/pipelines/nginx-error.conf" | nindent 6 }}
syslog-logs.conf: |
{{- readFile "config/pipelines/syslog-logs.conf" | nindent 6 }}
tcp-logs.conf: |
{{- readFile "config/pipelines/tcp-logs.conf" | nindent 6 }}
udp-debug.conf: |
{{- readFile "config/pipelines/udp-debug.conf" | nindent 6 }}
udp-logs.conf: |
{{- readFile "config/pipelines/udp-logs.conf" | nindent 6 }}
certificates:
ca.crt: |
{{- readFile "config/certificates/ca.crt" | nindent 6 }}
logstash.crt: |
{{- readFile "config/certificates/logstash.crt" | nindent 6 }}
logstash.key: |
{{- readFile "config/certificates/logstash.key" | nindent 6 }}
```
With `readDir` the above snippet can be rewritten as follows:
```yaml
logstash:
configs:
{{- range readDir "config" }}
{{ base . }}: |
{{- if hasSuffix "gotmpl" . }}
{{- tpl (readFile .) $ | nindent 6 }}
{{- else }}
{{- readFile . | nindent 6 }}
{{- end }}
{{- end }}
pipelines:
{{- range readDir "config/pipelines" }}
{{ base . }}: |
{{- readFile . | nindent 6 }}
{{- end }}
certificates:
{{- range readDir "config/certificates" }}
{{ base . }}: |
{{- readFile . | nindent 6 }}
{{- end }}
```
Related to #494
This feature is mostly a built-in alternative to the `incubator/raw` chart without external dependency and has
access to helmfile's own template functions and template data.
The expected use-case of this feature is to add arbitrary K8s resources to your deployment.
Unlike the original issue raised in #494 this doesn't enable you to add arbitary resources to a release. That's another story. But this would be a good foundation for that, too.
* Bump sprig to v3.1.0
test for mergeOverwrite
* Let mergo not (accidentally) try to merge unexported fields
This is also a good chance separate `HelmState` with the config loaded from YAML, which I had been wanting to do for a long time.
Co-authored-by: Johannes Alkjær <johannes.alkjaer@wunderman.com>
Co-authored-by: Yusuke Kuoka <ykuoka@gmail.com>
* feat(tmpl): added fetchSecretValue template function
This adds a tmpl `fetchSecretValue` and `expandSecretRefs` function by:
- Adding:
- `expandSecretRefs` function in tmpl package that uses vals
package to fetch secrets
- `fetchSecretValue` function in tmpl package like below but for
single string value
- gomock for tests purpose
- Changing:
- move init of vals package to function (so the same instance can be used for template values and rendering the whole template)
* doc(secret): added doc how to use new tmpl methods
Added example usage of `fetchSecretValue` and `expandSecretRefs`
Enhances Helmfile to print more helpful message on error while calling `exec` template function.
Helmfile has been printing error messages like the below:
```
in ./helmfile.yaml: error during helmfile.yaml.part.0 parsing: template: stringTemplate:5:8: executing "stringTemplate" at <exec "./exectest.sha" (list)>: error calling exec: exit status 1
```
Adding captured stdout and stderr, with some indentation to make it readable, it now produces the following message on missing executable:
```
$ make build && ./helmfile build
go build
in ./helmfile.yaml: error during helmfile.yaml.part.0 parsing: template: stringTemplate:5:8: executing "stringTemplate" at <exec "./exectest.sha" (list)>: error calling exec: fork/exec ./exectest.sha: no such file or directory
COMMAND:
./exectest.sha
ERROR:
fork/exec ./exectest.sha: no such file or directory
```
On non-zero exit status without output:
```
$ make build && ./helmfile build
go build
in ./helmfile.yaml: error during helmfile.yaml.part.0 parsing: template: stringTemplate:5:8: executing "stringTemplate" at <exec "./exectest.sh" (list)>: error calling exec: exit status 1
COMMAND:
./exectest.sh
ERROR:
exit status 1
```
On non-zero exit status with output:
```
$ make build && ./helmfile build
go build
in ./helmfile.yaml: error during helmfile.yaml.part.0 parsing: template: stringTemplate:5:8: executing "stringTemplate" at <exec "./exectest.sh" (list)>: error calling exec: exit status 2
COMMAND:
./exectest.sh
ERROR:
exit status 2
COMBINED OUTPUT:
out1
err1
```
Resolves#1158
Probably since #647 helmfile has been unable to merge nested maps in environment values if they were loaded from files. This fixes it.
The relevant test is also enhanced so that no further regression like this happens.
Fixes#677
Seems like we are affected by https://github.com/golang/go/issues/24963. That is, even though we internally use the template option `missingkey=zero`, in some cases it still prints `<no value>` instead of zero values, which has been confusing the state yaml parsing.
This fixes the issue by naively replacing all the remaining occurrences of `<no value>` in the rendered text, while printing debug logs to ease debugging in the future when there is unexpected side-effects introduced by this native method.
Fixes#553
Dmitry Chepurovskiy
yxxhero
Yusuke Kuoka
yxxhero
Arkaitz Jimenez
vasicvuk
Eng Zer Jun
Quan TRAN
austin ce
Anton Bretting
Nils
Yoann Ciabaud
Johannes Alkjær
Marcin Kaciuba
RaymondKYLiu
a-hat