This should fix#435 for Helmfile v0.x releases since the next v0.150.0.
We introduce a new envvar to opt-in to the new YAML library, so that you can give it a shot before upgrading your Helmfile to v1. The same envvar can be used to opt-out of the new YAML library after you upgrade to Helmfile v1, giving you a more flexible migration story.
Signed-off-by: Yusuke Kuoka <ykuoka@gmail.com>
This is a successor to #596. We need a smooth migration path from `gopkg.in/yaml.v2`, and this pull request moves it forward with `goccy/go-yaml` instead of `gopkg.in/yaml.v3`. Merging this unblocks users stuck in Helmfile v0.146.x or earlier due to #435, so that they can upgrade to 0.147.x or greater without updating their helmfile configs.
We previously tried to upgrade to `yaml.v3` (https://github.com/helmfile/helmfile/issues/394) in Helmfile v0.x, presuming it won't break anything. Apparently, it broke use-cases where you want to layer release's `values` field over three or more release templates and releases (#435).
We then tried to bring back `yaml.v2` for Helmfile v0.x and keep `yaml.v3` for the upcoming Helmfile v1. However, it failed due to incompatibility in the Unmarshaller interface between `yaml.v2` and `yaml.v3` (https://github.com/helmfile/helmfile/pull/596).
`goccy/go-yaml` is, from my observation, a well-maintained alternative to `yaml.v2`. One of its premises is that it enables us to swap the implementation from `gopkg.in/yaml.v2` to `goccy/go-yaml` just by replacing the import directive. It seems to use the same `Unmarshaller` interface as yaml.v2 too.
Once this PR gets merged, I'd like to follow-up with adding a new build-time variable and an envvar to set the proper default for the yaml parser Helmfile uses and the ability to switch the parser at runtime. All in all, the next Helmfile release, v0.150.0 will get reverted to use `gopkg.in/yaml.v2` by default which resolves#435.
New users who started using Helmfile since any of v0.148.0, v0.148.1, and v0.149.0 might be already relying on the new behavior, They might need to specify a new envvar to enable `goccy/go-yaml`.
Signed-off-by: yxxhero <aiopsclub@163.com>
Signed-off-by: yxxhero <aiopsclub@163.com>
Co-authored-by: yxxhero <aiopsclub@163.com>
* introduce DISABLE_INSECURE_FEATURES to disable insecure executions
Signed-off-by: Quan TRAN <account@itscaro.me>
* disable remote sources when DISABLE_INSECURE_FEATURES is set to "true"
Signed-off-by: Quan TRAN <account@itscaro.me>
* refactor envvar package
Signed-off-by: Quan TRAN <account@itscaro.me>
* (test) fix test fixtures
Signed-off-by: Quan TRAN <account@itscaro.me>
* use absolute path to avoid unit test failure
Signed-off-by: Quan TRAN <account@itscaro.me>
* Fix conflicts
Co-authored-by: Yusuke Kuoka <ykuoka@gmail.com>
Adds a new templating function called `readDir`. With `readDir` users can read the contents of a specified directory and get a list of all contained files. This is useful when reading a bunch of files from a directory. The following example shows a snippet of a values file for configuring a Logstash deployment. Using only `readFile`, a user must specify each file by hand and adjust this list as the number of files to be read grows.
```yaml
logstash:
configs:
logstash.yml: |
{{- tpl (readFile "config/logstash.yml.gotmpl") . | nindent 6 }}
jvm.options: |
{{- readFile "config/jvm.options" | nindent 6 }}
pipelines.yml: |
{{- readFile "config/pipelines.yml" | nindent 6 }}
pipelines:
beats-log4j.conf: |
{{- readFile "config/pipelines/beats-log4j.conf" | nindent 6 }}
nginx-access.conf: |
{{- readFile "config/pipelines/nginx-access.conf" | nindent 6 }}
nginx-error.conf: |
{{- readFile "config/pipelines/nginx-error.conf" | nindent 6 }}
syslog-logs.conf: |
{{- readFile "config/pipelines/syslog-logs.conf" | nindent 6 }}
tcp-logs.conf: |
{{- readFile "config/pipelines/tcp-logs.conf" | nindent 6 }}
udp-debug.conf: |
{{- readFile "config/pipelines/udp-debug.conf" | nindent 6 }}
udp-logs.conf: |
{{- readFile "config/pipelines/udp-logs.conf" | nindent 6 }}
certificates:
ca.crt: |
{{- readFile "config/certificates/ca.crt" | nindent 6 }}
logstash.crt: |
{{- readFile "config/certificates/logstash.crt" | nindent 6 }}
logstash.key: |
{{- readFile "config/certificates/logstash.key" | nindent 6 }}
```
With `readDir` the above snippet can be rewritten as follows:
```yaml
logstash:
configs:
{{- range readDir "config" }}
{{ base . }}: |
{{- if hasSuffix "gotmpl" . }}
{{- tpl (readFile .) $ | nindent 6 }}
{{- else }}
{{- readFile . | nindent 6 }}
{{- end }}
{{- end }}
pipelines:
{{- range readDir "config/pipelines" }}
{{ base . }}: |
{{- readFile . | nindent 6 }}
{{- end }}
certificates:
{{- range readDir "config/certificates" }}
{{ base . }}: |
{{- readFile . | nindent 6 }}
{{- end }}
```
* feat(tmpl): added fetchSecretValue template function
This adds a tmpl `fetchSecretValue` and `expandSecretRefs` function by:
- Adding:
- `expandSecretRefs` function in tmpl package that uses vals
package to fetch secrets
- `fetchSecretValue` function in tmpl package like below but for
single string value
- gomock for tests purpose
- Changing:
- move init of vals package to function (so the same instance can be used for template values and rendering the whole template)
* doc(secret): added doc how to use new tmpl methods
Added example usage of `fetchSecretValue` and `expandSecretRefs`
Enhances Helmfile to print more helpful message on error while calling `exec` template function.
Helmfile has been printing error messages like the below:
```
in ./helmfile.yaml: error during helmfile.yaml.part.0 parsing: template: stringTemplate:5:8: executing "stringTemplate" at <exec "./exectest.sha" (list)>: error calling exec: exit status 1
```
Adding captured stdout and stderr, with some indentation to make it readable, it now produces the following message on missing executable:
```
$ make build && ./helmfile build
go build
in ./helmfile.yaml: error during helmfile.yaml.part.0 parsing: template: stringTemplate:5:8: executing "stringTemplate" at <exec "./exectest.sha" (list)>: error calling exec: fork/exec ./exectest.sha: no such file or directory
COMMAND:
./exectest.sha
ERROR:
fork/exec ./exectest.sha: no such file or directory
```
On non-zero exit status without output:
```
$ make build && ./helmfile build
go build
in ./helmfile.yaml: error during helmfile.yaml.part.0 parsing: template: stringTemplate:5:8: executing "stringTemplate" at <exec "./exectest.sh" (list)>: error calling exec: exit status 1
COMMAND:
./exectest.sh
ERROR:
exit status 1
```
On non-zero exit status with output:
```
$ make build && ./helmfile build
go build
in ./helmfile.yaml: error during helmfile.yaml.part.0 parsing: template: stringTemplate:5:8: executing "stringTemplate" at <exec "./exectest.sh" (list)>: error calling exec: exit status 2
COMMAND:
./exectest.sh
ERROR:
exit status 2
COMBINED OUTPUT:
out1
err1
```
Resolves#1158
Dmitry Chepurovskiy
Yusuke Kuoka
yxxhero
Arkaitz Jimenez
vasicvuk
Quan TRAN
austin ce
Anton Bretting
Nils
Yoann Ciabaud
Marcin Kaciuba
RaymondKYLiu
a-hat