* Use goccy/go-yaml fork to not break dynamic values on helmfile v1
I forked goccy/go-yaml to https://github.com/helmfile/go-yaml and added a commit 259d2ed450 to let it produce YAML data compatible with other legacy YAML parsers that do not support YAML 1.2.
Ref https://github.com/helmfile/helmfile/discussions/656
Signed-off-by: Yusuke Kuoka <ykuoka@gmail.com>
* Add test for goccy/go-yaml fork behavior
Signed-off-by: Yusuke Kuoka <ykuoka@gmail.com>
---------
Signed-off-by: Yusuke Kuoka <ykuoka@gmail.com>
* Disable double-rendering in V1
We are going to force users to separate environments and releases sections in every helmfile.yaml in the [Helmfile V1](https://github.com/helmfile/helmfile/blob/main/docs/proposals/towards-1.0.md). The goal of separation was to make the helmfile.yaml rendering result not dependent on the double-rendering and therefore the feature should be safe to be turned off in V1.
Signed-off-by: Yusuke Kuoka <ykuoka@gmail.com>
* Fix helmfile template rendering error log for v1
Signed-off-by: Yusuke Kuoka <ykuoka@gmail.com>
* Add test for helmfile template debug log differences between v0 and v1 modes
Signed-off-by: Yusuke Kuoka <ykuoka@gmail.com>
* Fix helmfile template render debug log to not mention "first-pass" in v1 mode
Signed-off-by: Yusuke Kuoka <ykuoka@gmail.com>
Signed-off-by: Yusuke Kuoka <ykuoka@gmail.com>
The go-getter driver (named `Remote`) is used to implement remote values. I found some unused functions in its implementation. This commit removes those unused functions so that it becomes more maintainable.
Signed-off-by: Yusuke Kuoka <ykuoka@gmail.com>
Signed-off-by: Yusuke Kuoka <ykuoka@gmail.com>
Ref https://github.com/helmfile/helmfile/discussions/593
* implemented --reset-values flag overriding .helmDefaults.reuseValues=true
* tests for --reset-values flag
* fixed pkg/app tests
* added tests for both reset & reuse flags given
* added appendValuesControlMode method for --reuse/reset-values flag processing
* updated code comments & docs
Signed-off-by: Karol Ossowski <k@koralsky.pl>
This should fix#435 for Helmfile v0.x releases since the next v0.150.0.
We introduce a new envvar to opt-in to the new YAML library, so that you can give it a shot before upgrading your Helmfile to v1. The same envvar can be used to opt-out of the new YAML library after you upgrade to Helmfile v1, giving you a more flexible migration story.
Signed-off-by: Yusuke Kuoka <ykuoka@gmail.com>
Previously a values template rendering failure resulted in a cryptic error message like:
```
g release templates in "input.yaml": failed executing templates in release "input.yaml"."foo1": failed executing template expressions in release "foo1".values[0] = "[98 97 115 101 76 97 98 101 108 58 32 34 123 123 32 103 101 116 32 46 82 101 108 101 97 115 101 46 76 97 98 101 108 115 32 92 34 98 97 115 101 92 34 32 92 34 78 79 84 95 73 78 72 69 82 73 84 69 68 92 34 32 125 125 34 10]": template: stringTemplate:1: unexpected "\\" in operand
```
The issue was that the values template at the value position of the YAML dict was printed as a formatted byte array. We now print it as a standard string so that it is easily readable.
Signed-off-by: Yusuke Kuoka <ykuoka@gmail.com>
Signed-off-by: Yusuke Kuoka <ykuoka@gmail.com>
* fix env value lost in environment values
Signed-off-by: yxxhero <aiopsclub@163.com>
* add more test
Signed-off-by: yxxhero <aiopsclub@163.com>
Signed-off-by: yxxhero <aiopsclub@163.com>
This is a successor to #596. We need a smooth migration path from `gopkg.in/yaml.v2`, and this pull request moves it forward with `goccy/go-yaml` instead of `gopkg.in/yaml.v3`. Merging this unblocks users stuck in Helmfile v0.146.x or earlier due to #435, so that they can upgrade to 0.147.x or greater without updating their helmfile configs.
We previously tried to upgrade to `yaml.v3` (https://github.com/helmfile/helmfile/issues/394) in Helmfile v0.x, presuming it won't break anything. Apparently, it broke use-cases where you want to layer release's `values` field over three or more release templates and releases (#435).
We then tried to bring back `yaml.v2` for Helmfile v0.x and keep `yaml.v3` for the upcoming Helmfile v1. However, it failed due to incompatibility in the Unmarshaller interface between `yaml.v2` and `yaml.v3` (https://github.com/helmfile/helmfile/pull/596).
`goccy/go-yaml` is, from my observation, a well-maintained alternative to `yaml.v2`. One of its premises is that it enables us to swap the implementation from `gopkg.in/yaml.v2` to `goccy/go-yaml` just by replacing the import directive. It seems to use the same `Unmarshaller` interface as yaml.v2 too.
Once this PR gets merged, I'd like to follow-up with adding a new build-time variable and an envvar to set the proper default for the yaml parser Helmfile uses and the ability to switch the parser at runtime. All in all, the next Helmfile release, v0.150.0 will get reverted to use `gopkg.in/yaml.v2` by default which resolves#435.
New users who started using Helmfile since any of v0.148.0, v0.148.1, and v0.149.0 might be already relying on the new behavior, They might need to specify a new envvar to enable `goccy/go-yaml`.
Signed-off-by: yxxhero <aiopsclub@163.com>
Signed-off-by: yxxhero <aiopsclub@163.com>
Co-authored-by: yxxhero <aiopsclub@163.com>
This is a successor to #440 rebuilt on top of #594 so that we can merge this while we are still at Hemlfile v0.x without worrying any backward-incompatibility. Much appreciation to @yxxhero for the original work!
Signed-off-by: Yusuke Kuoka <ykuoka@gmail.com>
Signed-off-by: Yusuke Kuoka <ykuoka@gmail.com>
* feat: Helmfile V1 mode
We add a new "V1 mode" to Helmfile so that you can seemlessly upgrade Helmfile from the current v0.x to the upcoming v1.0.
The idea is that we build both v0 and v1 binaries from the same tagged commit within the main branch, with different defaults for the "V1 mode"- the V1 mode is disabled by default for v0.x binaries, while it is enabled by default for v1.x binaries.
The V1 mode can be overrode at runtime via envvar. That is, even after upgrading the binary to v1, you will not see any backward-incompatible changes while you explicitly set an envvar, `HELMFILE_V1MODE=true`, at runtime.
Signed-off-by: Yusuke Kuoka <ykuoka@gmail.com>
Signed-off-by: yxxhero <aiopsclub@163.com>
All the dependencies get correctly installed when dealing with remote
charts.
If there's a local chart that depends on remote dependencies then those
don't get automatically installed. See #526. They end up with this
error:
```
Error: no cached repository for helm-manager-b6cf96b91af4f01317d185adfbe32610179e5246214be9646a52cb0b86032272 found. (try 'helm repo update'): open /root/.cache/helm/repository/helm-manager-b6cf96b91af4f01317d185adfbe32610179e5246214be9646a52cb0b86032272-index.yaml: no such file or directory
```
One workaround for that would be to add the repositories from the local
charts. Something like this:
```
cd local-chart/ && helm dependency list $dir 2> /dev/null | tail +2 | head -n -1 | awk '{ print "helm repo add " $1 " " $3 }' | while read cmd; do $cmd; done
```
This however is not trivial to parse and implement.
An easier fix which I did here is just to not allow doing
`--skip-refresh` for local repositories.
Fixes#526
Signed-off-by: Indrek Juhkam <indrek@urgas.eu>
Signed-off-by: Indrek Juhkam <indrek@urgas.eu>
Signed-off-by: yxxhero <aiopsclub@163.com>
1. only implement post-renderer flags this patch
2. As mumoshu advise, add helmfile flags `--post-render` and add the
postRenderer config in helmDefaults and release. the priority is
helmfile flags > release > helmDefaults.
3. fix the test case in state_test.go and some other tests.
Signed-off-by: guofutan <guofutan@tencent.com>
Signed-off-by: yxxhero <aiopsclub@163.com>
* Fix presync hooks are not called on no diff when run apply subcommand
Signed-off-by: xiaomudk <xiaomudk@gmail.com>
* Update docs/index.md
Co-authored-by: Yusuke Kuoka <ykuoka@gmail.com>
Signed-off-by: xiaomudk <xiaomudk@gmail.com>
Signed-off-by: xiaomudk <xiaomudk@gmail.com>
Co-authored-by: Yusuke Kuoka <ykuoka@gmail.com>
Use the new runWithLogCapture helper instead of the long boilerplate to capture the log for snapshot testing.
Signed-off-by: Yusuke Kuoka <ykuoka@gmail.com>
Allow configuring the lockfile in the state. This makes it possible for
example maintain a lock per environment.
Signed-off-by: Lassi Pölönen <lassi.polonen@iki.fi>
Signed-off-by: Lassi Pölönen <lassi.polonen@iki.fi>
This improves the `helmfile sync` performance.
From the code: `BuildDeps` is used only by `runHelmDepBuilds`, which
only is used by `PrepareCharts` which is finally only used by
`withPreparedCharts`.
`withPreparedCharts` already does `SyncReposOnce` which means we do not
have to refresh the local repository cache on each chart build.
This is only supported in Helm v3.
This seems to be mostly affecting helmfiles which have a lot of releases
and those release charts use sub dependencies.
I saw significant performance improvements for a helmfile with 45
releases, 2 repositories, and most of the charts also had their own
dependencies. Results:
Before the patch:
* real 9m10.565s
* real 9m38.335s
* real 9m14.941s
* real 5m13.106s (with cache)
After the patch:
* real 6m51.965s
* real 6m36.605s
* real 6m31.685s
* real 3m0.271s (with cache)
These were tested with:
```
rm -rf ~/.cache/helmfile ~/.cache/helm ~/.config/helm/repositories.* && helmfile sync ...
```
The result with `(with cache)` was without deleting the caches first.
From these metrics it seems that the sync duration decreased 20-45%
depending on the run, release count, dependencies and if the cache was
used or not.
As far as I understand, this should be backward-compatible change.
Signed-off-by: Indrek Juhkam <indrek@urgas.eu>
Signed-off-by: Indrek Juhkam <indrek@urgas.eu>
I had been unhappy with the fact that our go-test output had a lot of debug log messages which obfuscated test results.
I'm finally removeing all those by directing the test log output to io.Discard.
Signed-off-by: Yusuke Kuoka <ykuoka@gmail.com>
* fix: child process not exit when recive term signal
Signed-off-by: yxxhero <aiopsclub@163.com>
* fix: wait for clean done
Signed-off-by: Yusuke Kuoka <ykuoka@gmail.com>
Signed-off-by: yxxhero <aiopsclub@163.com>
Co-authored-by: Yusuke Kuoka <ykuoka@gmail.com>
Most shells do not support hyphens in environment variables.
However, there are cases where you may want to include hyphens in the repository name.
Therefore, I have included a process in `gatherOCIUsernamePassword` to replace hyphens with underbar.
Signed-off-by: mugioka <okamugi0722@gmail.com>
* feat: show live output from the Helm binary
Signed-off-by: Rodrigo Fior Kuntzer <rodrigo@miro.com>
* fixup! Merge branch 'main' into enable-live-output
Signed-off-by: Yusuke Kuoka <ykuoka@gmail.com>
This changes list command so it doesn't run withPreparedCharts,
and just lists releases instead
Signed-off-by: Viktor Oreshkin <imselfish@stek29.rocks>
* add interactive in sync & remove --interactive in global options
Signed-off-by: yxxhero <aiopsclub@163.com>
* fix unittest
Signed-off-by: yxxhero <aiopsclub@163.com>
* same behave as apply when in interactive
Signed-off-by: yxxhero <aiopsclub@163.com>
Signed-off-by: yxxhero <aiopsclub@163.com>
Fixing releases being included which do not match the environment
requested, which is a regression introduced by #234. The issue remains
when Helmfile state values are supplied, which is not a regression and
will be addressed separately.
Partial resolution for #271
Signed-off-by: David Ackroyd <dackroyd@fairfaxmedia.com.au>
Signed-off-by: David Ackroyd <dackroyd@fairfaxmedia.com.au>
If remote file isn't present in repo we will add it anyway to the files list and check `if len(files) == 0 {` never will be true. That leads to missing section with `MissingFileHandler`.
That fix check that cloned file actually exists. In that case if we add a link to non-existing remote file `MissingFileHandler` will be called as expected.
Signed-off-by: Vladimir Kuznichenkov <kuzaxak.tech@gmail.com>
Signed-off-by: Adam Gardner <adam.gardner@magicmemories.com>
chore: fix test case broken by change of helm subcommand
Signed-off-by: Adam Gardner <adam.gardner@magicmemories.com>
This commit replaces `os.Setenv` with `t.Setenv` in tests. The
environment variable is automatically restored to its original value
when the test and all its subtests complete.
Reference: https://pkg.go.dev/testing#T.Setenv
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
* Make a few helmfile sub-commands to consistently support needs-related flags
* helmfile-diff adds support for --include-transitive-needs
* helmfile-template adds support for --skip-needs
* helmfile-lint adds support for --skip-needs, --include-needs, and --include-transitive-needs
Ref https://github.com/roboll/helmfile/issues/2055
Signed-off-by: Yusuke Kuoka <ykuoka@gmail.com>
* Fix a few helmfile-lint needs related bugs and add tests
Signed-off-by: Yusuke Kuoka <ykuoka@gmail.com>
* Is include-transitive-needs realy working as intended? 🤔
Signed-off-by: Yusuke Kuoka <ykuoka@gmail.com>
* Confirm that it does fail on unselected need by default
Signed-off-by: Yusuke Kuoka <ykuoka@gmail.com>
* Add missing testdata
Signed-off-by: Yusuke Kuoka <ykuoka@gmail.com>
* Test helmfile-template for include/skip needs support
Signed-off-by: Yusuke Kuoka <ykuoka@gmail.com>
* Fix a few terms
Signed-off-by: Yusuke Kuoka <ykuoka@gmail.com>
* Add more tests to better know the current helmfile-diff behavior around needs
Signed-off-by: Yusuke Kuoka <ykuoka@gmail.com>
* Fix failing tests
Signed-off-by: Yusuke Kuoka <ykuoka@gmail.com>
* Fix helmfile-diff to consistently handle skip/include-needs
Signed-off-by: Yusuke Kuoka <ykuoka@gmail.com>
* Extract testhelper.RequireLog for reusing
Signed-off-by: Yusuke Kuoka <ykuoka@gmail.com>
* Fix all bugs and test cases for TestDiff and TestDiff_2
Signed-off-by: Yusuke Kuoka <ykuoka@gmail.com>
* Fix TestDiff_2
Signed-off-by: Yusuke Kuoka <ykuoka@gmail.com>
* Fix TestDiff
Signed-off-by: Yusuke Kuoka <ykuoka@gmail.com>
* Fix TestDiffWithNeeds
Signed-off-by: Yusuke Kuoka <ykuoka@gmail.com>
* Unify behavior on including disabled releases as needs for lint and template
Signed-off-by: Yusuke Kuoka <ykuoka@gmail.com>
* Fix bug that --include-transitive-needs does not imply include-needs
Signed-off-by: Yusuke Kuoka <ykuoka@gmail.com>
* introduce DISABLE_INSECURE_FEATURES to disable insecure executions
Signed-off-by: Quan TRAN <account@itscaro.me>
* disable remote sources when DISABLE_INSECURE_FEATURES is set to "true"
Signed-off-by: Quan TRAN <account@itscaro.me>
* refactor envvar package
Signed-off-by: Quan TRAN <account@itscaro.me>
* (test) fix test fixtures
Signed-off-by: Quan TRAN <account@itscaro.me>
* use absolute path to avoid unit test failure
Signed-off-by: Quan TRAN <account@itscaro.me>
* Fix conflicts
Co-authored-by: Yusuke Kuoka <ykuoka@gmail.com>
* Adds feature to fetch environment values from remote
The releases and environment section allow for values files on the local
disk.
This enhancement allows for referencing remote (go-getter) files to be
fetched, cached and referenced.
In addition when fetching a remote git source with a ssh key the ssh key
will not be part of the caching folder name. This avoids two problems:
1. Don't leak sensitive information in the name of the caching folder
2. Base64 encoded SSH keys are very long. On some file systems the max
lenght of the directory name is hit when using the full base64
information in the path name.
The sshkey informations are reducted. Because of this fixed string
there is a change of colloding cache names. The likelihood of this
collision is very low. The git repo and git reference need to be the
same, but the sshkey can change. This will result in the same source to
be checkout out and referenced.
Signed-off-by: Lüchinger Dominic <dev@snowgarden.ch>
* Update pkg/state/storage.go
Co-authored-by: Yusuke Kuoka <ykuoka@gmail.com>
Since helm-diff has added an ability to auto-detect the term to decide if it should output with color or not, helmfile had been defaulted to no-color.
This resoloves that, by adding a term-detection logic that is same as helm-diff.
As a part of this work, I have also implemented a new global flag `--color`, which is used for forcing color without relying on the term-detection logic implemented in helmfile or explicitly setting the HELM_DIFF_COLOR envvar. I hope it is useful for folks.
Ref https://github.com/roboll/helmfile/issues/2043
Signed-off-by: Yusuke Kuoka <ykuoka@gmail.com>
When chartify is involved due to the use of `forceNamespace`, `strategicMergePatches`, `jsonPatches`, and so on, We had been internally mutating the Release.Chart with the path to the local temporary directory that contains the modified version of the chart.
This resulted in us unintentionally making `helmfile deps` to remove entries for the chart being modified out of helmfile.lock file, which resulted in issues like #2110.
To be clear, although the original issue is reported to occur for `strategicMergePatches`, I believe that it occurered also for any remote charts using `jsonPatches` and `forceNamespace` too.
I also believe this has been the issue since our introduction of chartify (maybe a year or so ago??), and I guess why it took so much time to be found and reported is that not so many people with chartify in combination with `helmfile deps` 🤔
Lastly, this changes chart names surfaced in the various log output from Helmfile, from temporary chart paths to the chart name/path declared in the helmfile.yaml. I think this is generally a good change, no fear of being a breaking change. But if anyone has any concern about that, please feel free to comment/report/etc.
Ref https://github.com/roboll/helmfile/issues/2110
Signed-off-by: Yusuke Kuoka <ykuoka@gmail.com>
This adds support for `kube-version` and `api-versions` to be available to `chartify` so that it works even if your release requires `chartify` due to that you use features like `forceNamespace`, `jsonPatches`, `strategicMergePatches`, and so on.
This also enhances `ReleaseSpec` which corresponds to each item of `releases[]` in your `helmfile.yaml` to also accept `kubeVersion` and `apiVersions`, in addition to the top-level `kubeVersion` and `apiVersions` we have today.
The top-level ones works as the default values for release-specific ones. If you have been using the top-level ones, keep using it. It is backward-compatible. If you want to specify it per release, because, for example, your releases are deployed across clusters(in case you differentiate `kubeContext` fields), try the new fields added to the release spec.
Resolves#1864
It turned out that Helmfile has never had support for release template on `needs`.
This adds that, along with the new end-to-end test suite to verify helmfile template output with snapshot testing involving a real `helmfile build` command.
Ref #2098
yxxhero
Mitsuo Heijo
xiaomudk
Yusuke Kuoka
koralsky
Quan TRAN
Indrek Juhkam
guofutan
koralsky
yxxhero
Lassi Pölönen
Helge Eichelberg
Rui Chen
Kevin Leturc
Felipe Santos
mugioka
Anton Bretting
Tsubasa Nagasawa
Arpan
Jean-Yves CAMIER
Rodrigo Fior Kuntzer
KqLLL
Viktor Oreshkin
Arkaitz Jimenez
Michael Lorant
David Ackroyd
Vladimir Kuznichenkov
Sergii Iefremov
vasicvuk
Quan TRAN
Adam Gardner
Radon Rosborough
Eng Zer Jun
Sergio Morales
Dominic
austin ce
Grégory SANCHEZ