diff --git a/docs/hcl_funcs.md b/docs/hcl_funcs.md new file mode 100644 index 00000000..0f54183f --- /dev/null +++ b/docs/hcl_funcs.md @@ -0,0 +1,667 @@ +# HCL Functions + +## Standard Library +The following functions are all from the [go-cty](https://pkg.go.dev/github.com/zclconf/go-cty/cty/function/stdlib#pkg-functions), [go-cty-yaml](https://pkg.go.dev/github.com/zclconf/go-cty-yaml) and [hcl](https://pkg.go.dev/github.com/hashicorp/hcl/v2@v2.20.1/ext/tryfunc#section-readme) libraries +#### abs +`abs` returns the absolute value +``` +abs(number) +``` +``` +abs(-1) +# 1 +abs(2) +# 2 +``` +#### can +`can` evaluates an expression and returns a boolean if a result can be produced without any error +``` +can(expr) +``` +``` +map = { + myvar = "myvar" +} +can1 = can(hv.map.myVar) +# true +can2 = can(hv.map.notMyVar) +# false +``` +#### ceil +`ceil` returns the ceiling value of a given number +``` +ceil(number) +``` +``` +ceil(1) +# 1 +ceil(1.1) +# 2 +``` +#### chomp +`chomp` removes newline characters at the end of a string. +``` +chomp(string) +``` +``` +chomp("myVar\n") +# myVar +``` +#### coalesce + +`coalesce` returns the first of the given arguments that is not null. If all arguments are null, an error is produced. +All arguments must be of the same type apart from some cases +``` +coalesce(any...) +``` +``` +coalesce(null, 2) +# 2 +coalesce(null, "value") +# value +``` +Use the three dots notation `...` to expand a list +``` +coalesce([null, "value"]...) +# value +``` +#### coalescelist +`coalescelist` takes any number of list arguments and returns the first one that isn't empty. +``` +coalescelist(list) +``` +``` +coalescelist([], ["value"]) +# ["value"] +``` +Use the three dots notation `...` when using list of lists +``` +coalescelist([[], ["val1", "val2"]]...) +# ["val1", "val2"] +``` +#### compact +`compact` returns a new list with any empty string elements removed. +``` +compact(list) +``` +``` +compact(["", "val1", "val2"]) +# ["val1", "val2"] +``` +#### concat +`concat` takes one or more sequences (lists or tuples) and returns the single sequence that results from concatenating them together in order. +``` +concat(list, list...) +``` +``` +concat(["val1"], ["val2", "val3"]) +# ["val1", "val2", "val3"] +``` +#### contains +`contains` returns a boolean if a list contains a given value +``` +contains(list, value) +``` +``` +contains(["val1", "val2"], "val2") +# true +``` +#### csvdecode +`csvdecode` decodes a CSV-formatted string into a list of maps +``` +csvdecode(string) +``` +``` +csvdecode("col1,col2\nv1,v2\nv3,v4") +### +[ + { + "col1" = "v1" + "col2" = "v2" + }, + { + "col1" = "v3" + "col2" = "v4" + } +] +``` +#### distinct +`distinct` returns a new list from another by removing all duplicates +``` +distinct(list) +``` +``` +distinct(["v1","v1","v2"]) +["v1", "v2"] +``` +#### element +`element` returns a single element from a given list at the given index. If index is greater than the length of the list then it is wrapped modulo the list length +``` +element(list, index) +``` +``` +element(["val1","val2"], 1) +# val2 +``` + +#### chunklist +`chunklist` splits a single list into fixed-size chunks, returning a list of lists. +``` +chunklist(list, size) +``` +``` +chunklist(["a","b"], 1) +# [["a"], ["b"]] +``` +#### flatten +`flatten` takes a list and replaces any elements that are lists with a flattened sequence of the list contents. +``` +flatten(list) +``` +``` +flatten([["a"], ["a","b"], ["c"]]) +# ["a","a","b","c"] +``` +#### floor +`floor` returns the closest whole number lesser than or equal to the given value. +``` +floor(number) +``` +``` +floor(1) +# 1 +floor(0.7) +# 0 +``` +#### format +`format` produces a string representation of zero or more values using a format string similar to the "printf" function in C. +[Verbs details](https://pkg.go.dev/github.com/zclconf/go-cty/cty/function/stdlib#Format) +``` +format(format, values) +``` +``` +format("Hello %s", "world") +# Hello world +``` +#### formatdate +`formatdate` reformats a timestamp given in RFC3339 syntax into another time syntax defined by a given format string. +[Syntax details](https://pkg.go.dev/github.com/zclconf/go-cty/cty/function/stdlib#FormatDate) +``` +formatdate(string, timestampString) +``` +``` +formatdate("MMM DD YYYY", "2024-01-01T00:12:00Z") +# Jan 01 2024 +``` + +#### formatlist +`formatlist` does the same as `format` but for a list of strings +``` +formatlist(formatString, values...) +``` +``` +formatlist("%s", ["Hello", "World"]) +### +[ + "Hello", + "World" +] + +formatlist("%s %s", "hello", ["World", "You"]) +### +[ + "hello World", + "hello You", +] +``` +#### indent +`indent` adds a given number of spaces to the beginnings of all but the first line in a given multi-line string. +``` +indent(number, string) +``` + +``` +indent(4, "hello,\nWorld\n!") +### +hello + World + ! +``` +#### int +`int` removes the fractional component of the given number returning an integer representing the whole number component, rounding towards zero. + +``` +int(number) +``` +``` +int(6.2) +# 6 +``` + +#### join +`join` concatenates together the string elements of one or more lists with a given separator. +``` +join(listOfStrings, separator) +``` +``` +join(" ", ["hello", "world"]) +# hello world +``` + +#### jsondecode +`jsondecode` parses the given JSON string and, if it is valid, returns the value it represents. +``` +jsonencode(string) +``` +Example : +``` +jsonencode({"hello"="world"}) +# {"hello": "world"} +``` + +#### jsonencode +`jsonencode` returns a JSON serialization of the given value. +``` +jsondecode(string) +``` +Example : +``` +jsondecode("{\"hello\": \"world\"}") +# { hello = "world" } +``` +#### keys +`keys` takes a map and returns a sorted list of the map keys. + +``` +keys(map) +``` + +``` +keys({val1=1, val2=2, val3=3}) +# ["val1","val2","val3"] +``` +#### length +`length` returns the number of elements in the given __collection__. +See `strlen` for strings +``` +length(list) +``` + +``` +length([1,2,3]) +# 3 +``` +#### log +`log` returns returns the logarithm of a given number in a given base. +``` +log(number, base) +``` + +``` +log(1, 10) +# 0 +``` +#### lookup +`lookup` performs a dynamic lookup into a map. There are three required arguments, inputMap and key, plus a defaultValue, which is a value to return if the given key is not found in the inputMap. +``` +lookup(inputMap, key, defaultValue) +``` + +``` +map = { "luke" = "skywalker"} +lookup(hv.maptest, "luke", "none") +# skywalker +lookup(hv.maptest, "leia", "none") +# none +``` +#### lower +`lower` is a Function that converts a given string to lowercase. +``` +lower(string) +``` + +``` +lower("HELLO world") +# hello world +``` +#### max +`max` returns the maximum number from the given numbers. +``` +max(numbers) +``` + +``` +max(1,128,70) +# 128 + +``` +#### merge +`merge` takes an arbitrary number of maps and returns a single map that contains a merged set of elements from all of the maps. +``` +merge(maps) +``` +``` +merge({a="1"}, {a=[1,2], c="world"}, {d=40}) +# { a = [1,2], c = "world", d = 40} + +``` +#### min +`min` returns the minimum number from the given numbers. +``` +min(numbers) +``` +``` +min(1,128,70) +# 1 +``` +#### parseint +`parseint` parses a string argument and returns an integer of the specified base. +``` +parseint(string, base) +``` +``` +parseint("190", 10) +# 190 +parseint("11001", 2) +# 25 +``` +#### pow +`pow` returns the logarithm of a given number in a given base. +``` +pow(number, power) +``` + +``` +pow(1, 10) +# 1 +pow(3, 12) +# 531441 +``` +#### range +`range` creates a list of numbers by starting from the given starting value, then adding the given step value until the result is greater than or equal to the given stopping value. Each intermediate result becomes an element in the resulting list. +``` +range(startingNumber, stoppingNumber, stepNumber) +``` + +``` +range(1, 10, 3) +# [1, 4, 7] +``` +#### regex +`regex` is a function that extracts one or more substrings from a given string by applying a regular expression pattern, describing the first match. +The return type depends on the composition of the capture groups (if any) in the pattern: + + If there are no capture groups at all, the result is a single string representing the entire matched pattern. + If all of the capture groups are named, the result is an object whose keys are the named groups and whose values are their sub-matches, or null if a particular sub-group was inside another group that didn't match. + If none of the capture groups are named, the result is a tuple whose elements are the sub-groups in order and whose values are their sub-matches, or null if a particular sub-group was inside another group that didn't match. + It is invalid to use both named and un-named capture groups together in the same pattern. + +If the pattern doesn't match, this function returns an error. To test for a match, call `regexall` and check if the length of the result is greater than zero. +``` +regex(pattern, string) +``` + +``` +regex("[0-9]+", "v1.2.3") +# 1 +``` +#### regexall +`regexall` is similar to Regex but it finds all of the non-overlapping matches in the given string and returns a list of them. + +The result type is always a list, whose element type is deduced from the pattern in the same way as the return type for Regex is decided. + +If the pattern doesn't match at all, this function returns an empty list. +``` +regexall(pattern, string) +``` + +``` +regexall("[0-9]+", "v1.2.3") +# [1 2 3] +``` +#### setintersection +`setintersection` returns a new set containing the elements that exist in all of the given sets, which must have element types that can all be converted to some common type using the standard type unification rules. If conversion is not possible, an error is returned. +``` +setintersection(sets...) +``` + +``` +setintersection(["val1", "val2"], ["val1", "val3"], ["val1", "val2"]) +# ["val1"] +``` +#### setproduct +`setproduct` computes the Cartesian product of sets or sequences. +``` +setproduct(sets...) +``` +``` +setproduct(["host1", "host2"], ["stg.domain", "prod.domain"]) +### +[ + [ + "host1", + "stg.domain" + ], + [ + "host2", + "stg.domain" + ], + [ + "host1", + "prod.domain" + ], + [ + "host2", + "prod.domain" + ], +] +``` +#### setsubtract +`setsubtract` returns a new set containing the elements from the first set that are not present in the second set. The sets must have element types that can both be converted to some common type using the standard type unification rules. If conversion is not possible, an error is returned. +``` +setsubtract(sets...) +``` +``` +setsubtract(["a", "b", "c"], ["a", "b"]) +### +["c"] +``` +#### setunion +`setunion` returns a new set containing all of the elements from the given sets, which must have element types that can all be converted to some common type using the standard type unification rules. If conversion is not possible, an error is returned. +``` +setunion(sets...) +``` + +``` +setunion(["a", "b"], ["b", "c"], ["a", "d"]) +### +["a", "b", "c", "d"] +``` +#### signum +`signum` determines the sign of a number, returning a number between -1 and 1 to represent the sign. +``` +signum(number) +``` +``` +signum(-182) +# -1 +``` +#### slice +`slice` extracts some consecutive elements from within a list. +startIndex is inclusive, endIndex is exclusive +``` +slice(list, startIndex, endIndex) +``` +``` +slice([{"a" = "b"}, {"c" = "d"}, , {"e" = "f"}], 1, 1) +# [] +slice([{"a" = "b"}, {"c" = "d"}, {"e" = "f"}], 1, 2) +# [{"c" = "d"}] +``` +#### sort +`sort` re-orders the elements of a given list of strings so that they are in ascending lexicographical order. +``` +sort(list) +``` +``` +sort(["1", "h", "r", "p", "word"]) +# ["1", "h", "p", "r", "word"] +``` +#### split +`split` divides a given string by a given separator, returning a list of strings containing the characters between the separator sequences. +``` +split(separatorString, string) +``` +``` +split(".", "host.domain") +# ["host", "domain"] +``` +#### strlen +`strlen` is a Function that returns the length of the given string in characters. +``` +strlen(string) +``` +``` +strlen("yes") +# 3 +``` +#### strrev +`strrev` is a Function that reverses the order of the characters in the given string. +``` +strrev(string) +``` +``` +strrev("yes") +# "sey" +``` +#### substr +`substr` is a Function that extracts a sequence of characters from another string and creates a new string. +``` +substr(string, offsetNumber, length) +``` +``` +substr("host.domain", 0, 4) +# "host" +``` +#### timeadd +`timeadd` adds a duration to a timestamp, returning a new timestamp. +Only units "inferior" or equal to `h` are supported. +The duration can be negative. +``` +substr(timestamp, duration) +``` +``` +timeadd("2024-01-01T00:00:00Z", "-2600h10m") +# 2023-09-14T15:50:00Z +``` +#### trim +`trim` removes the specified characters from the start and end of the given string. +``` +trim(string, string) +``` +``` +trim("Can you do that ? Yes ?", "?") +# "Can you do that ? Yes" +``` +#### trimprefix +`trimprefix` removes the specified prefix from the start of the given string. +``` +trimprefix(stringToTrim, trimmingString) +``` +``` +trimprefix("please, do it", "please, ") +# "do it" +``` +#### trimspace +`trimspace` removes any space characters from the start and end of the given string. +``` +trimspace(string) +``` +``` +trimspace(" Hello World ") +# "Hello World" +``` +#### trimsuffix +`trimsuffix` removes the specified suffix from the end of the given string. +``` +trimsuffix(stringToTrim, trimmingString) +``` +``` +trimsuffix("Hello World", " World") +# "Hello" +``` +#### try +`try` is a variadic function that tries to evaluate all of is arguments in sequence until one succeeds, in which case it returns that result, or returns an error if none of them succeed. +``` +try(expressions...) +``` +``` +values { + map = { + hello = "you" + world = "us" + } + try(hv.map.do_not_exist, hv.map.world) +} +# "us" +``` +#### upper +`upper` is a Function that converts a given string to uppercase. +``` +upper(string) +``` +``` +upper("up") +# "UP" +``` +#### values +`values` returns a list of the map values, in the order of the sorted keys. This function only works on flat maps. +``` +values(map) +``` +``` +values({"a" = 1,"b" = 2}) +# [1, 2] +``` +#### yamldecode +`yamldecode` parses the given JSON string and, if it is valid, returns the value it represents. +``` +yamldecode(string) +``` +``` +yamldecode("hello: world\narray: [1, 2, 3]") +### +{ + array = [1, 2, 3] + hello = "world" +} +``` +#### yamlencode +`yamlencode` returns a JSON serialization of the given value. +``` +yamlencode({array = [1, 2, 3], hello = "world"}) +``` + +``` +yamlencode({array = [1, 2, 3], hello = "world"}) +### +"array": +- 1 +- 2 +- 3 +"hello": "world" +``` +#### zipmap +`zipmap` constructs a map from a list of keys and a corresponding list of values. +The lenght of each list must be equal +``` +zipmap(keysList, valuesList) +``` +``` +zipmap(["key1", "key2"], ["val1", "val2"]) +### +{ + "key1" = "val1" + "key2" = "val2" +} +``` \ No newline at end of file diff --git a/docs/index.md b/docs/index.md index e1dc489e..93a42f57 100644 --- a/docs/index.md +++ b/docs/index.md @@ -403,6 +403,9 @@ environments: # `{{ .Values.foo.bar }}` is evaluated to `1`. values: - environments/default/values.yaml + # Everything from the values.hcl in the `values` block is available via `{{ .Values.KEY }}`. + # More details in its dedicated section + - environments/default/values.hcl # Each entry in values can be either a file path or inline values. # The below is an example of inline values, which is merged to the `.Values` - myChartVer: 1.0.0-dev @@ -896,7 +899,11 @@ releases: # snip ``` -## Environment Values +### Environment Values +Helmfile supports 3 values languages : +- Straight yaml +- Go templates to generate straight yaml +- HCL Environment Values allows you to inject a set of values specific to the selected environment, into `values.yaml` templates. Use it to inject common values from the environment to multiple values files, to make your configuration DRY. @@ -967,7 +974,45 @@ releases: ... ``` -### Note on Environment.Values vs Values +#### HCL specifications + +Since Helmfile v0.164.0, HCL language is supported for environment values only. +HCL values supports interpolations and sharing values across files + +* Only `.hcl` suffixed files will be interpreted as is +* Helmfile supports 2 differents blocks: `values` and `locals` +* `values` block is a shared block where all values are accessible everywhere in all loaded files +* `locals` block can't reference external values apart from the ones in the block itself, and where its defined values are only accessible in its local file +* Only values in `values` blocks are made available to the final root `.Values` (e.g : ` values { myvar = "var" }` is accessed through `{{ .Values.myvar }}`) +* There can only be 1 `locals` block per file +* Helmfile hcl `values` are referenced using the `hv` accessor. +* Helmfile hcl `locals` are referenced using the `local` accessor. +* Duplicated variables across .hcl `values` blocks are forbidden (An error will pop up specifying where are the duplicates) +* All cty [standard library functions](`https://pkg.go.dev/github.com/zclconf/go-cty@v1.14.3/cty/function/stdlib`) are available and custom functions could be created in the future + +Consider the following example : + +```terraform +# values1.hcl +locals { + hostname = "host1" +} +values { + domain = "DEV.EXAMPLE.COM" + hostnameV1 = "${local.hostname}.${lower(hv.domain)}" # "host1.dev.example.com" +} +``` +```terraform +# values2.hcl +locals { + hostname = "host2" +} + +values { + hostnameV2 = "${local.hostname}.${hv.domain}" # "host2.DEV.EXAMPLE.COM" +} +``` +#### Note on Environment.Values vs Values The `{{ .Values.foo }}` syntax is the recommended way of using environment values. @@ -976,46 +1021,7 @@ This is still working but is **deprecated** and the new `{{ .Values.foo }}` synt You can read more infos about the feature proposal [here](https://github.com/roboll/helmfile/issues/640). -### Loading remote Environment values files - -Since Helmfile v0.118.8, you can use `go-getter`-style URLs to refer to remote values files: - -```yaml -environments: - cluster-azure-us-west: - values: - - git::https://git.company.org/helmfiles/global/azure.yaml?ref=master - - git::https://git.company.org/helmfiles/global/us-west.yaml?ref=master - - git::https://gitlab.com/org/repository-name.git@/config/config.test.yaml?ref=main # Public Gilab Repo - cluster-gcp-europe-west: - values: - - git::https://git.company.org/helmfiles/global/gcp.yaml?ref=master - - git::https://git.company.org/helmfiles/global/europe-west.yaml?ref=master - - git::https://ci:{{ env "CI_JOB_TOKEN" }}@gitlab.com/org/repository-name.git@/config.dev.yaml?ref={{ env "APP_COMMIT_SHA" }} # Private Gitlab Repo - staging: - values: - - git::https://{{ env "GITHUB_PAT" }}@github.com/[$GITHUB_ORGorGITHUB_USER]/repository-name.git@/values.dev.yaml?ref=main #Github Private repo - - http://$HOSTNAME/artifactory/example-repo-local/test.tgz@values.yaml #Artifactory url ---- - -releases: - - ... -``` - -Since Helmfile v0.158.0, support more protocols, such as: s3, https, http -``` -values: - - s3::https://helm-s3-values-example.s3.us-east-2.amazonaws.com/values.yaml - - s3://helm-s3-values-example/subdir/values.yaml - - https://john:doe@helm-s3-values-example.s3.us-east-2.amazonaws.com/values.yaml - - http://helm-s3-values-example.s3.us-east-2.amazonaws.com/values.yaml -``` - -For more information about the supported protocols see: [go-getter Protocol-Specific Options](https://github.com/hashicorp/go-getter#protocol-specific-options-1). - -This is particularly useful when you co-locate helmfiles within your project repo but want to reuse the definitions in a global repo. - -## Environment Secrets +### Environment Secrets Environment Secrets *(not to be confused with Kubernetes Secrets)* are encrypted versions of `Environment Values`. You can list any number of `secrets.yaml` files created using `helm secrets` or `sops`, so that @@ -1056,7 +1062,7 @@ Then the environment secret `foo.bar` can be referenced by the below template ex {{ .Values.foo.bar }} ``` -### Loading remote Environment secrets files +#### Loading remote Environment secrets files Since Helmfile v0.149.0, you can use `go-getter`-style URLs to refer to remote secrets files, the same way as in values files: ```yaml @@ -1071,6 +1077,91 @@ environments: - http://$HOSTNAME/artifactory/example-repo-local/test.tgz@environments/production.secret.yaml ``` +### Loading remote Environment values files + +Since Helmfile v0.118.8, you can use `go-getter`-style URLs to refer to remote values files: + +```yaml +environments: + cluster-azure-us-west: + values: + - git::https://git.company.org/helmfiles/global/azure.yaml?ref=master + - git::https://git.company.org/helmfiles/global/us-west.yaml?ref=master + - git::https://gitlab.com/org/repository-name.git@/config/config.test.yaml?ref=main # Public Gilab Repo + cluster-gcp-europe-west: + values: + - git::https://git.company.org/helmfiles/global/gcp.yaml?ref=master + - git::https://git.company.org/helmfiles/global/europe-west.yaml?ref=master + - git::https://ci:{{ env "CI_JOB_TOKEN" }}@gitlab.com/org/repository-name.git@/config.dev.yaml?ref={{ env "APP_COMMIT_SHA" }} # Private Gitlab Repo + staging: + values: + - git::https://{{ env "GITHUB_PAT" }}@github.com/[$GITHUB_ORGorGITHUB_USER]/repository-name.git@/values.dev.yaml?ref=main #Github Private repo + - http://$HOSTNAME/artifactory/example-repo-local/test.tgz@values.yaml #Artifactory url +--- + +releases: + - ... +``` + +Since Helmfile v0.158.0, support more protocols, such as: s3, https, http +``` +values: + - s3::https://helm-s3-values-example.s3.us-east-2.amazonaws.com/values.yaml + - s3://helm-s3-values-example/subdir/values.yaml + - https://john:doe@helm-s3-values-example.s3.us-east-2.amazonaws.com/values.yaml + - http://helm-s3-values-example.s3.us-east-2.amazonaws.com/values.yaml +``` + +For more information about the supported protocols see: [go-getter Protocol-Specific Options](https://github.com/hashicorp/go-getter#protocol-specific-options-1). + +This is particularly useful when you co-locate helmfiles within your project repo but want to reuse the definitions in a global repo. + +### Environment values precedence +With the introduction of HCL, a new value precedence was introduced over environment values. +Here is the order of precedence from least to greatest (the last one overrides all others) +1. `yaml` / `yaml.gotmpl` +2. `hcl` +3. `yaml` secrets + +Example: + +--- + +```yaml +# values1.yaml +domain: "dev.example.com" +``` + +```terraform +# values2.hcl +values { + domain = "overdev.example.com" + willBeOverriden = "override_me" +} +``` + +```yaml +# secrets.yml (assuming this one has been encrypted) +willBeOverriden: overrided +``` + +``` +# helmfile.yaml.gotmpl +environments: + default: + values: + - value1.yaml + - value2.hcl + secrets: + - secrets.yml +--- +releases: +- name: random-release + [...] + values: + domain: "{{ .Values.domain }}" # == "overdev.example.com" + willBeOverriden: "{{ .Values.willBeOverriden }}" # == "overrided" +``` ## DAG-aware installation/deletion ordering with `needs` `needs` controls the order of the installation/deletion of the release: diff --git a/go.mod b/go.mod index b7becb73..81ead953 100644 --- a/go.mod +++ b/go.mod @@ -14,6 +14,7 @@ require ( github.com/google/go-cmp v0.6.0 github.com/gosuri/uitable v0.0.4 github.com/hashicorp/go-getter v1.7.4 + github.com/hashicorp/hcl/v2 v2.19.1 github.com/helmfile/chartify v0.19.0 github.com/helmfile/vals v0.37.0 github.com/imdario/mergo v0.3.16 @@ -22,6 +23,8 @@ require ( github.com/stretchr/testify v1.9.0 github.com/tatsushid/go-prettytable v0.0.0-20141013043238-ed2d14c29939 github.com/variantdev/dag v1.1.0 + github.com/zclconf/go-cty v1.14.3 + github.com/zclconf/go-cty-yaml v1.0.3 go.szostok.io/version v1.2.0 go.uber.org/zap v1.27.0 golang.org/x/sync v0.7.0 @@ -65,9 +68,9 @@ require ( github.com/hashicorp/go-multierror v1.1.1 // indirect github.com/hashicorp/go-rootcerts v1.0.2 // indirect github.com/hashicorp/go-safetemp v1.0.0 // indirect - github.com/hashicorp/go-slug v0.8.1 // indirect + github.com/hashicorp/go-slug v0.12.2 // indirect github.com/hashicorp/go-sockaddr v1.0.2 // indirect - github.com/hashicorp/go-tfe v1.2.0 // indirect + github.com/hashicorp/go-tfe v1.36.0 // indirect github.com/hashicorp/golang-lru v1.0.2 // indirect github.com/hashicorp/hcl v1.0.0 // indirect github.com/hashicorp/jsonapi v0.0.0-20210826224640-ee7dae0fb22d // indirect @@ -132,9 +135,11 @@ require ( github.com/DopplerHQ/cli v0.5.11-0.20230908185655-7aef4713e1a4 // indirect github.com/Microsoft/hcsshim v0.11.4 // indirect github.com/ProtonMail/go-crypto v0.0.0-20230923063757-afb1ddc0824c // indirect + github.com/agext/levenshtein v1.2.3 // indirect github.com/alessio/shellescape v1.4.1 // indirect github.com/antchfx/jsonquery v1.3.3 // indirect github.com/antchfx/xpath v1.2.3 // indirect + github.com/apparentlymart/go-textseg/v15 v15.0.0 // indirect github.com/araddon/dateparse v0.0.0-20210429162001-6b43995a97de // indirect github.com/asaskevich/govalidator v0.0.0-20230301143203-a9d515a09cc2 // indirect github.com/atotto/clipboard v0.1.4 // indirect @@ -206,7 +211,7 @@ require ( github.com/googleapis/enterprise-certificate-proxy v0.3.2 // indirect github.com/gorilla/mux v1.8.0 // indirect github.com/gregjones/httpcache v0.0.0-20180305231024-9cad4c3443a7 // indirect - github.com/hashicorp/go-retryablehttp v0.7.1 // indirect + github.com/hashicorp/go-retryablehttp v0.7.4 // indirect github.com/hashicorp/go-secure-stdlib/parseutil v0.1.6 // indirect github.com/hashicorp/go-secure-stdlib/strutil v0.1.2 // indirect github.com/hashicorp/go-version v1.6.0 // indirect diff --git a/go.sum b/go.sum index 18ab94cf..4e6acc8a 100644 --- a/go.sum +++ b/go.sum @@ -266,6 +266,8 @@ github.com/Shopify/logrus-bugsnag v0.0.0-20171204204709-577dee27f20d h1:UrqY+r/O github.com/Shopify/logrus-bugsnag v0.0.0-20171204204709-577dee27f20d/go.mod h1:HI8ITrYtUY+O+ZhtlqUnD8+KwNPOyugEhfP9fdUIaEQ= github.com/a8m/envsubst v1.4.2 h1:4yWIHXOLEJHQEFd4UjrWDrYeYlV7ncFWJOCBRLOZHQg= github.com/a8m/envsubst v1.4.2/go.mod h1:MVUTQNGQ3tsjOOtKCNd+fl8RzhsXcDvvAEzkhGtlsbY= +github.com/agext/levenshtein v1.2.3 h1:YB2fHEn0UJagG8T1rrWknE3ZQzWM06O8AMAatNn7lmo= +github.com/agext/levenshtein v1.2.3/go.mod h1:JEDfjyjHDjOF/1e4FlBE/PkbqA9OfWu2ki2W0IB5558= github.com/alecthomas/template v0.0.0-20160405071501-a0175ee3bccc/go.mod h1:LOuyumcjzFXgccqObfd/Ljyb9UuFJ6TxHnclSeseNhc= github.com/alecthomas/units v0.0.0-20151022065526-2efee857e7cf/go.mod h1:ybxpYRFXyAe+OPACYpWeL0wqObRcbAqCMya13uyzqw0= github.com/alessio/shellescape v1.4.1 h1:V7yhSDDn8LP4lc4jS8pFkt0zCnzVJlG5JXy9BVKJUX0= @@ -275,6 +277,8 @@ github.com/antchfx/jsonquery v1.3.3/go.mod h1:1JG4DqRlRCHgVYDPY1ioYFAGSXGfWHzNgr github.com/antchfx/xpath v1.2.3 h1:CCZWOzv5bAqjVv0offZ2LVgVYFbeldKQVuLNbViZdes= github.com/antchfx/xpath v1.2.3/go.mod h1:i54GszH55fYfBmoZXapTHN8T8tkcHfRgLyVwwqzXNcs= github.com/antihax/optional v1.0.0/go.mod h1:uupD/76wgC+ih3iEmQUL+0Ugr19nfwCT1kdvxnR2qWY= +github.com/apparentlymart/go-textseg/v15 v15.0.0 h1:uYvfpb3DyLSCGWnctWKGj857c6ew1u1fNQOlOtuGxQY= +github.com/apparentlymart/go-textseg/v15 v15.0.0/go.mod h1:K8XmNZdhEBkdlyDdvbmmsvpAG721bKi0joRfFdHIWJ4= github.com/araddon/dateparse v0.0.0-20210429162001-6b43995a97de h1:FxWPpzIjnTlhPwqqXc4/vE0f7GvRjuAsbW+HOIe8KnA= github.com/araddon/dateparse v0.0.0-20210429162001-6b43995a97de/go.mod h1:DCaWoUhZrYW9p1lxo/cm8EmUOOzAPSEZNGF2DK1dJgw= github.com/armon/go-radix v0.0.0-20180808171621-7fddfc383310/go.mod h1:ufUuZ+zHj4x4TnLV4JWEpy2hxWSpsRywHrMgIH9cCH8= @@ -638,7 +642,6 @@ github.com/grpc-ecosystem/grpc-gateway v1.16.0/go.mod h1:BDjrQk3hbvj6Nolgz8mAMFb github.com/hashicorp/errwrap v1.0.0/go.mod h1:YH+1FKiLXxHSkmPseP+kNlulaMuP3n2brvKWEqk/Jc4= github.com/hashicorp/errwrap v1.1.0 h1:OxrOeh75EUXMY8TBjag2fzXGZ40LB6IKw45YeGUDY2I= github.com/hashicorp/errwrap v1.1.0/go.mod h1:YH+1FKiLXxHSkmPseP+kNlulaMuP3n2brvKWEqk/Jc4= -github.com/hashicorp/go-cleanhttp v0.5.1/go.mod h1:JpRdi6/HCYpAwUzNwuwqhbovhLtngrth3wmdIIUrZ80= github.com/hashicorp/go-cleanhttp v0.5.2 h1:035FKYIWjmULyFRBKPs8TBQoi0x6d9G4xc9neXJWAZQ= github.com/hashicorp/go-cleanhttp v0.5.2/go.mod h1:kO/YDlP8L1346E6Sodw+PrpBSV4/SoxCXGY6BqNFT48= github.com/hashicorp/go-getter v1.7.4 h1:3yQjWuxICvSpYwqSayAdKRFcvBl1y/vogCxczWSmix0= @@ -649,8 +652,8 @@ github.com/hashicorp/go-hclog v1.2.1/go.mod h1:W4Qnvbt70Wk/zYJryRzDRU/4r0kIg0PVH github.com/hashicorp/go-multierror v1.0.0/go.mod h1:dHtQlpGsu+cZNNAkkCN/P3hoUDHhCYQXV3UM06sGGrk= github.com/hashicorp/go-multierror v1.1.1 h1:H5DkEtf6CXdFp0N0Em5UCwQpXMWke8IA0+lD48awMYo= github.com/hashicorp/go-multierror v1.1.1/go.mod h1:iw975J/qwKPdAO1clOe2L8331t/9/fmwbPZ6JB6eMoM= -github.com/hashicorp/go-retryablehttp v0.7.1 h1:sUiuQAnLlbvmExtFQs72iFW/HXeUn8Z1aJLQ4LJJbTQ= -github.com/hashicorp/go-retryablehttp v0.7.1/go.mod h1:vAew36LZh98gCBJNLH42IQ1ER/9wtLZZ8meHqQvEYWY= +github.com/hashicorp/go-retryablehttp v0.7.4 h1:ZQgVdpTdAL7WpMIwLzCfbalOcSUdkDZnpUv3/+BxzFA= +github.com/hashicorp/go-retryablehttp v0.7.4/go.mod h1:Jy/gPYAdjqffZ/yFGCFV2doI5wjtH1ewM9u8iYVjtX8= github.com/hashicorp/go-rootcerts v1.0.2 h1:jzhAVGtqPKbwpyCPELlgNWhE1znq+qwJtW5Oi2viEzc= github.com/hashicorp/go-rootcerts v1.0.2/go.mod h1:pqUvnprVnM5bf7AOirdbb01K4ccR319Vf4pU3K5EGc8= github.com/hashicorp/go-safetemp v1.0.0 h1:2HR189eFNrjHQyENnQMMpCiBAsRxzbTMIgBhEyExpmo= @@ -660,14 +663,14 @@ github.com/hashicorp/go-secure-stdlib/parseutil v0.1.6/go.mod h1:QmrqtbKuxxSWTN3 github.com/hashicorp/go-secure-stdlib/strutil v0.1.1/go.mod h1:gKOamz3EwoIoJq7mlMIRBpVTAUn8qPCrEclOKKWhD3U= github.com/hashicorp/go-secure-stdlib/strutil v0.1.2 h1:kes8mmyCpxJsI7FTwtzRqEy9CdjCtrXrXGuOpxEA7Ts= github.com/hashicorp/go-secure-stdlib/strutil v0.1.2/go.mod h1:Gou2R9+il93BqX25LAKCLuM+y9U2T4hlwvT1yprcna4= -github.com/hashicorp/go-slug v0.8.1 h1:srN7ivgAjHfZddYY1DjBaihRCFy20+vCcOrlx1O2AfE= -github.com/hashicorp/go-slug v0.8.1/go.mod h1:Ib+IWBYfEfJGI1ZyXMGNbu2BU+aa3Dzu41RKLH301v4= +github.com/hashicorp/go-slug v0.12.2 h1:Gb6nxnV5GI1UVa3aLJGUj66J8AOZFnjIoYalNCp2Cbo= +github.com/hashicorp/go-slug v0.12.2/go.mod h1:JZVtycnZZbiJ4oxpJ/zfhyfBD8XxT4f0uOSyjNLCqFY= github.com/hashicorp/go-sockaddr v1.0.2 h1:ztczhD1jLxIRjVejw8gFomI1BQZOe2WoVOu0SyteCQc= github.com/hashicorp/go-sockaddr v1.0.2/go.mod h1:rB4wwRAUzs07qva3c5SdrY/NEtAUjGlgmH/UkBUC97A= -github.com/hashicorp/go-tfe v1.2.0 h1:L29LCo/qIjOqBUjfiUsZSAzBdxmsOLzwnwZpA+68WW8= -github.com/hashicorp/go-tfe v1.2.0/go.mod h1:tJF/OlAXzVbmjiimAPLplSLgwg6kZDUOy0MzHuMwvF4= -github.com/hashicorp/go-uuid v1.0.2 h1:cfejS+Tpcp13yd5nYHWDI6qVCny6wyX2Mt5SGur2IGE= -github.com/hashicorp/go-uuid v1.0.2/go.mod h1:6SBZvOh/SIDV7/2o3Jml5SYk/TvGqwFJ/bN7x4byOro= +github.com/hashicorp/go-tfe v1.36.0 h1:Wq73gjjDo/f9gkKQ5MVSb+4NNJ6T7c5MVTivA0s/bZ0= +github.com/hashicorp/go-tfe v1.36.0/go.mod h1:awOuTZ4K9F1EJsKBIoxonJlb7Axn3PIb8YeBLtm/G/0= +github.com/hashicorp/go-uuid v1.0.3 h1:2gKiV6YVmrJ1i2CKKa9obLvRieoRGviZFL26PcT/Co8= +github.com/hashicorp/go-uuid v1.0.3/go.mod h1:6SBZvOh/SIDV7/2o3Jml5SYk/TvGqwFJ/bN7x4byOro= github.com/hashicorp/go-version v1.6.0 h1:feTTfFNnjP967rlCxM/I9g701jU+RN74YKx2mOkIeek= github.com/hashicorp/go-version v1.6.0/go.mod h1:fltr4n8CU8Ke44wwGCBoEymUuxUHl09ZGVZPK5anwXA= github.com/hashicorp/golang-lru v0.5.0/go.mod h1:/m3WP610KZHVQ1SGc6re/UDhFvYD7pJ4Ao+sR/qLZy8= @@ -676,6 +679,8 @@ github.com/hashicorp/golang-lru v1.0.2 h1:dV3g9Z/unq5DpblPpw+Oqcv4dU/1omnb4Ok8iP github.com/hashicorp/golang-lru v1.0.2/go.mod h1:iADmTwqILo4mZ8BN3D2Q6+9jd8WM5uGBxy+E8yxSoD4= github.com/hashicorp/hcl v1.0.0 h1:0Anlzjpi4vEasTeNFn2mLJgTSwt0+6sfsiTG8qcWGx4= github.com/hashicorp/hcl v1.0.0/go.mod h1:E5yfLk+7swimpb2L/Alb/PJmXilQ/rhwaUYs4T20WEQ= +github.com/hashicorp/hcl/v2 v2.19.1 h1://i05Jqznmb2EXqa39Nsvyan2o5XyMowW5fnCKW5RPI= +github.com/hashicorp/hcl/v2 v2.19.1/go.mod h1:ThLC89FV4p9MPW804KVbe/cEXoQ8NZEh+JtMeeGErHE= github.com/hashicorp/hcp-sdk-go v0.91.0 h1:XFJuB/KWP7kh6u3c/ruUy/lSA272vRrHJ1kzusMx9zw= github.com/hashicorp/hcp-sdk-go v0.91.0/go.mod h1:vQ4fzdL1AmhIAbCw+4zmFe5Hbpajj3NvRWkJoVuxmAk= github.com/hashicorp/jsonapi v0.0.0-20210826224640-ee7dae0fb22d h1:9ARUJJ1VVynB176G1HCwleORqCaXm/Vx0uUi0dL26I0= @@ -950,6 +955,10 @@ github.com/yvasiyarov/newrelic_platform_go v0.0.0-20140908184405-b21fdbd4370f h1 github.com/yvasiyarov/newrelic_platform_go v0.0.0-20140908184405-b21fdbd4370f/go.mod h1:GlGEuHIJweS1mbCqG+7vt2nvWLzLLnRHbXz5JKd/Qbg= github.com/zalando/go-keyring v0.2.3-0.20230503081219-17db2e5354bd h1:D+eeEnOlWcMXbwZ5X3oy68nHafBtGcj1jMKFHtVdybY= github.com/zalando/go-keyring v0.2.3-0.20230503081219-17db2e5354bd/go.mod h1:sI3evg9Wvpw3+n4SqplGSJUMwtDeROfD4nsFz4z9PG0= +github.com/zclconf/go-cty v1.14.3 h1:1JXy1XroaGrzZuG6X9dt7HL6s9AwbY+l4UNL8o5B6ho= +github.com/zclconf/go-cty v1.14.3/go.mod h1:VvMs5i0vgZdhYawQNq5kePSpLAoz8u1xvZgrPIxfnZE= +github.com/zclconf/go-cty-yaml v1.0.3 h1:og/eOQ7lvA/WWhHGFETVWNduJM7Rjsv2RRpx1sdFMLc= +github.com/zclconf/go-cty-yaml v1.0.3/go.mod h1:9YLUH4g7lOhVWqUbctnVlZ5KLpg7JAprQNgxSZ1Gyxs= go.mongodb.org/mongo-driver v1.14.0 h1:P98w8egYRjYe3XDjxhYJagTokP/H6HzlsnojRgZRd80= go.mongodb.org/mongo-driver v1.14.0/go.mod h1:Vzb0Mk/pa7e6cWw85R4F/endUC3u0U9jGcNU603k65c= go.opencensus.io v0.21.0/go.mod h1:mSImk1erAIZhrmZN+AvHh14ztQfjbGwt4TtuofqLduU= diff --git a/mkdocs.yml b/mkdocs.yml index ea15a8c1..de411ec5 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -18,6 +18,7 @@ nav: - Getting Started: - Paths Overview: paths.md - Templating Funcs: templating_funcs.md + - HCL Funcs: hcl_funcs.md - Built-in Objects: builtin-objects.md - Advanced Features: - Best Practices Guide: writing-helmfile.md diff --git a/pkg/hcllang/hcl_functions.go b/pkg/hcllang/hcl_functions.go new file mode 100644 index 00000000..f3207122 --- /dev/null +++ b/pkg/hcllang/hcl_functions.go @@ -0,0 +1,82 @@ +package hcllang + +import ( + "github.com/hashicorp/hcl/v2/ext/tryfunc" + "github.com/imdario/mergo" + ctyyaml "github.com/zclconf/go-cty-yaml" + "github.com/zclconf/go-cty/cty/function" + "github.com/zclconf/go-cty/cty/function/stdlib" +) + +func HCLFunctions(additionnalFunctions map[string]function.Function) (map[string]function.Function, error) { + var hclFunctions = map[string]function.Function{ + "abs": stdlib.AbsoluteFunc, + "can": tryfunc.CanFunc, + "ceil": stdlib.CeilFunc, + "chomp": stdlib.ChompFunc, + "coalesce": stdlib.CoalesceFunc, + "coalescelist": stdlib.CoalesceListFunc, + "compact": stdlib.CompactFunc, + "concat": stdlib.ConcatFunc, + "contains": stdlib.ContainsFunc, + "csvdecode": stdlib.CSVDecodeFunc, + "distinct": stdlib.DistinctFunc, + "element": stdlib.ElementFunc, + "chunklist": stdlib.ChunklistFunc, + "flatten": stdlib.FlattenFunc, + "floor": stdlib.FloorFunc, + "format": stdlib.FormatFunc, + "formatdate": stdlib.FormatDateFunc, + "formatlist": stdlib.FormatListFunc, + "indent": stdlib.IndentFunc, + "int": stdlib.IntFunc, + "join": stdlib.JoinFunc, + "jsondecode": stdlib.JSONDecodeFunc, + "jsonencode": stdlib.JSONEncodeFunc, + "keys": stdlib.KeysFunc, + "length": stdlib.LengthFunc, + "log": stdlib.LogFunc, + "lookup": stdlib.LookupFunc, + "lower": stdlib.LowerFunc, + "max": stdlib.MaxFunc, + "merge": stdlib.MergeFunc, + "min": stdlib.MinFunc, + "parseint": stdlib.ParseIntFunc, + "pow": stdlib.PowFunc, + "range": stdlib.RangeFunc, + "regex": stdlib.RegexFunc, + "regexall": stdlib.RegexAllFunc, + "reverselist": stdlib.ReverseListFunc, + "setintersection": stdlib.SetIntersectionFunc, + "setproduct": stdlib.SetProductFunc, + "setsubtract": stdlib.SetSubtractFunc, + "setunion": stdlib.SetUnionFunc, + "signum": stdlib.SignumFunc, + "slice": stdlib.SliceFunc, + "sort": stdlib.SortFunc, + "split": stdlib.SplitFunc, + "strrev": stdlib.ReverseFunc, + "strlen": stdlib.StrlenFunc, + "substr": stdlib.SubstrFunc, + "timeadd": stdlib.TimeAddFunc, + "title": stdlib.TitleFunc, + "trim": stdlib.TrimFunc, + "trimprefix": stdlib.TrimPrefixFunc, + "trimspace": stdlib.TrimSpaceFunc, + "trimsuffix": stdlib.TrimSuffixFunc, + "try": tryfunc.TryFunc, + "upper": stdlib.UpperFunc, + "values": stdlib.ValuesFunc, + "yamldecode": ctyyaml.YAMLDecodeFunc, + "yamlencode": ctyyaml.YAMLEncodeFunc, + "zipmap": stdlib.ZipmapFunc, + } + + if additionnalFunctions != nil { + err := mergo.MapWithOverwrite(hclFunctions, additionnalFunctions) + if err != nil { + return nil, err + } + } + return hclFunctions, nil +} diff --git a/pkg/hcllang/hcl_loader.go b/pkg/hcllang/hcl_loader.go new file mode 100644 index 00000000..16d559f2 --- /dev/null +++ b/pkg/hcllang/hcl_loader.go @@ -0,0 +1,378 @@ +package hcllang + +import ( + nativejson "encoding/json" + "fmt" + "slices" + "strings" + + "github.com/hashicorp/hcl/v2" + "github.com/hashicorp/hcl/v2/hclparse" + "github.com/hashicorp/hcl/v2/hclsyntax" + "github.com/imdario/mergo" + "github.com/variantdev/dag/pkg/dag" + "github.com/zclconf/go-cty/cty" + "github.com/zclconf/go-cty/cty/json" + "go.uber.org/zap" + + "github.com/helmfile/helmfile/pkg/filesystem" +) + +const ( + badIdentifierDetail = "A name must start with a letter or underscore and may contain only letters, digits, underscores, and dashes." + ValuesBlockIdentifier = "values" + LocalsBlockIdentifier = "locals" + valuesAccessorPrefix = "hv" + localsAccessorPrefix = "local" +) + +// HelmfileHCLValue represents a single entry from a "values" or "locals" block file. +// The blocks itself is not represented, because it serves only to +// provide context for us to interpret its contents. +type HelmfileHCLValue struct { + Name string + Expr hcl.Expression + Range hcl.Range +} + +type HCLLoader struct { + hclFilesPath []string + fs *filesystem.FileSystem + logger *zap.SugaredLogger +} + +func NewHCLLoader(fs *filesystem.FileSystem, logger *zap.SugaredLogger) *HCLLoader { + return &HCLLoader{ + fs: fs, + logger: logger, + } +} +func (hl *HCLLoader) AddFile(file string) { + hl.hclFilesPath = append(hl.hclFilesPath, file) +} + +func (hl *HCLLoader) AddFiles(files []string) { + hl.hclFilesPath = append(hl.hclFilesPath, files...) +} + +func (hl *HCLLoader) Length() int { + return len(hl.hclFilesPath) +} + +func (hl *HCLLoader) HCLRender() (map[string]any, error) { + if hl.Length() == 0 { + return nil, fmt.Errorf("nothing to render") + } + + HelmfileHCLValues, locals, diags := hl.readHCLs() + if len(diags) > 0 { + return nil, diags.Errs()[0] + } + + // Decode all locals from all files first + // in order for them to be usable in values blocks + localsCty := map[string]map[string]cty.Value{} + for k, local := range locals { + dagPlan, err := hl.createDAGGraph(local, LocalsBlockIdentifier) + if err != nil { + return nil, err + } + localFileCty, err := hl.decodeGraph(dagPlan, LocalsBlockIdentifier, locals[k], nil) + if err != nil { + return nil, err + } + localsCty[k] = make(map[string]cty.Value) + localsCty[k][localsAccessorPrefix] = localFileCty[localsAccessorPrefix] + } + + // Decode Values + dagHelmfileValuePlan, err := hl.createDAGGraph(HelmfileHCLValues, ValuesBlockIdentifier) + if err != nil { + return nil, err + } + helmfileVarCty, err := hl.decodeGraph(dagHelmfileValuePlan, ValuesBlockIdentifier, HelmfileHCLValues, localsCty) + if err != nil { + return nil, err + } + nativeGovals, err := hl.convertToGo(helmfileVarCty) + if err != nil { + return nil, err + } + return nativeGovals, nil +} + +func (hl *HCLLoader) createDAGGraph(HelmfileHCLValues map[string]*HelmfileHCLValue, blockType string) (*dag.Topology, error) { + dagGraph := dag.New() + + for _, hv := range HelmfileHCLValues { + var traversals []string + for _, tr := range hv.Expr.Variables() { + attr, diags := hl.parseSingleAttrRef(tr, blockType) + if diags != nil { + return nil, fmt.Errorf("%s", diags.Errs()[0]) + } + if attr != "" && !slices.Contains(traversals, attr) { + traversals = append(traversals, attr) + } + } + hl.logger.Debugf("Adding Dependency : %s => [%s]", hv.Name, strings.Join(traversals, ", ")) + dagGraph.Add(hv.Name, dag.Dependencies(traversals)) + } + + //Generate Dag Plan which will provide the order from which to interpolate vars + plan, err := dagGraph.Plan(dag.SortOptions{ + WithDependencies: true, + }) + if err == nil { + return &plan, nil + } + + if ude, ok := err.(*dag.UndefinedDependencyError); ok { + var quotedVariableNames []string + for _, d := range ude.Dependents { + quotedVariableNames = append(quotedVariableNames, fmt.Sprintf("%q", d)) + } + return nil, fmt.Errorf("variables %s depend(s) on undefined vars %q", strings.Join(quotedVariableNames, ", "), ude.UndefinedNode) + } else { + return nil, fmt.Errorf("error while building the DAG variable graph : %s", err.Error()) + } +} + +func (hl *HCLLoader) decodeGraph(dagTopology *dag.Topology, blocktype string, vars map[string]*HelmfileHCLValue, additionalLocalContext map[string]map[string]cty.Value) (map[string]cty.Value, error) { + values := map[string]cty.Value{} + helmfileHCLValuesValues := map[string]cty.Value{} + var diags hcl.Diagnostics + hclFunctions, err := HCLFunctions(nil) + if err != nil { + return nil, err + } + for groupIndex := 0; groupIndex < len(*dagTopology); groupIndex++ { + dagNodesInGroup := (*dagTopology)[groupIndex] + + for _, node := range dagNodesInGroup { + v := vars[node.String()] + if blocktype != LocalsBlockIdentifier && additionalLocalContext[v.Range.Filename] != nil { + values[localsAccessorPrefix] = additionalLocalContext[v.Range.Filename][localsAccessorPrefix] + } + ctx := &hcl.EvalContext{ + Variables: values, + Functions: hclFunctions, + } + // Decode Value + helmfileHCLValuesValues[node.String()], diags = v.Expr.Value(ctx) + if len(diags) > 0 { + return nil, fmt.Errorf("error when trying to evaluate variable %s : %s", v.Name, diags.Errs()[0]) + } + switch blocktype { + case ValuesBlockIdentifier: + // Update the eval context for the next value evaluation iteration + values[valuesAccessorPrefix] = cty.ObjectVal(helmfileHCLValuesValues) + // Set back local to nil to avoid an unexpected behavior when the next iteration is in another file + values[localsAccessorPrefix] = cty.NilVal + case LocalsBlockIdentifier: + values[localsAccessorPrefix] = cty.ObjectVal(helmfileHCLValuesValues) + } + } + } + return values, nil +} + +func (hl *HCLLoader) readHCLs() (map[string]*HelmfileHCLValue, map[string]map[string]*HelmfileHCLValue, hcl.Diagnostics) { + var variables map[string]*HelmfileHCLValue + var local map[string]*HelmfileHCLValue + locals := map[string]map[string]*HelmfileHCLValue{} + var diags hcl.Diagnostics + for _, file := range hl.hclFilesPath { + variables, local, diags = hl.readHCL(variables, file) + if diags != nil { + return nil, nil, diags + } + locals[file] = make(map[string]*HelmfileHCLValue) + locals[file] = local + } + return variables, locals, nil +} + +func (hl *HCLLoader) readHCL(hvars map[string]*HelmfileHCLValue, file string) (map[string]*HelmfileHCLValue, map[string]*HelmfileHCLValue, hcl.Diagnostics) { + src, err := hl.fs.ReadFile(file) + if err != nil { + return nil, nil, hcl.Diagnostics{ + { + Severity: hcl.DiagError, + Summary: fmt.Sprintf("%s", err), + Detail: "could not read file", + Subject: &hcl.Range{}, + }, + } + } + + // Parse file as HCL + p := hclparse.NewParser() + hclFile, diags := p.ParseHCL(src, file) + if hclFile == nil || hclFile.Body == nil || diags != nil { + return nil, nil, diags + } + + HelmfileHCLValuesSchema := &hcl.BodySchema{ + Blocks: []hcl.BlockHeaderSchema{ + { + Type: ValuesBlockIdentifier, + }, + { + Type: LocalsBlockIdentifier, + }, + }, + } + // make sure content has a struct with helmfile_vars Schema defined + content, diags := hclFile.Body.Content(HelmfileHCLValuesSchema) + if diags != nil { + return nil, nil, diags + } + + var helmfileLocalsVars map[string]*HelmfileHCLValue + // Decode blocks to return HelmfileHCLValue object => (each var with expr + Name ) + + if len(content.Blocks.OfType(LocalsBlockIdentifier)) > 1 { + return nil, nil, hcl.Diagnostics{ + &hcl.Diagnostic{ + Severity: hcl.DiagError, + Summary: "A file can only support exactly 1 `locals` block", + Subject: &content.Blocks[0].DefRange, + }} + } + for _, block := range content.Blocks { + var helmfileBlockVars map[string]*HelmfileHCLValue + if block.Type == ValuesBlockIdentifier { + helmfileBlockVars, diags = hl.decodeHelmfileHCLValuesBlock(block) + if diags != nil { + return nil, nil, diags + } + } + + if block.Type == LocalsBlockIdentifier { + helmfileLocalsVars, diags = hl.decodeHelmfileHCLValuesBlock(block) + if diags != nil { + return nil, nil, diags + } + } + + // make sure vars are unique across blocks + for k := range helmfileBlockVars { + if hvars[k] != nil { + var diags hcl.Diagnostics + diags = append(diags, &hcl.Diagnostic{ + Severity: hcl.DiagError, + Summary: "Duplicate helmfile_vars definition", + Detail: fmt.Sprintf("The helmfile_var %q was already defined at %s:%d", + k, hvars[k].Range.Filename, hvars[k].Range.Start.Line), + Subject: &helmfileBlockVars[k].Range, + }) + return nil, nil, diags + } + } + err = mergo.Merge(&hvars, &helmfileBlockVars) + if err != nil { + var diags hcl.Diagnostics + diags = append(diags, &hcl.Diagnostic{ + Severity: hcl.DiagError, + Summary: "Merge failed", + Detail: err.Error(), + Subject: nil, + }) + return nil, nil, diags + } + } + + return hvars, helmfileLocalsVars, nil +} + +func (hl *HCLLoader) decodeHelmfileHCLValuesBlock(block *hcl.Block) (map[string]*HelmfileHCLValue, hcl.Diagnostics) { + attrs, diags := block.Body.JustAttributes() + if len(attrs) == 0 || diags != nil { + return nil, diags + } + + hfVars := map[string]*HelmfileHCLValue{} + for name, attr := range attrs { + if !hclsyntax.ValidIdentifier(name) { + diags = append(diags, &hcl.Diagnostic{ + Severity: hcl.DiagError, + Summary: "Invalid helmfile_vars variable name", + Detail: badIdentifierDetail, + Subject: &attr.NameRange, + }) + } + + hfVars[name] = &HelmfileHCLValue{ + Name: name, + Expr: attr.Expr, + Range: attr.Range, + } + } + return hfVars, diags +} + +func (hl *HCLLoader) parseSingleAttrRef(traversal hcl.Traversal, blockType string) (string, hcl.Diagnostics) { + if len(traversal) == 0 { + return "", hcl.Diagnostics{ + &hcl.Diagnostic{ + Severity: hcl.DiagError, + Summary: "An empty traversal can't be parsed", + }, + } + } + root := traversal.RootName() + // In `values` blocks, Locals are always precomputed, so they don't need to be in the graph + if root == localsAccessorPrefix && blockType != LocalsBlockIdentifier { + return "", nil + } + rootRange := traversal[0].SourceRange() + + if len(traversal) < 2 { + return "", hcl.Diagnostics{ + &hcl.Diagnostic{ + Severity: hcl.DiagError, + Summary: "Invalid reference", + Detail: fmt.Sprintf("The %q object cannot be accessed directly. Instead, access it from one of its root.", root), + Subject: &rootRange, + }, + } + } + + if attrTrav, ok := traversal[1].(hcl.TraverseAttr); ok { + return attrTrav.Name, nil + } + + return "", hcl.Diagnostics{ + &hcl.Diagnostic{ + Severity: hcl.DiagError, + Summary: "Invalid reference", + Detail: fmt.Sprintf("The %q object does not support this operation.", root), + Subject: traversal[1].SourceRange().Ptr(), + }, + } +} + +func (hl *HCLLoader) convertToGo(src map[string]cty.Value) (map[string]any, error) { + // Ugly workaround on value conversion + // CTY conversion to go natives requires much processing and complexity + // All of this, in our context, can go away because of the CTY capability to dump a cty.Value as Json + // The Json document outputs 2 keys : "type" and "value" which describe the mapping between the two + // We only care about the value + b, err := json.Marshal(src[valuesAccessorPrefix], cty.DynamicPseudoType) + if err != nil { + return nil, fmt.Errorf("could not marshal cty value : %s", err.Error()) + } + + var jsonunm map[string]any + err = nativejson.Unmarshal(b, &jsonunm) + if err != nil { + return nil, fmt.Errorf("could not unmarshall json : %s", err.Error()) + } + + if result, ok := jsonunm["value"].(map[string]any); ok { + return result, nil + } else { + return nil, fmt.Errorf("could extract a map object from json \"value\" key") + } +} diff --git a/pkg/hcllang/hcl_loader_test.go b/pkg/hcllang/hcl_loader_test.go new file mode 100644 index 00000000..2cce2aee --- /dev/null +++ b/pkg/hcllang/hcl_loader_test.go @@ -0,0 +1,163 @@ +package hcllang + +import ( + "io" + "testing" + + "github.com/google/go-cmp/cmp" + + ffs "github.com/helmfile/helmfile/pkg/filesystem" + "github.com/helmfile/helmfile/pkg/helmexec" +) + +func newHCLLoader() *HCLLoader { + log := helmexec.NewLogger(io.Discard, "debug") + return &HCLLoader{ + fs: ffs.DefaultFileSystem(), + logger: log, + } +} + +func TestHCL_localsTraversalsParser(t *testing.T) { + l := newHCLLoader() + files := []string{"testdata/values.1.hcl"} + l.AddFiles(files) + + _, filesLocals, diags := l.readHCLs() + if diags != nil { + t.Errorf("Test file parsing error : %s", diags.Errs()[0].Error()) + } + + actual := make(map[string]map[string]int) + for file, locals := range filesLocals { + actual[file] = make(map[string]int) + for k, v := range locals { + actual[file][k] = len(v.Expr.Variables()) + } + } + + expected := map[string]map[string]int{ + "testdata/values.1.hcl": { + "myLocal": 0, + "myLocalRef": 1, + }, + } + + if diff := cmp.Diff(expected, actual); diff != "" { + t.Errorf(diff) + } +} + +func TestHCL_localsTraversalsAttrParser(t *testing.T) { + l := newHCLLoader() + files := []string{"testdata/values.1.hcl"} + l.AddFiles(files) + + _, filesLocals, diags := l.readHCLs() + if diags != nil { + t.Errorf("Test file parsing error : %s", diags.Errs()[0].Error()) + } + + actual := make(map[string]map[string]string) + for file, locals := range filesLocals { + actual[file] = make(map[string]string) + for k, v := range locals { + str := "" + for _, v := range v.Expr.Variables() { + tr, _ := l.parseSingleAttrRef(v, LocalsBlockIdentifier) + str += tr + } + actual[file][k] = str + } + } + + expected := map[string]map[string]string{ + "testdata/values.1.hcl": { + "myLocal": "", + "myLocalRef": "myLocal", + }, + } + + if diff := cmp.Diff(expected, actual); diff != "" { + t.Errorf(diff) + } +} +func TestHCL_valuesTraversalsParser(t *testing.T) { + l := newHCLLoader() + files := []string{"testdata/values.1.hcl"} + l.AddFiles(files) + + fileValues, _, diags := l.readHCLs() + if diags != nil { + t.Errorf("Test file parsing error : %s", diags.Errs()[0].Error()) + } + + actual := make(map[string]int) + for k, v := range fileValues { + actual[k] = len(v.Expr.Variables()) + } + + expected := map[string]int{ + "val1": 0, + "val2": 1, + "val3": 2, + "val4": 1, + } + + if diff := cmp.Diff(expected, actual); diff != "" { + t.Errorf(diff) + } +} + +func TestHCL_valuesTraversalsAttrParser(t *testing.T) { + l := newHCLLoader() + files := []string{"testdata/values.1.hcl"} + l.AddFiles(files) + + fileValues, _, diags := l.readHCLs() + if diags != nil { + t.Errorf("Test file parsing error : %s", diags.Errs()[0].Error()) + } + + actual := make(map[string]string) + for k, v := range fileValues { + str := "" + for _, v := range v.Expr.Variables() { + tr, _ := l.parseSingleAttrRef(v, ValuesBlockIdentifier) + str += tr + } + actual[k] = str + } + + expected := map[string]string{ + "val1": "", + "val2": "", + "val3": "val1", + "val4": "val1", + } + + if diff := cmp.Diff(expected, actual); diff != "" { + t.Errorf(diff) + } +} + +func TestHCL_resultValidate(t *testing.T) { + l := newHCLLoader() + files := []string{"testdata/values.1.hcl"} + l.AddFiles(files) + actual, err := l.HCLRender() + if err != nil { + t.Errorf("Render error : %s", err.Error()) + } + + expected := map[string]any{ + "val1": float64(1), + "val2": "LOCAL", + "val3": "local1", + "val4": float64(-1), + } + + if diff := cmp.Diff(expected, actual); diff != "" { + t.Errorf(diff) + } +} diff --git a/pkg/hcllang/testdata/values.1.hcl b/pkg/hcllang/testdata/values.1.hcl new file mode 100644 index 00000000..66455ae9 --- /dev/null +++ b/pkg/hcllang/testdata/values.1.hcl @@ -0,0 +1,11 @@ +locals { + myLocal = "local" + myLocalRef = local.myLocal +} + +values { + val1 = 1 + val2 = upper(local.myLocal) + val3 = "${local.myLocal}${hv.val1}" + val4 = min(hv.val1, 10, -1) +} \ No newline at end of file diff --git a/pkg/helmexec/exec.go b/pkg/helmexec/exec.go index 3d76e049..111b2614 100644 --- a/pkg/helmexec/exec.go +++ b/pkg/helmexec/exec.go @@ -354,6 +354,22 @@ func (helm *execer) DecryptSecret(context HelmContext, name string, flags ...str return "", err } + // When the source encrypted file is not a yaml file AND helm secrets < 4 + // secrets plugin returns a yaml file with all the content in a yaml `data` key + // which isn't parsable from an hcl perspective + if strings.HasSuffix(name, ".hcl") && pluginVersion.Major() < 4 { + type helmSecretDataV3 struct { + Data string `yaml:"data"` + } + var data helmSecretDataV3 + err := yaml.Unmarshal(secretBytes, &data) + if err != nil { + return "", fmt.Errorf("Could not unmarshall helm secret plugin V3 decrypted file to a yaml string\n"+ + "You may consider upgrading your helm secrets plugin to >4.0.\n %s", err.Error()) + } + secretBytes = []byte(data.Data) + } + secret.bytes = secretBytes } else { // Cache hit diff --git a/pkg/state/create.go b/pkg/state/create.go index 361be7d7..f83337c6 100644 --- a/pkg/state/create.go +++ b/pkg/state/create.go @@ -4,6 +4,8 @@ import ( "errors" "fmt" "io" + "slices" + "strings" "github.com/helmfile/vals" "github.com/imdario/mergo" @@ -18,8 +20,9 @@ import ( ) const ( - DefaultHelmBinary = "helm" - DefaultKustomizeBinary = "kustomize" + DefaultHelmBinary = "helm" + DefaultKustomizeBinary = "kustomize" + DefaultHCLFileExtension = ".hcl" ) type StateLoadError struct { @@ -233,17 +236,19 @@ func (c *StateCreator) loadBases(envValues, overrodeEnv *environment.Environment // nolint: unparam func (c *StateCreator) loadEnvValues(st *HelmState, name string, failOnMissingEnv bool, ctxEnv, overrode *environment.Environment) (*environment.Environment, error) { - envVals := map[string]any{} + secretVals := map[string]any{} + valuesVals := map[string]any{} envSpec, ok := st.Environments[name] + decryptedFiles := []string{} if ok { var err error - envVals, err = st.loadValuesEntries(envSpec.MissingFileHandler, envSpec.Values, c.remote, ctxEnv, name) - if err != nil { - return nil, err - } - + // To keep supporting the secrets entries having precedence over the values + // This require to be done in 2 steps for HCL encrypted file support : + // 1. Get the Secrets + // 2. Merge the secrets with the envValues after + // Also makes the fail +- faster as it's trying to decrypt before loading values + var envSecretFiles []string if len(envSpec.Secrets) > 0 { - var envSecretFiles []string for _, urlOrPath := range envSpec.Secrets { resolved, skipped, err := st.storage().resolveFile(envSpec.MissingFileHandler, "environment values", urlOrPath, envSpec.MissingFileHandlerConfig.resolveFileOptions()...) if err != nil { @@ -252,18 +257,40 @@ func (c *StateCreator) loadEnvValues(st *HelmState, name string, failOnMissingEn if skipped { continue } - envSecretFiles = append(envSecretFiles, resolved...) } - if err = c.scatterGatherEnvSecretFiles(st, envSecretFiles, envVals); err != nil { + keepSecretFilesExtensions := []string{DefaultHCLFileExtension} + decryptedFiles, err = c.scatterGatherEnvSecretFiles(st, envSecretFiles, secretVals, keepSecretFilesExtensions) + if err != nil { return nil, err } + + defer func() { + for _, file := range decryptedFiles { + if err := c.fs.DeleteFile(file); err != nil { + c.logger.Warnf("failed removing decrypted file %s: %w", file, err) + } + } + }() + } + var valuesFiles []any + for _, f := range decryptedFiles { + valuesFiles = append(valuesFiles, f) + } + envValuesEntries := append(valuesFiles, envSpec.Values...) + valuesVals, err = st.loadValuesEntries(envSpec.MissingFileHandler, envValuesEntries, c.remote, ctxEnv, name) + if err != nil { + return nil, err + } + + if err = mergo.Merge(&valuesVals, &secretVals, mergo.WithOverride); err != nil { + return nil, err } } else if ctxEnv == nil && overrode == nil && name != DefaultEnv && failOnMissingEnv { return nil, &UndefinedEnvError{Env: name} } - newEnv := &environment.Environment{Name: name, Values: envVals, KubeContext: envSpec.KubeContext} + newEnv := &environment.Environment{Name: name, Values: valuesVals, KubeContext: envSpec.KubeContext} if ctxEnv != nil { intCtxEnv := *ctxEnv @@ -288,9 +315,14 @@ func (c *StateCreator) loadEnvValues(st *HelmState, name string, failOnMissingEn return newEnv, nil } -func (c *StateCreator) scatterGatherEnvSecretFiles(st *HelmState, envSecretFiles []string, envVals map[string]any) error { +// For all keepFileExtensions, the decrypted files will be retained +// with the specified extensions +// They will not be parsed nor added to the envVals. +// Only their decrypted filePath will be returned +// Up to the caller to remove them +func (c *StateCreator) scatterGatherEnvSecretFiles(st *HelmState, envSecretFiles []string, envVals map[string]any, keepFileExtensions []string) ([]string, error) { var errs []error - + var decryptedFilesKeeper []string helm := c.getHelm(st) inputs := envSecretFiles inputsSize := len(inputs) @@ -325,31 +357,39 @@ func (c *StateCreator) scatterGatherEnvSecretFiles(st *HelmState, envSecretFiles results <- secretResult{secret.id, nil, err, secret.path} continue } - + for _, ext := range keepFileExtensions { + if strings.HasSuffix(secret.path, ext) { + decryptedFilesKeeper = append(decryptedFilesKeeper, decFile) + } + } // nolint: staticcheck defer func() { - if err := c.fs.DeleteFile(decFile); err != nil { - c.logger.Warnf("removing decrypted file %s: %w", decFile, err) + if !slices.Contains(decryptedFilesKeeper, decFile) { + if err := c.fs.DeleteFile(decFile); err != nil { + c.logger.Warnf("removing decrypted file %s: %w", decFile, err) + } } }() - - bytes, err := c.fs.ReadFile(decFile) - if err != nil { - results <- secretResult{secret.id, nil, fmt.Errorf("failed to load environment secrets file \"%s\": %v", secret.path, err), secret.path} - continue - } - m := map[string]any{} - if err := yaml.Unmarshal(bytes, &m); err != nil { - results <- secretResult{secret.id, nil, fmt.Errorf("failed to load environment secrets file \"%s\": %v", secret.path, err), secret.path} - continue - } - // All the nested map key should be string. Otherwise we get strange errors due to that - // mergo or reflect is unable to merge map[any]any with map[string]any or vice versa. - // See https://github.com/roboll/helmfile/issues/677 - vals, err := maputil.CastKeysToStrings(m) - if err != nil { - results <- secretResult{secret.id, nil, fmt.Errorf("failed to load environment secrets file \"%s\": %v", secret.path, err), secret.path} - continue + var vals map[string]any + if !slices.Contains(decryptedFilesKeeper, decFile) { + bytes, err := c.fs.ReadFile(decFile) + if err != nil { + results <- secretResult{secret.id, nil, fmt.Errorf("failed to load environment secrets file \"%s\": %v", secret.path, err), secret.path} + continue + } + m := map[string]any{} + if err := yaml.Unmarshal(bytes, &m); err != nil { + results <- secretResult{secret.id, nil, fmt.Errorf("failed to load environment secrets file \"%s\": %v", secret.path, err), secret.path} + continue + } + // All the nested map key should be string. Otherwise we get strange errors due to that + // mergo or reflect is unable to merge map[any]any with map[string]any or vice versa. + // See https://github.com/roboll/helmfile/issues/677 + vals, err = maputil.CastKeysToStrings(m) + if err != nil { + results <- secretResult{secret.id, nil, fmt.Errorf("failed to load environment secrets file \"%s\": %v", secret.path, err), secret.path} + continue + } } results <- secretResult{secret.id, vals, nil, secret.path} } @@ -379,9 +419,9 @@ func (c *StateCreator) scatterGatherEnvSecretFiles(st *HelmState, envSecretFiles for _, err := range errs { st.logger.Error(err) } - return fmt.Errorf("failed loading environment secrets with %d errors", len(errs)) + return decryptedFilesKeeper, fmt.Errorf("failed loading environment secrets with %d errors", len(errs)) } - return nil + return decryptedFilesKeeper, nil } func (st *HelmState) loadValuesEntries(missingFileHandler *string, entries []any, remote *remote.Remote, ctxEnv *environment.Environment, envName string) (map[string]any, error) { diff --git a/pkg/state/envvals_loader.go b/pkg/state/envvals_loader.go index e5332937..882c3b23 100644 --- a/pkg/state/envvals_loader.go +++ b/pkg/state/envvals_loader.go @@ -3,12 +3,14 @@ package state import ( "fmt" "path/filepath" + "strings" "github.com/imdario/mergo" "go.uber.org/zap" "github.com/helmfile/helmfile/pkg/environment" "github.com/helmfile/helmfile/pkg/filesystem" + "github.com/helmfile/helmfile/pkg/hcllang" "github.com/helmfile/helmfile/pkg/maputil" "github.com/helmfile/helmfile/pkg/remote" "github.com/helmfile/helmfile/pkg/tmpl" @@ -35,7 +37,11 @@ func NewEnvironmentValuesLoader(storage *Storage, fs *filesystem.FileSystem, log } func (ld *EnvironmentValuesLoader) LoadEnvironmentValues(missingFileHandler *string, valuesEntries []any, ctxEnv *environment.Environment, envName string) (map[string]any, error) { - result := map[string]any{} + var ( + result = map[string]any{} + hclLoader = hcllang.NewHCLLoader(ld.fs, ld.logger) + err error + ) for _, entry := range valuesEntries { maps := []any{} @@ -49,7 +55,6 @@ func (ld *EnvironmentValuesLoader) LoadEnvironmentValues(missingFileHandler *str if skipped { continue } - for _, f := range files { var env environment.Environment if ctxEnv == nil { @@ -57,38 +62,61 @@ func (ld *EnvironmentValuesLoader) LoadEnvironmentValues(missingFileHandler *str } else { env = *ctxEnv } - - tmplData := NewEnvironmentTemplateData(env, "", map[string]any{}) - r := tmpl.NewFileRenderer(ld.fs, filepath.Dir(f), tmplData) - bytes, err := r.RenderToBytes(f) - if err != nil { - return nil, fmt.Errorf("failed to load environment values file \"%s\": %v", f, err) + if strings.HasSuffix(f, ".hcl") { + hclLoader.AddFile(f) + } else { + tmplData := NewEnvironmentTemplateData(env, "", map[string]any{}) + r := tmpl.NewFileRenderer(ld.fs, filepath.Dir(f), tmplData) + bytes, err := r.RenderToBytes(f) + if err != nil { + return nil, fmt.Errorf("failed to load environment values file \"%s\": %v", f, err) + } + m := map[string]any{} + if err := yaml.Unmarshal(bytes, &m); err != nil { + return nil, fmt.Errorf("failed to load environment values file \"%s\": %v\n\nOffending YAML:\n%s", f, err, bytes) + } + maps = append(maps, m) + ld.logger.Debugf("envvals_loader: loaded %s:%v", strOrMap, m) } - m := map[string]any{} - if err := yaml.Unmarshal(bytes, &m); err != nil { - return nil, fmt.Errorf("failed to load environment values file \"%s\": %v\n\nOffending YAML:\n%s", f, err, bytes) - } - maps = append(maps, m) - ld.logger.Debugf("envvals_loader: loaded %s:%v", strOrMap, m) } case map[any]any, map[string]any: maps = append(maps, strOrMap) default: return nil, fmt.Errorf("unexpected type of value: value=%v, type=%T", strOrMap, strOrMap) } - for _, m := range maps { - // All the nested map key should be string. Otherwise we get strange errors due to that - // mergo or reflect is unable to merge map[any]any with map[string]any or vice versa. - // See https://github.com/roboll/helmfile/issues/677 - vals, err := maputil.CastKeysToStrings(m) - if err != nil { - return nil, err - } - if err := mergo.Merge(&result, &vals, mergo.WithOverride); err != nil { - return nil, fmt.Errorf("failed to merge %v: %v", m, err) - } + + result, err = mapMerge(result, maps) + if err != nil { + return nil, err } } - + maps := []any{} + if hclLoader.Length() > 0 { + m, err := hclLoader.HCLRender() + if err != nil { + return nil, err + } + maps = append(maps, m) + } + result, err = mapMerge(result, maps) + if err != nil { + return nil, err + } return result, nil } + +func mapMerge(dest map[string]any, maps []any) (map[string]any, error) { + for _, m := range maps { + // All the nested map key should be string. Otherwise we get strange errors due to that + // mergo or reflect is unable to merge map[any]any with map[string]any or vice versa. + // See https://github.com/roboll/helmfile/issues/677 + vals, err := maputil.CastKeysToStrings(m) + if err != nil { + return nil, err + } + if err := mergo.Merge(&dest, &vals, mergo.WithOverride); err != nil { + return nil, fmt.Errorf("failed to merge %v: %v", m, err) + } + } + return dest, nil +} diff --git a/pkg/state/envvals_loader_test.go b/pkg/state/envvals_loader_test.go index 4a8fe349..b0f37719 100644 --- a/pkg/state/envvals_loader_test.go +++ b/pkg/state/envvals_loader_test.go @@ -187,3 +187,44 @@ func TestEnvValsLoad_OverwriteEmptyValue_Issue1168(t *testing.T) { t.Errorf(diff) } } + +func TestEnvValsLoad_MultiHCL(t *testing.T) { + l := newLoader() + + actual, err := l.LoadEnvironmentValues(nil, []any{"testdata/values.7.hcl", "testdata/values.8.hcl"}, nil, "") + if err != nil { + t.Fatal(err) + } + + expected := map[string]any{ + "a": "a", + "b": "b", + "c": "ab", + "map": map[string]any{ + "a": "a", + }, + "list": []any{ + "b", + }, + "nestedmap": map[string]any{ + "submap": map[string]any{ + "subsubmap": map[string]any{ + "hello": "ab", + }, + }, + }, + "ternary": true, + "fromMap": "aab", + "expressionInText": "yes", + "insideFor": "b", + "multi_block": "block", + "block": "block", + "crossfile": "crossfile var", + "crossfile_var": "crossfile var", + "localRef": "localInValues7", + } + + if diff := cmp.Diff(expected, actual); diff != "" { + t.Errorf(diff) + } +} diff --git a/pkg/state/testdata/values.7.hcl b/pkg/state/testdata/values.7.hcl new file mode 100644 index 00000000..685485bd --- /dev/null +++ b/pkg/state/testdata/values.7.hcl @@ -0,0 +1,42 @@ +locals { + localInValues7File = "localInValues7" +} + +values { + a = "a" + b = "b" + c = "${hv.a}${hv.b}" + + map = { + "a" = "${hv.a}" + } + + list = [ + hv.b + ] + + nestedmap = { + submap = { + subsubmap = { + hello = hv.c + } + } + } + + ternary = true ? true : false + + fromMap = "${hv.map.a}${hv.nestedmap.submap.subsubmap.hello}" + + expressionInText = "%{if hv.ternary }yes%{else}no%{endif}" + insideFor = "%{for i in hv.list }${i}%{endfor}" + + multi_block = hv.block + + crossfile = hv.crossfile_var + + localRef = local.localInValues7File +} + +values { + block = "block" +} \ No newline at end of file diff --git a/pkg/state/testdata/values.8.hcl b/pkg/state/testdata/values.8.hcl new file mode 100644 index 00000000..0c0c3425 --- /dev/null +++ b/pkg/state/testdata/values.8.hcl @@ -0,0 +1,3 @@ +values { + crossfile_var = "crossfile var" +} \ No newline at end of file diff --git a/test/e2e/template/helmfile/testdata/snapshot/hcl_mix/config.yaml b/test/e2e/template/helmfile/testdata/snapshot/hcl_mix/config.yaml new file mode 100644 index 00000000..162a6575 --- /dev/null +++ b/test/e2e/template/helmfile/testdata/snapshot/hcl_mix/config.yaml @@ -0,0 +1,4 @@ +helmfileArgs: +- template +- --concurrency=1 +- -q diff --git a/test/e2e/template/helmfile/testdata/snapshot/hcl_mix/input.yaml b/test/e2e/template/helmfile/testdata/snapshot/hcl_mix/input.yaml new file mode 100644 index 00000000..5500a57f --- /dev/null +++ b/test/e2e/template/helmfile/testdata/snapshot/hcl_mix/input.yaml @@ -0,0 +1,38 @@ +environments: + default: + values: + - values1.hcl + - values1.yaml + - values2.hcl +--- + +helmDefaults: + kubeContext: minikube + +releases: + + - name: hcl-demo + chart: ../../charts/raw-0.1.0 + values: + - templates: + - | + apiVersion: v1 + kind: ConfigMap + metadata: + name: configmap + data: + a: {{ .Values.a }} + b: {{ .Values.b }} + c: {{ .Values.c }} + crossfile: {{ .Values.crossfile }} + expressionInText: "{{ .Values.expressionInText }}" + fromMap: {{ .Values.fromMap }} + insideFor: {{ .Values.insideFor }} + list: "{{ .Values.list }}" + {{- .Values.locals | toYaml | nindent 10 -}} + multiBlock: {{ .Values.multiBlock }} + simpleCompute: "{{ .Values.simpleCompute }}" + yamlOverride: {{ .Values.yamlOverride }} + + + diff --git a/test/e2e/template/helmfile/testdata/snapshot/hcl_mix/output.yaml b/test/e2e/template/helmfile/testdata/snapshot/hcl_mix/output.yaml new file mode 100644 index 00000000..9cebeaef --- /dev/null +++ b/test/e2e/template/helmfile/testdata/snapshot/hcl_mix/output.yaml @@ -0,0 +1,20 @@ +--- +# Source: raw/templates/resources.yaml +apiVersion: v1 +kind: ConfigMap +metadata: + name: configmap +data: + a: a + b: b + c: ab + crossfile: values2.hcl + expressionInText: "yes" + fromMap: ab + insideFor: b + list: "[b]" + filename: values1.hcl + multiBlock: block + simpleCompute: "4" + yamlOverride: yaml_overrode + diff --git a/test/e2e/template/helmfile/testdata/snapshot/hcl_mix/values1.hcl b/test/e2e/template/helmfile/testdata/snapshot/hcl_mix/values1.hcl new file mode 100644 index 00000000..83c3a2ff --- /dev/null +++ b/test/e2e/template/helmfile/testdata/snapshot/hcl_mix/values1.hcl @@ -0,0 +1,53 @@ +locals { + filename = "values1.hcl" +} + +values { + a = "a" + b = "b" + c = "${hv.a}${hv.b}" + + list = [ + hv.b + ] + + locals = local + + // Maps and some functions + nestedmap = { + submap = { + subsubmap = { + hello = hv.a + } + } + } + + nestedmap2 = { + submap = { + subsubmap = { + hello = hv.c + } + } + } + + merged = merge(hv.nestedmap, hv.nestedmap2) + fromMap = "${hv.merged.submap.subsubmap.hello}" + + // precedence demo + yamlOverride = "yaml_overrode" + + + // Simple Expressions + ternary = true ? true : false + expressionInText = "%{if hv.ternary }yes%{else}no%{endif}" + insideFor = "%{for i in hv.list }${i}%{endfor}" + simpleCompute = 2 + 2 + + multiBlock = hv.block + + crossfile = hv.crossfile_var +} + +values { + block = "block" +} \ No newline at end of file diff --git a/test/e2e/template/helmfile/testdata/snapshot/hcl_mix/values1.yaml b/test/e2e/template/helmfile/testdata/snapshot/hcl_mix/values1.yaml new file mode 100644 index 00000000..f6c346b3 --- /dev/null +++ b/test/e2e/template/helmfile/testdata/snapshot/hcl_mix/values1.yaml @@ -0,0 +1 @@ +yamlOverride: "will_be_overriden" \ No newline at end of file diff --git a/test/e2e/template/helmfile/testdata/snapshot/hcl_mix/values2.hcl b/test/e2e/template/helmfile/testdata/snapshot/hcl_mix/values2.hcl new file mode 100644 index 00000000..7bf1b5df --- /dev/null +++ b/test/e2e/template/helmfile/testdata/snapshot/hcl_mix/values2.hcl @@ -0,0 +1,7 @@ +locals { + filename = "values2.hcl" +} + +values { + crossfile_var = local.filename +} \ No newline at end of file diff --git a/test/integration/run.sh b/test/integration/run.sh index 655f6049..896742e8 100755 --- a/test/integration/run.sh +++ b/test/integration/run.sh @@ -94,6 +94,7 @@ ${kubectl} create namespace ${test_ns} || fail "Could not create namespace ${tes . ${dir}/test-cases/chartify.sh . ${dir}/test-cases/deps-mr-1011.sh . ${dir}/test-cases/deps-kustomization-i-1402.sh +. ${dir}/test-cases/hcl-secrets.sh # ALL DONE ----------------------------------------------------------------------------------------------------------- diff --git a/test/integration/test-cases/hcl-secrets.sh b/test/integration/test-cases/hcl-secrets.sh new file mode 100644 index 00000000..3979dfd4 --- /dev/null +++ b/test/integration/test-cases/hcl-secrets.sh @@ -0,0 +1,28 @@ + +export VAULT_ADDR=http://127.0.0.1:8200 +export VAULT_TOKEN=toor +info "Inject sops key" +sops="sops --hc-vault-transit $VAULT_ADDR/v1/sops/keys/key" + +hcl_secrets_case_input_dir="${cases_dir}/hcl-secrets/input" +hcl_secrets_case_output_dir="${cases_dir}/hcl-secrets/output" + +mkdir -p ${hcl_secrets_case_input_dir}/tmp + +info "Encrypt secrets" +${sops} -e ${hcl_secrets_case_input_dir}/secrets.hcl > ${hcl_secrets_case_input_dir}/tmp/secrets.hcl || fail "${sops} failed at ${hcl_secrets_case_input_dir}/secrets.hcl" +${sops} -e ${hcl_secrets_case_input_dir}/secrets.yaml > ${hcl_secrets_case_input_dir}/tmp/secrets.yaml || fail "${sops} failed at ${hcl_secrets_case_input_dir}/secrets.yaml" + + +info "values precedence order : yamlFile < hcl = hclSecrets < secretYamlFile" +test_start "hcl-yaml-mix - should output secrets with proper overrides" + +hcl_secrets_tmp=$(mktemp -d) +result=${hcl_secrets_tmp}/result.yaml + +info "Building output" +${helmfile} -f ${hcl_secrets_case_input_dir}/_helmfile.yaml.gotmpl template --skip-deps > ${result} || fail "\"helmfile template\" shouldn't fail" + diff -u ${hcl_secrets_case_output_dir}/output.yaml ${result} || fail "helmdiff should be consistent" + echo code=$? + +test_pass "hcl-yaml-mix" \ No newline at end of file diff --git a/test/integration/test-cases/hcl-secrets/input/_helmfile.yaml.gotmpl b/test/integration/test-cases/hcl-secrets/input/_helmfile.yaml.gotmpl new file mode 100644 index 00000000..898bda53 --- /dev/null +++ b/test/integration/test-cases/hcl-secrets/input/_helmfile.yaml.gotmpl @@ -0,0 +1,28 @@ +environments: + default: + values: + - values.hcl + - values.yaml + secrets: + - tmp/secrets.hcl + - tmp/secrets.yaml +--- + +helmDefaults: + kubeContext: minikube + +releases: + + - name: hcl-demo + chart: ../../../charts/raw + values: + - templates: + - | + apiVersion: v1 + kind: ConfigMap + metadata: + name: configmap + data: + secretOveriddenByPrecedence: {{ .Values.secretOveriddenByPrecedence }} + secretRef: {{ .Values.secretref }} + yamlOverride: {{ .Values.yamlOverride }} diff --git a/test/integration/test-cases/hcl-secrets/input/secrets.hcl b/test/integration/test-cases/hcl-secrets/input/secrets.hcl new file mode 100644 index 00000000..4ed20529 --- /dev/null +++ b/test/integration/test-cases/hcl-secrets/input/secrets.hcl @@ -0,0 +1,3 @@ +values { + secret = "mySecretValue" +} \ No newline at end of file diff --git a/test/integration/test-cases/hcl-secrets/input/secrets.yaml b/test/integration/test-cases/hcl-secrets/input/secrets.yaml new file mode 100644 index 00000000..8d7797d3 --- /dev/null +++ b/test/integration/test-cases/hcl-secrets/input/secrets.yaml @@ -0,0 +1 @@ +secretOveriddenByPrecedence: OverrodeHclVarByPrecedence \ No newline at end of file diff --git a/test/integration/test-cases/hcl-secrets/input/values.hcl b/test/integration/test-cases/hcl-secrets/input/values.hcl new file mode 100644 index 00000000..aedb45d3 --- /dev/null +++ b/test/integration/test-cases/hcl-secrets/input/values.hcl @@ -0,0 +1,6 @@ +values { + // Secrets and precedence demo + secretref = upper(hv.secret) + yamlOverride = "yaml_overrode" + secretOveriddenByPrecedence = "will_be_overwrittten" +} diff --git a/test/integration/test-cases/hcl-secrets/input/values.yaml b/test/integration/test-cases/hcl-secrets/input/values.yaml new file mode 100644 index 00000000..f6c346b3 --- /dev/null +++ b/test/integration/test-cases/hcl-secrets/input/values.yaml @@ -0,0 +1 @@ +yamlOverride: "will_be_overriden" \ No newline at end of file diff --git a/test/integration/test-cases/hcl-secrets/output/output.yaml b/test/integration/test-cases/hcl-secrets/output/output.yaml new file mode 100644 index 00000000..31a76cb3 --- /dev/null +++ b/test/integration/test-cases/hcl-secrets/output/output.yaml @@ -0,0 +1,11 @@ +--- +# Source: raw/templates/resources.yaml +apiVersion: v1 +kind: ConfigMap +metadata: + name: configmap +data: + secretOveriddenByPrecedence: OverrodeHclVarByPrecedence + secretRef: MYSECRETVALUE + yamlOverride: yaml_overrode +