31 KiB

Raw Blame History

Values Merging and Data Flow

This document explains how Helmfile merges values from various sources and the overall data-flow architecture. Understanding this is essential for writing effective helmfiles.

Core Architecture Overview

Helmfile processes your configuration in a specific order, with values from different sources being merged together. The key concept is that later values override earlier values at the map level (deep merge), while arrays use smart merging (sparse auto-detection by default, with CLI overrides using element-by-element merging).

Values Sources and Precedence

Values in Helmfile come from multiple sources, merged in this order (lowest to highest priority):

┌─────────────────────────────────────────────────────────────────┐
│                    VALUES MERGING ORDER                         │
├─────────────────────────────────────────────────────────────────┤
│  1. Base files (from `bases:`)                                  │
│  2. Root-level `values:` block (Defaults)                       │
│  3. Environment values (yaml/yaml.gotmpl)                       │
│  4. Environment values (HCL, including HCL secrets)             │
│  5. Environment secrets (non-HCL, decrypted)                    │
│  6. CLI overrides (--state-values-set, --state-values-file)     │
└─────────────────────────────────────────────────────────────────┘
       ↓ Result: Template-accessible `.Values`

Important: Non-HCL secrets are loaded and decrypted first, but merged last (step 5), which means they have higher priority than regular environment values and will override duplicate keys.

1. Base Files (`bases:`)

Base files are merged first. They allow you to share common configuration across multiple helmfiles:

# common.yaml
environments:
  default:
  production:

helmDefaults:
  wait: true
  timeout: 300

# helmfile.yaml
bases:
  - common.yaml

releases:
  - name: myapp
    chart: mychart

2. Root-Level `values:` Block

The root-level values: block defines default values available to all templates in your helmfile. These are overridden by environment-specific values.

# Root-level values - available as {{ .Values.KEY }}
values:
  - appVersion: "1.0.0"
    region: "us-west-2"
    logLevel: "info"

environments:
  production:
    values:
      # This overrides the root-level logLevel
      - logLevel: "warning"

releases:
  - name: myapp
    chart: mychart
    values:
      - image:
          tag: {{ .Values.appVersion }}  # Uses "1.0.0"
        config:
          logLevel: {{ .Values.logLevel }}  # Uses "warning" in production

3. Environment Values

Environment values are loaded based on the selected environment (--environment flag, defaults to default):

environments:
  default:
    values:
      - env/default.yaml
      - env/common.yaml.gotmpl
  production:
    values:
      - env/production.yaml
      - replicas: 3  # Inline values are also supported

4. Environment Values Precedence

Within environment values, the precedence is:

YAML / YAML.gotmpl files (lowest)
HCL files (including HCL secrets)
Secrets files - non-HCL only (highest)

Technical Detail:

HCL secrets (.hcl files in secrets:) are decrypted and then processed in the HCL loading phase (step 2)
Non-HCL secrets (.yaml/.yaml.gotmpl files in secrets:) are decrypted and loaded first, but merged last (step 3), which is why they have the highest priority

environments:
  production:
    values:
      - config.yaml        # Loaded first (step 1)
      - config.hcl         # Loaded second (step 2)
    secrets:
      - secrets.yaml       # Merged last (step 3) - highest priority

4a. Merge Strategy: `override` vs `fallback`

By default, when an environment lists multiple files under values:, later files override earlier files (the historical helmfile behavior, equivalent to mergeStrategy: override).

You can flip this per environment so that earlier files take precedence and later files only fill in missing keys:

environments:
  production:
    mergeStrategy: fallback
    values:
      - cluster-specific.yaml   # wins on every key it defines
      - shared-defaults.yaml    # only fills gaps

Under fallback:

An explicit non-nil value in an earlier file is preserved against any later file, including the zero values false, 0, "", and empty list. An explicit enabled: false in cluster-specific.yaml is not silently overwritten by enabled: true from shared-defaults.yaml.
Maps are deep-merged. An earlier map does not block later files from adding nested keys it didn't set; only the keys an earlier file explicitly defines win on conflict.
An explicit null in an earlier file falls through to a later file's value, matching how MergeMaps treats nil from the override side elsewhere in helmfile.
Within a single values: entry that expands to multiple files (e.g. via a glob), the first file in the expansion wins.
A later .gotmpl values file can reference values from earlier files via .Values, so derived defaults work natively:
```
# cluster-specific.yaml
cluster:
  domain: prod.example.com
```
```
# shared-defaults.yaml.gotmpl
service:
  domain: "service.{{ .Values.cluster.domain }}"
```
→ service.domain: service.prod.example.com. Under mergeStrategy: override this cross-file template reference is not available.

Valid values: override (default) and fallback. Any other value is rejected at load time.

Interaction with `.hcl` values files

.hcl files are evaluated as a single unit so that HCL locals and values blocks can reference each other across files. Helmfile collects every .hcl entry from the values: list, renders them together, and merges the combined result after the YAML pass. Two consequences worth knowing:

mergeStrategy does not reshuffle the position of HCL within the list. HCL's combined output is always merged after the YAML pass. Under override it overrides YAML on conflicts (the historical behavior); under fallback it fills gaps only.
Among multiple .hcl files, the precedence is HCL's own (last-file-wins within HCL), independent of mergeStrategy. The strategy applies at the YAML-vs-HCL boundary, not within HCL.

If you need first-file-wins precedence between specific HCL files, restructure them into one HCL file (or split the values into YAML).

5. CLI Overrides

The highest priority values come from CLI flags:

# Override values from command line
helmfile --state-values-set image.tag=v2.0.0 sync

# Or from a file
helmfile --state-values-file overrides.yaml sync

How Merging Works

Deep Merge for Maps

Helmfile uses deep merge for map values. This means nested maps are merged recursively:

# Base values
values:
  - database:
      host: "localhost"
      port: 5432
      credentials:
        username: "admin"

# Environment override
environments:
  production:
    values:
      - database:
          host: "prod-db.example.com"
          credentials:
            password: "secret"  # Added

# Result in production:
# database:
#   host: "prod-db.example.com"    # Overridden
#   port: 5432                      # Preserved from base
#   credentials:
#     username: "admin"             # Preserved from base
#     password: "secret"            # Added from env

Array Merge Strategies

Helmfile uses different array merge strategies depending on the context:

1. Default Strategy (Sparse Auto-Detection)

For most values (state values, environment values), Helmfile uses sparse auto-detection:

If the override array contains nil values → merge element-by-element
If the override array has NO nil values → replace entirely

# Example with nil values (sparse array) - merges element-by-element
environments:
  production:
    values:
      - servers:
          - prod1.example.com  # index 0
          - null               # index 1 = nil, triggers merge mode
          - prod3.example.com  # index 2

# Result: merges with base array, preserving index 1 from base

# Example without nil values (complete array) - replaces entirely
environments:
  production:
    values:
      - servers:
          - prod1.example.com
          - prod2.example.com
          - prod3.example.com

# Result: completely replaces any base array

Important: An empty array [] has no nils, so it replaces entirely. This is intentional: explicitly setting an empty array clears the base array.

2. CLI Override Strategy (Element-by-Element Merge)

For --state-values-set CLI overrides, arrays are always merged element-by-element:

# This only changes index 0, preserves other indices
helmfile --state-values-set 'servers[0]=prod1.example.com' sync

3. Environment Values (Replace by Default)

Within environment values files (yaml/yaml.gotmpl/hcl), arrays use sparse auto-detection (strategy 1 above).

Practical Array Handling

Recommendation: To avoid confusion, treat arrays in one of these ways:

Complete replacement (default for most cases)

# Override array completely
environments:
  production:
    values:
      - servers: [prod1.example.com, prod2.example.com, prod3.example.com]

Sparse array merge (using explicit nil/null values)

# Merge specific array indices
environments:
  production:
    values:
      - servers:
          - prod1.example.com  # index 0: override
          - null                 # index 1: preserve from base
          - prod3.example.com    # index 2: add new

Use maps instead of arrays (recommended for complex configurations)

# Use maps for better control and merging
servers:
  server1:
    host: server1.example.com
    enabled: true
  server2:
    host: server2.example.com
    enabled: true

# Environment can add or override specific servers
environments:
  production:
    values:
      - servers:
          server3:
            host: prod3.example.com
            enabled: true

Release-Level Values

Each release can also define its own values, which are separate from the helmfile-level values discussed above:

# These are STATE values (accessible via {{ .Values }})
values:
  - globalSetting: "value"

environments:
  production:
    values:
      - envSetting: "prod-value"

releases:
  - name: myapp
    chart: mychart
    values:
      # These are RELEASE values (passed to Helm)
      - image:
          tag: {{ .Values.globalSetting }}  # Uses state value
      - values.yaml.gotmpl                  # Template file using state values
      - config.yaml                         # Static file

Release Values Merging

Within a release, values from different sources are merged in this order:

Values from release template (templates: with values:)
Inline values in the release
Values files listed in release
Values from valuesTemplate:
set and setString values

templates:
  myTemplate:
    values:
      - templateDefaults:
          option: "a"  # Lowest priority for this release

releases:
  - name: myapp
    chart: mychart
    inherit:
      - template: myTemplate
    values:
      - releaseDefaults:
          option: "b"    # Overrides template
      - env-specific.yaml # Can override above
    set:
      - name: option
        value: "c"       # Highest priority

Data Flow Diagram

Here's the complete data flow when running helmfile sync:

┌─────────────────────────────────────────────────────────────────────────────────────┐
│                         HELMFILE DATA FLOW                                  │
└─────────────────────────────────────────────────────────────────────────────────────┘

1. INITIALIZATION PHASE (app.go, desired_state_file_loader.go)
   ┌──────────────────┐
   │ Parse CLI flags │ ──> --state-values-set / --state-values-file
   └──────────────────┘
           │
           ▼
   ┌──────────────────────────────────────────────────────────────────────────┐
   │ Create Environment {                                               │
   │   Name: "default" or --environment value                        │
   │   Defaults: {} (will hold root-level values:)                  │
   │   Values: {} (will hold environment values + secrets)          │
   │   CLIOverrides: <parsed CLI values> (highest priority)    │
   │ }                                                                   │
   └──────────────────────────────────────────────────────────────────────────┘

2. LOAD PHASE (for each helmfile.yaml) (create.go: LoadEnvValues)
   ┌──────────────┐
   │ bases:        │ ──> Load and merge base files (if any)
   └──────────────┘
           │
           ▼
   ┌──────────────────────────────────────────────────────────────────────────┐
   │ values: (root-level)│ ──> Load via loadValuesEntries()                            │
   │                    │     Store in Environment.Defaults                              │
   └──────────────────────────────────────────────────────────────────────────┘
           │
           ▼
   ┌──────────────────────────────────────────────────────────────────────────┐
   │ environments:      │ ──> loadEnvValues() processes in order:                   │
   │                      │     1. Decrypt all secrets (HCL and non-HCL)                  │
   │                      │     2. Load environment values:                                 │
   │                      │        - yaml/yaml.gotmpl files (merged)                        │
   │                      │        - HCL files incl. decrypted HCL (merged)            │
   │                      │     3. Merge non-HCL decrypted secrets (highest priority)  │
   │                      │     Store result in Environment.Values                          │
   └──────────────────────────────────────────────────────────────────────────┘
           │
           ▼
   ┌───────────────────────────────────────────────────────────────────┐
   │ Update Environment from ctxEnv and overrodeEnv:                     │
   │   - Merge ctxEnv values (for multi-part helmfiles)                 │
   │   - Merge overrodeEnv (CLIOverrides already included here)         │
   └───────────────────────────────────────────────────────────────────┘

3. FINAL MERGE PHASE (environment.go: GetMergedValues)
   ┌───────────────────────────────────────────────────────────────────┐
   │ GetMergedValues() called when accessing .Values in templates:      │
   │   result = {}                                                        │
   │   result = merge(result, Defaults)      # Root-level values:        │
   │   result = merge(result, Values)        # Environment values+secrets│
   │   result = merge(result, CLIOverrides)  # CLI flags (highest priority)│
   │                                                                     │
   │ Special: CLIOverrides uses ArrayMergeStrategyMerge (element-by-element)│
   │         Defaults and Values use ArrayMergeStrategySparse (auto-detect) │
   └───────────────────────────────────────────────────────────────────┘
           │
           ▼
   ┌───────────────────────────────────────────────────────────────────┐
   │ state.RenderedValues = result                                    │
   │ Accessible via {{ .Values.KEY }} in all templates                 │
   └───────────────────────────────────────────────────────────────────┘

4. TEMPLATE RENDERING PHASE
   ┌──────────────────────────────────────────────┐
   │ Render helmfile.yaml templates             │
   │ Render .gotmpl values files                │
   │ Prepare release-specific values            │
   └──────────────────────────────────────────────┘

5. HELM EXECUTION PHASE
   ┌──────────────────────────────────────────────┐
   │ helm upgrade --install RELEASE CHART \       │
   │   -f /tmp/generated-values-xxx.yaml \        │
   │   --set key=value                            │
   └──────────────────────────────────────────────┘

Key Implementation Details:

CLI Overrides are parsed first (Load phase) and merged last (GetMergedValues)
Root-level values: are loaded into Environment.Defaults (not Values!)
Environment values + secrets are loaded into Environment.Values
Final merge order in GetMergedValues():
- Defaults → Values → CLIOverrides
Array merge strategies:
- Defaults & Values: ArrayMergeStrategySparse (auto-detect nil values)
- CLIOverrides: ArrayMergeStrategyMerge (always element-by-element)

Common Patterns

Pattern 1: Global Defaults with Environment Overrides

# Global defaults
values:
  - replicas: 1
    logLevel: "debug"
    resources:
      requests:
        cpu: "100m"
        memory: "128Mi"

# Environment-specific overrides
environments:
  production:
    values:
      - replicas: 3
        logLevel: "warning"
        resources:
          requests:
            cpu: "500m"
            memory: "512Mi"

Pattern 2: Shared Values Across Releases

values:
  - registry: "docker.io"
    imageTag: "latest"

releases:
  - name: frontend
    chart: charts/frontend
    values:
      - image:
          repository: {{ .Values.registry }}/frontend
          tag: {{ .Values.imageTag }}

  - name: backend
    chart: charts/backend
    values:
      - image:
          repository: {{ .Values.registry }}/backend
          tag: {{ .Values.imageTag }}

Pattern 3: Complex Nested Merging

values:
  - monitoring:
      enabled: true
      endpoints:
        health: "/health"
        metrics: "/metrics"
      alerts:
        email:
          enabled: true
          recipients:
            - ops@example.com

environments:
  production:
    values:
      - monitoring:
          alerts:
            slack:
              enabled: true
              channel: "#alerts"
            email:
              recipients:
                - ops@example.com
                - oncall@example.com

# Result in production:
# monitoring:
#   enabled: true
#   endpoints:
#     health: "/health"
#     metrics: "/metrics"
#   alerts:
#     email:
#       enabled: true
#       recipients: [ops@example.com, oncall@example.com]  # REPLACED!
#     slack:
#       enabled: true
#       channel: "#alerts"

Technical Implementation Details

Secret Handling

Helmfile processes secrets in a special way:

Non-HCL secrets (.yaml, .yaml.gotmpl):

Decrypted using helm-secrets plugin
Parsed into values immediately (during load phase)
Stored separately from regular values
Merged last (highest priority) after all environment values are loaded

HCL secrets (.hcl):

Decrypted using helm-secrets plugin
Decrypted file paths added to values file list
Processed in step 2 (HCL loading phase)
Can reference values from other HCL files (using hv. accessor)

This separation allows:

Secrets to override regular values without being re-decrypted multiple times
HCL secrets to participate in HCL's cross-file referencing system

Multiple Helmfiles and State files

When using multi-part helmfiles (multiple YAML documents separated by ---):

# Part 1: base.yaml
helmDefaults:
  wait: true
  timeout: 300
---
# Part 2: environments.yaml
environments:
  default:
  production:

Each part is processed in order and the results are merged with later parts taking precedence.

Environment Structure

The Environment struct is the core data structure that holds all values:

type Environment struct {
    Name         string
    KubeContext  string
    Defaults     map[string]any  // Root-level values: block (loaded via loadValuesEntries)
    Values       map[string]any  // Environment values + secrets (loaded in loadEnvValues)
    CLIOverrides map[string]any  // CLI --state-values-set (parsed in Load phase)
}

Key Insight: These three fields are kept separate until the final merge in GetMergedValues().

Merge Process Details

1. Loading Root-Level `values:` (create.go:172-179)

// In LoadEnvValues():
newDefaults, err := state.loadValuesEntries(nil, state.DefaultValues, c.remote, ctxEnv, env)
if err := nil {
    return nil, err
}

if err := mergo.Merge(&e.Defaults, newDefaults, mergo.WithOverride); err != nil {
    return nil, err
}

Root-level values: block is loaded into Environment.Defaults
Uses mergo.Merge with mergo.WithOverride option
This happens after environment values are loaded (line 172 comes after line 167)

2. Loading Environment Values (create.go:385-439)

// In loadEnvValues:
// Step 1: Decrypt non-HCL secrets first
decryptedFiles, err := c.scatterGatherEnvSecretFiles(st, envSecretFiles, secretVals, keepSecretFilesExtensions)

// Step 2: Load environment values (yaml/gotmpl + HCL + non-HCL secrets)
envValuesEntries := append(decryptedFiles, envSpec.Values...)  // Non-HCL secrets first, then env values
 valuesVals, err = st.loadValuesEntries(envSpec.MissingFileHandler, envValuesEntries, c.remote, loadValuesEntriesEnv, name)

// Step 3: Merge secrets into valuesVals (highest priority among env values)
if err = mergo.Merge(&valuesVals, &secretVals, mergo.WithOverride); err != nil {
    return nil, err
}

// Step 4: Create Environment with merged values
newEnv := &environment.Environment{
    Name: name, 
    Values: valuesVals,  // Contains: env values + HCL secrets + non-HCL secrets
    Defaults: map[string]any{}, 
    CLIOverrides: map[string]any{},
}

// Step 5: Merge from ctxEnv and overrodeEnv (for multi-part helmfiles and CLI overrides)
if ctxEnv != nil {
    newEnv.Defaults = maputil.MergeMaps(ctxEnv.Defaults, newEnv.Defaults)
    newEnv.Values = maputil.MergeMaps(ctxEnv.Values, newEnv.Values)
    newEnv.CLIOverrides = maputil.MergeMaps(newEnv.CLIOverrides, ctxEnv.CLIOverrides,
        maputil.MergeOptions{ArrayStrategy: maputil.ArrayMergeStrategyMerge})
}
if overrode != nil {
    newEnv.Defaults = maputil.MergeMaps(newEnv.Defaults, overrode.Defaults)
    newEnv.Values = maputil.MergeMaps(newEnv.Values, overrode.Values)
    newEnv.CLIOverrides = maputil.MergeMaps(newEnv.CLIOverrides, overrode.CLIOverrides,
        maputil.MergeOptions{ArrayStrategy: maputil.ArrayMergeStrategyMerge})
}

Key Points:

Non-HCL secrets are decrypted first, but merged last (highest priority)
HCL secrets are processed with HCL loader (can reference other HCL values)
CLI overrides come from overrodeEnv parameter
Multiple merge operations use mergo.Merge with mergo.WithOverride

3. Final Merge (environment.go:115-129)

// Called when accessing .Values in templates:
func (e *Environment) GetMergedValues() (map[string]any, error) {
    vals := map[string]any{}
    vals = maputil.MergeMaps(vals, e.Defaults)      // 1. Merge Defaults
    vals = maputil.MergeMaps(vals, e.Values)        // 2. Merge Values
    vals = maputil.MergeMaps(vals, e.CLIOverrides,  // 3. Merge CLIOverrides
        maputil.MergeOptions{ArrayStrategy: maputil.ArrayMergeStrategyMerge})
    
    vals, err := maputil.CastKeysToStrings(vals)
    if err != nil {
        return nil, err
    }
    return vals, nil
}

Key Points:

Defaults (root-level values:) are merged first (lowest priority)
Values (environment values + secrets) are merged second
CLIOverrides are merged last (highest priority)
CLIOverrides always use ArrayMergeStrategyMerge (element-by-element)
Defaults/Values use ArrayMergeStrategySparse (auto-detect nil values)

Deep Merge Behavior

Helmfile uses the dario.cat/mergo library with WithOverride option:

Maps:

if err := mergo.Merge(&dest, &src, mergo.WithOverride); err != nil {
    // handle error
}

This performs:

Deep merge: Nested maps are merged recursively
Override: Values from src override values in dest
Type preservation: Types are preserved during merge

Arrays (via maputil.MergeMaps):

func MergeMaps(a, b map[string]interface{}, opts ...MergeOptions) map[string]interface{} {
    arrayStrategy := ArrayMergeStrategySparse  // default
    
    // ... merging logic ...
    
    if vSlice != nil {  // Array detected
        if outSlice != nil {
            out[k] = mergeSlices(outSlice, vSlice, arrayStrategy)
            continue
        }
    }
}

Three Array Merge Strategies:

ArrayMergeStrategySparse (default for Defaults & Values)
- Auto-detects based on nil values in override array
- If any nil values: merge element-by-element (sparse)
- If no nil values: replace entire array (complete)
ArrayMergeStrategyReplace (not used in standard flow)
- Always replaces entire array
- Used for complete array replacement
ArrayMergeStrategyMerge (used for CLIOverrides)
- Always merges element-by-element
- Preserves base array elements unless explicitly overridden
- Perfect for CLI array[index]=value syntax

Loading Order vs Merge Order

Important Distinction:

Values Source	Loading Order	Merge Priority
CLI Overrides	Loaded first (init)	Merged last (highest)
Root values:	Loaded second	Merged first (lowest)
Environment values	Loaded third	Merged second
Environment secrets	Loaded fourth	Merged second (with values)

Why this matters:

CLI overrides are available early (for template rendering)
But they don't override until the final merge
This allows environment values to reference CLI overrides in templates
But final merge ensures CLI values win

Key Takeaways

Values merge deeply - nested maps are merged recursively, not replaced
Arrays use smart merging - by default, complete arrays replace, sparse arrays merge element-by-element
Later values win - the order of sources determines precedence
Separate concerns - state values (.Values) vs release values (passed to Helm)
Use templates - avoid repetition by using templates: and root-level values:
Secrets have high priority - non-HCL secrets are merged last, overriding environment values
Prefer maps over arrays - for configuration that needs incremental updates, use maps instead of arrays
CLI overrides are special - loaded first but merged last, always using element-by-element array merging

Troubleshooting

Value not being overridden

Check the precedence order. A value defined in a base file might be overridden by environment values. Remember that non-HCL secrets have the highest priority among environment values.

Unexpected array behavior

Arrays use sparse auto-detection by default:

Arrays with nil values: merged element-by-element
Arrays without nil values: replaced entirely
Empty arrays []: replace entirely (clears the base array)

To merge arrays, either:

Use null (nil) values for indices you want to preserve from the base
Convert to maps for better control
Use --state-values-set CLI flags for element-by-element updates

Can't access value in template

Ensure the value is defined in state values (not just in release values). State values are accessible via {{ .Values.KEY }}, while release values are only passed to Helm.

Values file not being templated

Only files with .gotmpl extension are templated. Regular .yaml files are used verbatim.

Secrets not overriding values

Non-HCL secrets are merged last in the environment values loading process, giving them the highest priority among environment values. If your secrets aren't overriding, check:

The secrets file is properly encrypted
The secrets are listed in the secrets: section (not values:)
You're using the correct helm-secrets plugin

CLI overrides not working

CLI --state-values-set uses element-by-element array merging, which is different from file-based values. This means:

# This merges at index 0, preserving other indices
helmfile --state-values-set 'servers[0]=prod1.example.com' sync

# This replaces the entire array
helmfile --state-values-file <(echo 'servers: [prod1.example.com]') sync

31 KiB Raw Blame History