31 KiB
Values Merging and Data Flow
This document explains how Helmfile merges values from various sources and the overall data-flow architecture. Understanding this is essential for writing effective helmfiles.
Core Architecture Overview
Helmfile processes your configuration in a specific order, with values from different sources being merged together. The key concept is that later values override earlier values at the map level (deep merge), while arrays use smart merging (sparse auto-detection by default, with CLI overrides using element-by-element merging).
Values Sources and Precedence
Values in Helmfile come from multiple sources, merged in this order (lowest to highest priority):
┌─────────────────────────────────────────────────────────────────┐
│ VALUES MERGING ORDER │
├─────────────────────────────────────────────────────────────────┤
│ 1. Base files (from `bases:`) │
│ 2. Root-level `values:` block (Defaults) │
│ 3. Environment values (yaml/yaml.gotmpl) │
│ 4. Environment values (HCL, including HCL secrets) │
│ 5. Environment secrets (non-HCL, decrypted) │
│ 6. CLI overrides (--state-values-set, --state-values-file) │
└─────────────────────────────────────────────────────────────────┘
↓ Result: Template-accessible `.Values`
Important: Non-HCL secrets are loaded and decrypted first, but merged last (step 5), which means they have higher priority than regular environment values and will override duplicate keys.
1. Base Files (bases:)
Base files are merged first. They allow you to share common configuration across multiple helmfiles:
# common.yaml
environments:
default:
production:
helmDefaults:
wait: true
timeout: 300
# helmfile.yaml
bases:
- common.yaml
releases:
- name: myapp
chart: mychart
2. Root-Level values: Block
The root-level values: block defines default values available to all templates in your helmfile. These are overridden by environment-specific values.
# Root-level values - available as {{ .Values.KEY }}
values:
- appVersion: "1.0.0"
region: "us-west-2"
logLevel: "info"
environments:
production:
values:
# This overrides the root-level logLevel
- logLevel: "warning"
releases:
- name: myapp
chart: mychart
values:
- image:
tag: {{ .Values.appVersion }} # Uses "1.0.0"
config:
logLevel: {{ .Values.logLevel }} # Uses "warning" in production
3. Environment Values
Environment values are loaded based on the selected environment (--environment flag, defaults to default):
environments:
default:
values:
- env/default.yaml
- env/common.yaml.gotmpl
production:
values:
- env/production.yaml
- replicas: 3 # Inline values are also supported
4. Environment Values Precedence
Within environment values, the precedence is:
- YAML / YAML.gotmpl files (lowest)
- HCL files (including HCL secrets)
- Secrets files - non-HCL only (highest)
Technical Detail:
- HCL secrets (.hcl files in
secrets:) are decrypted and then processed in the HCL loading phase (step 2) - Non-HCL secrets (.yaml/.yaml.gotmpl files in
secrets:) are decrypted and loaded first, but merged last (step 3), which is why they have the highest priority
environments:
production:
values:
- config.yaml # Loaded first (step 1)
- config.hcl # Loaded second (step 2)
secrets:
- secrets.yaml # Merged last (step 3) - highest priority
5. CLI Overrides
The highest priority values come from CLI flags:
# Override values from command line
helmfile --state-values-set image.tag=v2.0.0 sync
# Or from a file
helmfile --state-values-file overrides.yaml sync
How Merging Works
Deep Merge for Maps
Helmfile uses deep merge for map values. This means nested maps are merged recursively:
# Base values
values:
- database:
host: "localhost"
port: 5432
credentials:
username: "admin"
# Environment override
environments:
production:
values:
- database:
host: "prod-db.example.com"
credentials:
password: "secret" # Added
# Result in production:
# database:
# host: "prod-db.example.com" # Overridden
# port: 5432 # Preserved from base
# credentials:
# username: "admin" # Preserved from base
# password: "secret" # Added from env
Array Merge Strategies
Helmfile uses different array merge strategies depending on the context:
1. Default Strategy (Sparse Auto-Detection)
For most values (state values, environment values), Helmfile uses sparse auto-detection:
- If the override array contains
nilvalues → merge element-by-element - If the override array has NO
nilvalues → replace entirely
# Example with nil values (sparse array) - merges element-by-element
environments:
production:
values:
- servers:
- prod1.example.com # index 0
- null # index 1 = nil, triggers merge mode
- prod3.example.com # index 2
# Result: merges with base array, preserving index 1 from base
# Example without nil values (complete array) - replaces entirely
environments:
production:
values:
- servers:
- prod1.example.com
- prod2.example.com
- prod3.example.com
# Result: completely replaces any base array
Important: An empty array [] has no nils, so it replaces entirely. This is intentional: explicitly setting an empty array clears the base array.
2. CLI Override Strategy (Element-by-Element Merge)
For --state-values-set CLI overrides, arrays are always merged element-by-element:
# This only changes index 0, preserves other indices
helmfile --state-values-set 'servers[0]=prod1.example.com' sync
3. Environment Values (Replace by Default)
Within environment values files (yaml/yaml.gotmpl/hcl), arrays use sparse auto-detection (strategy 1 above).
Practical Array Handling
Recommendation: To avoid confusion, treat arrays in one of these ways:
-
Complete replacement (default for most cases)
# Override array completely environments: production: values: - servers: [prod1.example.com, prod2.example.com, prod3.example.com] -
Sparse array merge (using explicit nil/null values)
# Merge specific array indices environments: production: values: - servers: - prod1.example.com # index 0: override - null # index 1: preserve from base - prod3.example.com # index 2: add new -
Use maps instead of arrays (recommended for complex configurations)
# Use maps for better control and merging servers: server1: host: server1.example.com enabled: true server2: host: server2.example.com enabled: true # Environment can add or override specific servers environments: production: values: - servers: server3: host: prod3.example.com enabled: true
Release-Level Values
Each release can also define its own values, which are separate from the helmfile-level values discussed above:
# These are STATE values (accessible via {{ .Values }})
values:
- globalSetting: "value"
environments:
production:
values:
- envSetting: "prod-value"
releases:
- name: myapp
chart: mychart
values:
# These are RELEASE values (passed to Helm)
- image:
tag: {{ .Values.globalSetting }} # Uses state value
- values.yaml.gotmpl # Template file using state values
- config.yaml # Static file
Release Values Merging
Within a release, values from different sources are merged in this order:
- Values from release template (
templates:withvalues:) - Inline values in the release
- Values files listed in release
- Values from
valuesTemplate: setandsetStringvalues
templates:
myTemplate:
values:
- templateDefaults:
option: "a" # Lowest priority for this release
releases:
- name: myapp
chart: mychart
inherit:
- template: myTemplate
values:
- releaseDefaults:
option: "b" # Overrides template
- env-specific.yaml # Can override above
set:
- name: option
value: "c" # Highest priority
Data Flow Diagram
Here's the complete data flow when running helmfile sync:
┌─────────────────────────────────────────────────────────────────────────────────────┐
│ HELMFILE DATA FLOW │
└─────────────────────────────────────────────────────────────────────────────────────┘
1. INITIALIZATION PHASE (app.go, desired_state_file_loader.go)
┌──────────────────┐
│ Parse CLI flags │ ──> --state-values-set / --state-values-file
└──────────────────┘
│
▼
┌──────────────────────────────────────────────────────────────────────────┐
│ Create Environment { │
│ Name: "default" or --environment value │
│ Defaults: {} (will hold root-level values:) │
│ Values: {} (will hold environment values + secrets) │
│ CLIOverrides: <parsed CLI values> (highest priority) │
│ } │
└──────────────────────────────────────────────────────────────────────────┘
2. LOAD PHASE (for each helmfile.yaml) (create.go: LoadEnvValues)
┌──────────────┐
│ bases: │ ──> Load and merge base files (if any)
└──────────────┘
│
▼
┌──────────────────────────────────────────────────────────────────────────┐
│ values: (root-level)│ ──> Load via loadValuesEntries() │
│ │ Store in Environment.Defaults │
└──────────────────────────────────────────────────────────────────────────┘
│
▼
┌──────────────────────────────────────────────────────────────────────────┐
│ environments: │ ──> loadEnvValues() processes in order: │
│ │ 1. Decrypt all secrets (HCL and non-HCL) │
│ │ 2. Load environment values: │
│ │ - yaml/yaml.gotmpl files (merged) │
│ │ - HCL files incl. decrypted HCL (merged) │
│ │ 3. Merge non-HCL decrypted secrets (highest priority) │
│ │ Store result in Environment.Values │
└──────────────────────────────────────────────────────────────────────────┘
│
▼
┌───────────────────────────────────────────────────────────────────┐
│ Update Environment from ctxEnv and overrodeEnv: │
│ - Merge ctxEnv values (for multi-part helmfiles) │
│ - Merge overrodeEnv (CLIOverrides already included here) │
└───────────────────────────────────────────────────────────────────┘
3. FINAL MERGE PHASE (environment.go: GetMergedValues)
┌───────────────────────────────────────────────────────────────────┐
│ GetMergedValues() called when accessing .Values in templates: │
│ result = {} │
│ result = merge(result, Defaults) # Root-level values: │
│ result = merge(result, Values) # Environment values+secrets│
│ result = merge(result, CLIOverrides) # CLI flags (highest priority)│
│ │
│ Special: CLIOverrides uses ArrayMergeStrategyMerge (element-by-element)│
│ Defaults and Values use ArrayMergeStrategySparse (auto-detect) │
└───────────────────────────────────────────────────────────────────┘
│
▼
┌───────────────────────────────────────────────────────────────────┐
│ state.RenderedValues = result │
│ Accessible via {{ .Values.KEY }} in all templates │
└───────────────────────────────────────────────────────────────────┘
4. TEMPLATE RENDERING PHASE
┌──────────────────────────────────────────────┐
│ Render helmfile.yaml templates │
│ Render .gotmpl values files │
│ Prepare release-specific values │
└──────────────────────────────────────────────┘
5. HELM EXECUTION PHASE
┌──────────────────────────────────────────────┐
│ helm upgrade --install RELEASE CHART \ │
│ -f /tmp/generated-values-xxx.yaml \ │
│ --set key=value │
└──────────────────────────────────────────────┘
Key Implementation Details:
- CLI Overrides are parsed first (Load phase) and merged last (GetMergedValues)
- Root-level
values:are loaded intoEnvironment.Defaults(not Values!) - Environment values + secrets are loaded into
Environment.Values - Final merge order in
GetMergedValues():- Defaults → Values → CLIOverrides
- Array merge strategies:
- Defaults & Values:
ArrayMergeStrategySparse(auto-detect nil values) - CLIOverrides:
ArrayMergeStrategyMerge(always element-by-element)
- Defaults & Values:
Technical Details
Secret Handling Intern
Helmfile processes secrets in a special way:
Non-HCL secrets (.yaml, .yaml.gotmpl):
- Decrypted using helm-secrets plugin
- Parsed into values immediately (during load phase)
- Stored separately from regular values
- Mrged last (highest priority) after all environment values are loaded
HCL secrets (.hcl):
- Decrypted using helm-secrets plugin
- Decrypted file paths added to values file list
- Processed in step 2 (HCL loading phase)
- Can reference values from other HCL files (using
hv.accessor)
This separation allows:
- Secrets to override regular values without being re-decrypted multiple times
- HCL secrets to participate in HCL's cross-file referencing system
Multiple Helmfiles and State files
When using multi-part helmfiles (multiple YAML documents separated by ---):
# Part 1: base.yaml
helmDefaults:
wait: true
timeout: 300
---
# Part 2: environments.yaml
environments:
default:
production:
Each part is processed in order: and the results are merged with later parts taking precedence.
Technical Details
Environment Structure Internals
The Environment struct has three key fields that affect merging:
type Environment struct {
Name string
KubeContext string
Values map[string]any // Environment values + secrets
Defaults map[string]any // Root-level values: block
CLIOverrides map[string]any // CLI --state-values-set
}
Final Merge Process (GetMergedValues)
When you access .Values in templates, Helmfile calls GetMergedValues() which merges in this order:
func (e *Environment) GetMergedValues() (map[string]any, error) {
vals := map[string]any{}
vals = maputil.MergeMaps(vals, e.Defaults) // 1. Defaults (root-level values:)
vals = maputil.MergeMaps(vals, e.Values) // 2. Values (environment values + secrets)
vals = maputil.MergeMaps(vals, e.CLIOverrides, // 3. CLIOverrides (highest priority)
maputil.MergeOptions{ArrayStrategy: maputil.ArrayMergeStrategyMerge})
return vals, nil
}
Important: CLIOverrides uses ArrayMergeStrategyMerge (element-by-element merging), while Defaults and Values use the default strategy (sparse auto-detection).
Merging Library: mergo
Helmfile uses the mergo library for deep merging with these key features:
- Deep merge for maps: Nested maps are merged recursively
- WithOverride option: Later values override earlier values
- Type-safe: Preserves value types during merge
Example from code:
// In loadEnvValues()
if err := mergo.Merge(&valuesVals, &secretVals, mergo.WithOverride); err != nil {
return nil, err
}
Array Merge Strategies Implementation
The maputil.MergeMaps function supports three array merge strategies:
type ArrayMergeStrategy int
const (
ArrayMergeStrategySparse ArrayMergeStrategy = iota // Auto-detect based on nil values
ArrayMergeStrategyReplace ArrayMergeStrategy = iota // Always replace arrays
ArrayMergeStrategyMerge ArrayMergeStrategy = iota // Always merge element-by-element
)
Sparse Strategy (Default for most cases):
func mergeSlices(base, override []any, strategy ArrayMergeStrategy) []any {
if strategy == ArrayMergeStrategySparse {
isSparse := false
for _, v := range override {
if v == nil {
isSparse = true
break
}
}
if !isSparse {
return override // Replace entirely
}
// Otherwise merge element-by-element
}
}
This means:
[1, 2, 3]merged with[4, 5]→ result:[4, 5](replaced, no nils)[null, 2]merged with[1, 2, 3]→ result:[1, 2, 3](merged, has nil)
Common Patterns
Pattern 1: Global Defaults with Environment Overrides
# Global defaults
values:
- replicas: 1
logLevel: "debug"
resources:
requests:
cpu: "100m"
memory: "128Mi"
# Environment-specific overrides
environments:
production:
values:
- replicas: 3
logLevel: "warning"
resources:
requests:
cpu: "500m"
memory: "512Mi"
Pattern 2: Shared Values Across Releases
values:
- registry: "docker.io"
imageTag: "latest"
releases:
- name: frontend
chart: charts/frontend
values:
- image:
repository: {{ .Values.registry }}/frontend
tag: {{ .Values.imageTag }}
- name: backend
chart: charts/backend
values:
- image:
repository: {{ .Values.registry }}/backend
tag: {{ .Values.imageTag }}
Pattern 3: Complex Nested Merging
values:
- monitoring:
enabled: true
endpoints:
health: "/health"
metrics: "/metrics"
alerts:
email:
enabled: true
recipients:
- ops@example.com
environments:
production:
values:
- monitoring:
alerts:
slack:
enabled: true
channel: "#alerts"
email:
recipients:
- ops@example.com
- oncall@example.com
# Result in production:
# monitoring:
# enabled: true
# endpoints:
# health: "/health"
# metrics: "/metrics"
# alerts:
# email:
# enabled: true
# recipients: [ops@example.com, oncall@example.com] # REPLACED!
# slack:
# enabled: true
# channel: "#alerts"
Technical Implementation Details
Environment Structure
The Environment struct is the core data structure that holds all values:
type Environment struct {
Name string
KubeContext string
Defaults map[string]any // Root-level values: block (loaded via loadValuesEntries)
Values map[string]any // Environment values + secrets (loaded in loadEnvValues)
CLIOverrides map[string]any // CLI --state-values-set (parsed in Load phase)
}
Key Insight: These three fields are kept separate until the final merge in GetMergedValues().
Merge Process Details
1. Loading Root-Level values: (create.go:172-179)
// In LoadEnvValues():
newDefaults, err := state.loadValuesEntries(nil, state.DefaultValues, c.remote, ctxEnv, env)
if err := nil {
return nil, err
}
if err := mergo.Merge(&e.Defaults, newDefaults, mergo.WithOverride); err != nil {
return nil, err
}
- Root-level
values:block is loaded intoEnvironment.Defaults - Uses
mergo.Mergewithmergo.WithOverrideoption - This happens after environment values are loaded (line 172 comes after line 167)
2. Loading Environment Values (create.go:385-439)
// In loadEnvValues:
// Step 1: Decrypt non-HCL secrets first
decryptedFiles, err := c.scatterGatherEnvSecretFiles(st, envSecretFiles, secretVals, keepSecretFilesExtensions)
// Step 2: Load environment values (yaml/gotmpl + HCL + non-HCL secrets)
envValuesEntries := append(decryptedFiles, envSpec.Values...) // Non-HCL secrets first, then env values
valuesVals, err = st.loadValuesEntries(envSpec.MissingFileHandler, envValuesEntries, c.remote, loadValuesEntriesEnv, name)
// Step 3: Merge secrets into valuesVals (highest priority among env values)
if err = mergo.Merge(&valuesVals, &secretVals, mergo.WithOverride); err != nil {
return nil, err
}
// Step 4: Create Environment with merged values
newEnv := &environment.Environment{
Name: name,
Values: valuesVals, // Contains: env values + HCL secrets + non-HCL secrets
Defaults: map[string]any{},
CLIOverrides: map[string]any{},
}
// Step 5: Merge from ctxEnv and overrodeEnv (for multi-part helmfiles and CLI overrides)
if ctxEnv != nil {
newEnv.Defaults = maputil.MergeMaps(ctxEnv.Defaults, newEnv.Defaults)
newEnv.Values = maputil.MergeMaps(ctxEnv.Values, newEnv.Values)
newEnv.CLIOverrides = maputil.MergeMaps(newEnv.CLIOverrides, ctxEnv.CLIOverrides,
maputil.MergeOptions{ArrayStrategy: maputil.ArrayMergeStrategyMerge})
}
if overrode != nil {
newEnv.Defaults = maputil.MergeMaps(newEnv.Defaults, overrode.Defaults)
newEnv.Values = maputil.MergeMaps(newEnv.Values, overrode.Values)
newEnv.CLIOverrides = maputil.MergeMaps(newEnv.CLIOverrides, overrode.CLIOverrides,
maputil.MergeOptions{ArrayStrategy: maputil.ArrayMergeStrategyMerge})
}
Key Points:
- Non-HCL secrets are decrypted first, but merged last (highest priority)
- HCL secrets are processed with HCL loader (can reference other HCL values)
- CLI overrides come from
overrodeEnvparameter - Multiple merge operations use
mergo.Mergewithmergo.WithOverride
3. Final Merge (environment.go:115-129)
// Called when accessing .Values in templates:
func (e *Environment) GetMergedValues() (map[string]any, error) {
vals := map[string]any{}
vals = maputil.MergeMaps(vals, e.Defaults) // 1. Merge Defaults
vals = maputil.MergeMaps(vals, e.Values) // 2. Merge Values
vals = maputil.MergeMaps(vals, e.CLIOverrides, // 3. Merge CLIOverrides
maputil.MergeOptions{ArrayStrategy: maputil.ArrayMergeStrategyMerge})
vals, err := maputil.CastKeysToStrings(vals)
if err != nil {
return nil, err
}
return vals, nil
}
Key Points:
- Defaults (root-level values:) are merged first (lowest priority)
- Values (environment values + secrets) are merged second
- CLIOverrides are merged last (highest priority)
- CLIOverrides always use
ArrayMergeStrategyMerge(element-by-element) - Defaults/Values use
ArrayMergeStrategySparse(auto-detect nil values)
Deep Merge Behavior
Helmfile uses the dario.cat/mergo library with WithOverride option:
Maps:
if err := mergo.Merge(&dest, &src, mergo.WithOverride); err != nil {
// handle error
}
This performs:
- Deep merge: Nested maps are merged recursively
- Override: Values from
srcoverride values indest - Type preservation: Types are preserved during merge
Arrays (via maputil.MergeMaps):
func MergeMaps(a, b map[string]interface{}, opts ...MergeOptions) map[string]interface{} {
arrayStrategy := ArrayMergeStrategySparse // default
// ... merging logic ...
if vSlice != nil { // Array detected
if outSlice != nil {
out[k] = mergeSlices(outSlice, vSlice, arrayStrategy)
continue
}
}
}
Three Array Merge Strategies:
-
ArrayMergeStrategySparse (default for Defaults & Values)
- Auto-detects based on nil values in override array
- If any nil values: merge element-by-element (sparse)
- If no nil values: replace entire array (complete)
-
ArrayMergeStrategyReplace (not used in standard flow)
- Always replaces entire array
- Used for complete array replacement
-
ArrayMergeStrategyMerge (used for CLIOverrides)
- Always merges element-by-element
- Preserves base array elements unless explicitly overridden
- Perfect for CLI
array[index]=valuesyntax
Loading Order vs Merge Order
Important Distinction:
| Values Source | Loading Order | Merge Priority |
|---|---|---|
| CLI Overrides | Loaded first (init) | Merged last (highest) |
| Root values: | Loaded second | Merged first (lowest) |
| Environment values | Loaded third | Merged second |
| Environment secrets | Loaded fourth | Merged second (with values) |
Why this matters:
- CLI overrides are available early (for template rendering)
- But they don't override until the final merge
- This allows environment values to reference CLI overrides in templates
- But final merge ensures CLI values win
Key Takeaways
- Values merge deeply - nested maps are merged recursively, not replaced
- Arrays use smart merging - by default, complete arrays replace, sparse arrays merge element-by-element
- Later values win - the order of sources determines precedence
- Separate concerns - state values (
.Values) vs release values (passed to Helm) - Use templates - avoid repetition by using
templates:and root-levelvalues: - Secrets have high priority - non-HCL secrets are merged last, overriding environment values
- Prefer maps over arrays - for configuration that needs incremental updates, use maps instead of arrays
- CLI overrides are special - loaded first but merged last, always using element-by-element array merging
Troubleshooting
Value not being overridden
Check the precedence order. A value defined in a base file might be overridden by environment values. Remember that non-HCL secrets have the highest priority among environment values.
Unexpected array behavior
Arrays use sparse auto-detection by default:
- Arrays with
nilvalues: merged element-by-element - Arrays without
nilvalues: replaced entirely - Empty arrays
[]: replace entirely (clears the base array)
To merge arrays, either:
- Use
null(nil) values for indices you want to preserve from the base - Convert to maps for better control
- Use
--state-values-setCLI flags for element-by-element updates
Can't access value in template
Ensure the value is defined in state values (not just in release values). State values are accessible via {{ .Values.KEY }}, while release values are only passed to Helm.
Values file not being templated
Only files with .gotmpl extension are templated. Regular .yaml files are used verbatim.
Secrets not overriding values
Non-HCL secrets are merged last in the environment values loading process, giving them the highest priority among environment values. If your secrets aren't overriding, check:
- The secrets file is properly encrypted
- The secrets are listed in the
secrets:section (notvalues:) - You're using the correct helm-secrets plugin
CLI overrides not working
CLI --state-values-set uses element-by-element array merging, which is different from file-based values. This means:
# This merges at index 0, preserving other indices
helmfile --state-values-set 'servers[0]=prod1.example.com' sync
# This replaces the entire array
helmfile --state-values-file <(echo 'servers: [prod1.example.com]') sync