From 4f6a83079e5ba8df9dd046f309a3512e9063daba Mon Sep 17 00:00:00 2001 From: yxxhero Date: Thu, 5 Mar 2026 22:11:16 +0800 Subject: [PATCH] docs: add comprehensive values merging and data flow guide Add new documentation explaining how Helmfile merges values from various sources: - Core architecture and data flow - Values sources and precedence order - Deep merge behavior for maps and arrays - Environment-specific value handling - Secret management and priorities - Common patterns and troubleshooting This guide helps users understand the foundational concepts needed for writing effective helmfiles, especially regarding value overrides and merge strategies. Signed-off-by: yxxhero --- docs/index.md | 1 + docs/values-and-merging.md | 861 +++++++++++++++++++++++++++++++++++++ docs/writing-helmfile.md | 4 +- mkdocs.yml | 2 + 4 files changed, 867 insertions(+), 1 deletion(-) create mode 100644 docs/values-and-merging.md diff --git a/docs/index.md b/docs/index.md index c553226f..1aeb0a9d 100644 --- a/docs/index.md +++ b/docs/index.md @@ -116,6 +116,7 @@ Iterate on the `helmfile.yaml` by referencing: * [Configuration](#configuration) * [CLI reference](#cli-reference). * [Helmfile Best Practices Guide](writing-helmfile.md) +* [Values Merging and Data Flow](values-and-merging.md) - Understanding how Helmfile merges values ## Configuration diff --git a/docs/values-and-merging.md b/docs/values-and-merging.md new file mode 100644 index 00000000..8c7e6832 --- /dev/null +++ b/docs/values-and-merging.md @@ -0,0 +1,861 @@ +# Values Merging and Data Flow + +This document explains how Helmfile merges values from various sources and the overall data-flow architecture. Understanding this is essential for writing effective helmfiles. + +## Core Architecture Overview + +Helmfile processes your configuration in a specific order, with values from different sources being merged together. The key concept is that **later values override earlier values** at the map level (deep merge), while **arrays use smart merging** (sparse auto-detection by default, with CLI overrides using element-by-element merging). + +## Values Sources and Precedence + +Values in Helmfile come from multiple sources, merged in this order (lowest to highest priority): + +``` +┌─────────────────────────────────────────────────────────────────┐ +│ VALUES MERGING ORDER │ +├─────────────────────────────────────────────────────────────────┤ +│ 1. Base files (from `bases:`) │ +│ 2. Root-level `values:` block (Defaults) │ +│ 3. Environment values (yaml/yaml.gotmpl) │ +│ 4. Environment values (HCL, including HCL secrets) │ +│ 5. Environment secrets (non-HCL, decrypted) │ +│ 6. CLI overrides (--state-values-set, --state-values-file) │ +└─────────────────────────────────────────────────────────────────┘ + ↓ Result: Template-accessible `.Values` +``` + +**Important:** Non-HCL secrets are loaded and decrypted first, but merged last (step 5), which means they have higher priority than regular environment values and will override duplicate keys. + +### 1. Base Files (`bases:`) + +Base files are merged first. They allow you to share common configuration across multiple helmfiles: + +```yaml +# common.yaml +environments: + default: + production: + +helmDefaults: + wait: true + timeout: 300 +``` + +```yaml +# helmfile.yaml +bases: + - common.yaml + +releases: + - name: myapp + chart: mychart +``` + +### 2. Root-Level `values:` Block + +The root-level `values:` block defines default values available to all templates in your helmfile. These are overridden by environment-specific values. + +```yaml +# Root-level values - available as {{ .Values.KEY }} +values: + - appVersion: "1.0.0" + region: "us-west-2" + logLevel: "info" + +environments: + production: + values: + # This overrides the root-level logLevel + - logLevel: "warning" + +releases: + - name: myapp + chart: mychart + values: + - image: + tag: {{ .Values.appVersion }} # Uses "1.0.0" + config: + logLevel: {{ .Values.logLevel }} # Uses "warning" in production +``` + +### 3. Environment Values + +Environment values are loaded based on the selected environment (`--environment` flag, defaults to `default`): + +```yaml +environments: + default: + values: + - env/default.yaml + - env/common.yaml.gotmpl + production: + values: + - env/production.yaml + - replicas: 3 # Inline values are also supported +``` + +### 4. Environment Values Precedence + +Within environment values, the precedence is: + +1. YAML / YAML.gotmpl files (lowest) +2. HCL files (including HCL secrets) +3. Secrets files - non-HCL only (highest) + +**Technical Detail:** +- **HCL secrets** (.hcl files in `secrets:`) are decrypted and then processed in the HCL loading phase (step 2) +- **Non-HCL secrets** (.yaml/.yaml.gotmpl files in `secrets:`) are decrypted and loaded first, but merged last (step 3), which is why they have the highest priority + +```yaml +environments: + production: + values: + - config.yaml # Loaded first (step 1) + - config.hcl # Loaded second (step 2) + secrets: + - secrets.yaml # Merged last (step 3) - highest priority +``` + +### 5. CLI Overrides + +The highest priority values come from CLI flags: + +```bash +# Override values from command line +helmfile --state-values-set image.tag=v2.0.0 sync + +# Or from a file +helmfile --state-values-file overrides.yaml sync +``` + +## How Merging Works + +### Deep Merge for Maps + +Helmfile uses **deep merge** for map values. This means nested maps are merged recursively: + +```yaml +# Base values +values: + - database: + host: "localhost" + port: 5432 + credentials: + username: "admin" + +# Environment override +environments: + production: + values: + - database: + host: "prod-db.example.com" + credentials: + password: "secret" # Added + +# Result in production: +# database: +# host: "prod-db.example.com" # Overridden +# port: 5432 # Preserved from base +# credentials: +# username: "admin" # Preserved from base +# password: "secret" # Added from env +``` + +### Array Merge Strategies + +Helmfile uses different array merge strategies depending on the context: + +#### 1. Default Strategy (Sparse Auto-Detection) + +For most values (state values, environment values), Helmfile uses **sparse auto-detection**: + +- **If the override array contains `nil` values** → merge element-by-element +- **If the override array has NO `nil` values** → replace entirely + +```yaml +# Example with nil values (sparse array) - merges element-by-element +environments: + production: + values: + - servers: + - prod1.example.com # index 0 + - null # index 1 = nil, triggers merge mode + - prod3.example.com # index 2 + +# Result: merges with base array, preserving index 1 from base +``` + +```yaml +# Example without nil values (complete array) - replaces entirely +environments: + production: + values: + - servers: + - prod1.example.com + - prod2.example.com + - prod3.example.com + +# Result: completely replaces any base array +``` + +**Important:** An empty array `[]` has no nils, so it **replaces entirely**. This is intentional: explicitly setting an empty array clears the base array. + +#### 2. CLI Override Strategy (Element-by-Element Merge) + +For `--state-values-set` CLI overrides, arrays are **always merged element-by-element**: + +```bash +# This only changes index 0, preserves other indices +helmfile --state-values-set 'servers[0]=prod1.example.com' sync +``` + +#### 3. Environment Values (Replace by Default) + +Within environment values files (yaml/yaml.gotmpl/hcl), arrays use sparse auto-detection (strategy 1 above). + +### Practical Array Handling + +**Recommendation:** To avoid confusion, treat arrays in one of these ways: + +1. **Complete replacement** (default for most cases) + ```yaml + # Override array completely + environments: + production: + values: + - servers: [prod1.example.com, prod2.example.com, prod3.example.com] + ``` + +2. **Sparse array merge** (using explicit nil/null values) + ```yaml + # Merge specific array indices + environments: + production: + values: + - servers: + - prod1.example.com # index 0: override + - null # index 1: preserve from base + - prod3.example.com # index 2: add new + ``` + +3. **Use maps instead of arrays** (recommended for complex configurations) + ```yaml + # Use maps for better control and merging + servers: + server1: + host: server1.example.com + enabled: true + server2: + host: server2.example.com + enabled: true + + # Environment can add or override specific servers + environments: + production: + values: + - servers: + server3: + host: prod3.example.com + enabled: true + ``` + +## Release-Level Values + +Each release can also define its own values, which are **separate** from the helmfile-level values discussed above: + +```yaml +# These are STATE values (accessible via {{ .Values }}) +values: + - globalSetting: "value" + +environments: + production: + values: + - envSetting: "prod-value" + +releases: + - name: myapp + chart: mychart + values: + # These are RELEASE values (passed to Helm) + - image: + tag: {{ .Values.globalSetting }} # Uses state value + - values.yaml.gotmpl # Template file using state values + - config.yaml # Static file +``` + +### Release Values Merging + +Within a release, values from different sources are merged in this order: + +1. Values from release template (`templates:` with `values:`) +2. Inline values in the release +3. Values files listed in release +4. Values from `valuesTemplate:` +5. `set` and `setString` values + +```yaml +templates: + myTemplate: + values: + - templateDefaults: + option: "a" # Lowest priority for this release + +releases: + - name: myapp + chart: mychart + inherit: + - template: myTemplate + values: + - releaseDefaults: + option: "b" # Overrides template + - env-specific.yaml # Can override above + set: + - name: option + value: "c" # Highest priority +``` + +## Data Flow Diagram + +Here's the complete data flow when running `helmfile sync`: + +``` +┌─────────────────────────────────────────────────────────────────────────────────────┐ +│ HELMFILE DATA FLOW │ +└─────────────────────────────────────────────────────────────────────────────────────┘ + +1. INITIALIZATION PHASE (app.go, desired_state_file_loader.go) + ┌──────────────────┐ + │ Parse CLI flags │ ──> --state-values-set / --state-values-file + └──────────────────┘ + │ + ▼ + ┌──────────────────────────────────────────────────────────────────────────┐ + │ Create Environment { │ + │ Name: "default" or --environment value │ + │ Defaults: {} (will hold root-level values:) │ + │ Values: {} (will hold environment values + secrets) │ + │ CLIOverrides: (highest priority) │ + │ } │ + └──────────────────────────────────────────────────────────────────────────┘ + +2. LOAD PHASE (for each helmfile.yaml) (create.go: LoadEnvValues) + ┌──────────────┐ + │ bases: │ ──> Load and merge base files (if any) + └──────────────┘ + │ + ▼ + ┌──────────────────────────────────────────────────────────────────────────┐ + │ values: (root-level)│ ──> Load via loadValuesEntries() │ + │ │ Store in Environment.Defaults │ + └──────────────────────────────────────────────────────────────────────────┘ + │ + ▼ + ┌──────────────────────────────────────────────────────────────────────────┐ + │ environments: │ ──> loadEnvValues() processes in order: │ + │ │ 1. Decrypt all secrets (HCL and non-HCL) │ + │ │ 2. Load environment values: │ + │ │ - yaml/yaml.gotmpl files (merged) │ + │ │ - HCL files incl. decrypted HCL (merged) │ + │ │ 3. Merge non-HCL decrypted secrets (highest priority) │ + │ │ Store result in Environment.Values │ + └──────────────────────────────────────────────────────────────────────────┘ + │ + ▼ + ┌───────────────────────────────────────────────────────────────────┐ + │ Update Environment from ctxEnv and overrodeEnv: │ + │ - Merge ctxEnv values (for multi-part helmfiles) │ + │ - Merge overrodeEnv (CLIOverrides already included here) │ + └───────────────────────────────────────────────────────────────────┘ + +3. FINAL MERGE PHASE (environment.go: GetMergedValues) + ┌───────────────────────────────────────────────────────────────────┐ + │ GetMergedValues() called when accessing .Values in templates: │ + │ result = {} │ + │ result = merge(result, Defaults) # Root-level values: │ + │ result = merge(result, Values) # Environment values+secrets│ + │ result = merge(result, CLIOverrides) # CLI flags (highest priority)│ + │ │ + │ Special: CLIOverrides uses ArrayMergeStrategyMerge (element-by-element)│ + │ Defaults and Values use ArrayMergeStrategySparse (auto-detect) │ + └───────────────────────────────────────────────────────────────────┘ + │ + ▼ + ┌───────────────────────────────────────────────────────────────────┐ + │ state.RenderedValues = result │ + │ Accessible via {{ .Values.KEY }} in all templates │ + └───────────────────────────────────────────────────────────────────┘ + +4. TEMPLATE RENDERING PHASE + ┌──────────────────────────────────────────────┐ + │ Render helmfile.yaml templates │ + │ Render .gotmpl values files │ + │ Prepare release-specific values │ + └──────────────────────────────────────────────┘ + +5. HELM EXECUTION PHASE + ┌──────────────────────────────────────────────┐ + │ helm upgrade --install RELEASE CHART \ │ + │ -f /tmp/generated-values-xxx.yaml \ │ + │ --set key=value │ + └──────────────────────────────────────────────┘ +``` + +**Key Implementation Details:** + +1. **CLI Overrides** are parsed first (Load phase) and merged last (GetMergedValues) +2. **Root-level `values:`** are loaded into `Environment.Defaults` (not Values!) +3. **Environment values + secrets** are loaded into `Environment.Values` +4. **Final merge order** in `GetMergedValues()`: + - Defaults → Values → CLIOverrides +5. **Array merge strategies**: + - Defaults & Values: `ArrayMergeStrategySparse` (auto-detect nil values) + - CLIOverrides: `ArrayMergeStrategyMerge` (always element-by-element) + +## Technical Details + +### Secret Handling Intern + +Helmfile processes secrets in a special way: + +**Non-HCL secrets (.yaml, .yaml.gotmpl):** +1. Decrypted using helm-secrets plugin +2. Parsed into values immediately (during load phase) +3. Stored separately from regular values +4. **Mrged last** (highest priority) after all environment values are loaded + +**HCL secrets (.hcl):** +1. Decrypted using helm-secrets plugin +2. Decrypted file paths added to values file list +3. Processed in step 2 (HCL loading phase) +4. Can reference values from other HCL files (using `hv.` accessor) + +This separation allows: +- Secrets to override regular values without being re-decrypted multiple times +- HCL secrets to participate in HCL's cross-file referencing system + +### Multiple Helmfiles and State files + +When using multi-part helmfiles (multiple YAML documents separated by `---`): + +```yaml +# Part 1: base.yaml +helmDefaults: + wait: true + timeout: 300 +--- +# Part 2: environments.yaml +environments: + default: + production: +``` + +Each part is processed in order: + and the results are merged with later parts taking precedence. + +## Technical Details + +### Environment Structure Internals + +The `Environment` struct has three key fields that affect merging: + +```go +type Environment struct { + Name string + KubeContext string + Values map[string]any // Environment values + secrets + Defaults map[string]any // Root-level values: block + CLIOverrides map[string]any // CLI --state-values-set +} +``` + +### Final Merge Process (GetMergedValues) + +When you access `.Values` in templates, Helmfile calls `GetMergedValues()` which merges in this order: + +```go +func (e *Environment) GetMergedValues() (map[string]any, error) { + vals := map[string]any{} + vals = maputil.MergeMaps(vals, e.Defaults) // 1. Defaults (root-level values:) + vals = maputil.MergeMaps(vals, e.Values) // 2. Values (environment values + secrets) + vals = maputil.MergeMaps(vals, e.CLIOverrides, // 3. CLIOverrides (highest priority) + maputil.MergeOptions{ArrayStrategy: maputil.ArrayMergeStrategyMerge}) + return vals, nil +} +``` + +**Important:** CLIOverrides uses `ArrayMergeStrategyMerge` (element-by-element merging), while Defaults and Values use the default strategy (sparse auto-detection). + +### Merging Library: mergo + +Helmfile uses the [mergo](https://github.com/imdario/mergo) library for deep merging with these key features: + +1. **Deep merge for maps**: Nested maps are merged recursively +2. **WithOverride option**: Later values override earlier values +3. **Type-safe**: Preserves value types during merge + +Example from code: +```go +// In loadEnvValues() +if err := mergo.Merge(&valuesVals, &secretVals, mergo.WithOverride); err != nil { + return nil, err +} +``` + +### Array Merge Strategies Implementation + +The `maputil.MergeMaps` function supports three array merge strategies: + +```go +type ArrayMergeStrategy int + +const ( + ArrayMergeStrategySparse ArrayMergeStrategy = iota // Auto-detect based on nil values + ArrayMergeStrategyReplace ArrayMergeStrategy = iota // Always replace arrays + ArrayMergeStrategyMerge ArrayMergeStrategy = iota // Always merge element-by-element +) +``` + +**Sparse Strategy (Default for most cases):** +```go +func mergeSlices(base, override []any, strategy ArrayMergeStrategy) []any { + if strategy == ArrayMergeStrategySparse { + isSparse := false + for _, v := range override { + if v == nil { + isSparse = true + break + } + } + if !isSparse { + return override // Replace entirely + } + // Otherwise merge element-by-element + } +} +``` + +This means: +- `[1, 2, 3]` merged with `[4, 5]` → result: `[4, 5]` (replaced, no nils) +- `[null, 2]` merged with `[1, 2, 3]` → result: `[1, 2, 3]` (merged, has nil) + +## Common Patterns + +### Pattern 1: Global Defaults with Environment Overrides + +```yaml +# Global defaults +values: + - replicas: 1 + logLevel: "debug" + resources: + requests: + cpu: "100m" + memory: "128Mi" + +# Environment-specific overrides +environments: + production: + values: + - replicas: 3 + logLevel: "warning" + resources: + requests: + cpu: "500m" + memory: "512Mi" +``` + +### Pattern 2: Shared Values Across Releases + +```yaml +values: + - registry: "docker.io" + imageTag: "latest" + +releases: + - name: frontend + chart: charts/frontend + values: + - image: + repository: {{ .Values.registry }}/frontend + tag: {{ .Values.imageTag }} + + - name: backend + chart: charts/backend + values: + - image: + repository: {{ .Values.registry }}/backend + tag: {{ .Values.imageTag }} +``` + +### Pattern 3: Complex Nested Merging + +```yaml +values: + - monitoring: + enabled: true + endpoints: + health: "/health" + metrics: "/metrics" + alerts: + email: + enabled: true + recipients: + - ops@example.com + +environments: + production: + values: + - monitoring: + alerts: + slack: + enabled: true + channel: "#alerts" + email: + recipients: + - ops@example.com + - oncall@example.com + +# Result in production: +# monitoring: +# enabled: true +# endpoints: +# health: "/health" +# metrics: "/metrics" +# alerts: +# email: +# enabled: true +# recipients: [ops@example.com, oncall@example.com] # REPLACED! +# slack: +# enabled: true +# channel: "#alerts" +``` + +## Technical Implementation Details + +### Environment Structure + +The `Environment` struct is the core data structure that holds all values: + +```go +type Environment struct { + Name string + KubeContext string + Defaults map[string]any // Root-level values: block (loaded via loadValuesEntries) + Values map[string]any // Environment values + secrets (loaded in loadEnvValues) + CLIOverrides map[string]any // CLI --state-values-set (parsed in Load phase) +} +``` + +**Key Insight:** These three fields are kept **separate** until the final merge in `GetMergedValues()`. + +### Merge Process Details + +#### 1. Loading Root-Level `values:` (create.go:172-179) + +```go +// In LoadEnvValues(): +newDefaults, err := state.loadValuesEntries(nil, state.DefaultValues, c.remote, ctxEnv, env) +if err := nil { + return nil, err +} + +if err := mergo.Merge(&e.Defaults, newDefaults, mergo.WithOverride); err != nil { + return nil, err +} +``` + +- Root-level `values:` block is loaded into `Environment.Defaults` +- Uses `mergo.Merge` with `mergo.WithOverride` option +- This happens **after** environment values are loaded (line 172 comes after line 167) + +#### 2. Loading Environment Values (create.go:385-439) + +```go +// In loadEnvValues: +// Step 1: Decrypt non-HCL secrets first +decryptedFiles, err := c.scatterGatherEnvSecretFiles(st, envSecretFiles, secretVals, keepSecretFilesExtensions) + +// Step 2: Load environment values (yaml/gotmpl + HCL + non-HCL secrets) +envValuesEntries := append(decryptedFiles, envSpec.Values...) // Non-HCL secrets first, then env values + valuesVals, err = st.loadValuesEntries(envSpec.MissingFileHandler, envValuesEntries, c.remote, loadValuesEntriesEnv, name) + +// Step 3: Merge secrets into valuesVals (highest priority among env values) +if err = mergo.Merge(&valuesVals, &secretVals, mergo.WithOverride); err != nil { + return nil, err +} + +// Step 4: Create Environment with merged values +newEnv := &environment.Environment{ + Name: name, + Values: valuesVals, // Contains: env values + HCL secrets + non-HCL secrets + Defaults: map[string]any{}, + CLIOverrides: map[string]any{}, +} + +// Step 5: Merge from ctxEnv and overrodeEnv (for multi-part helmfiles and CLI overrides) +if ctxEnv != nil { + newEnv.Defaults = maputil.MergeMaps(ctxEnv.Defaults, newEnv.Defaults) + newEnv.Values = maputil.MergeMaps(ctxEnv.Values, newEnv.Values) + newEnv.CLIOverrides = maputil.MergeMaps(newEnv.CLIOverrides, ctxEnv.CLIOverrides, + maputil.MergeOptions{ArrayStrategy: maputil.ArrayMergeStrategyMerge}) +} +if overrode != nil { + newEnv.Defaults = maputil.MergeMaps(newEnv.Defaults, overrode.Defaults) + newEnv.Values = maputil.MergeMaps(newEnv.Values, overrode.Values) + newEnv.CLIOverrides = maputil.MergeMaps(newEnv.CLIOverrides, overrode.CLIOverrides, + maputil.MergeOptions{ArrayStrategy: maputil.ArrayMergeStrategyMerge}) +} +``` + +**Key Points:** +1. Non-HCL secrets are decrypted **first**, but merged **last** (highest priority) +2. HCL secrets are processed with HCL loader (can reference other HCL values) +3. CLI overrides come from `overrodeEnv` parameter +4. Multiple merge operations use `mergo.Merge` with `mergo.WithOverride` + +#### 3. Final Merge (environment.go:115-129) + +```go +// Called when accessing .Values in templates: +func (e *Environment) GetMergedValues() (map[string]any, error) { + vals := map[string]any{} + vals = maputil.MergeMaps(vals, e.Defaults) // 1. Merge Defaults + vals = maputil.MergeMaps(vals, e.Values) // 2. Merge Values + vals = maputil.MergeMaps(vals, e.CLIOverrides, // 3. Merge CLIOverrides + maputil.MergeOptions{ArrayStrategy: maputil.ArrayMergeStrategyMerge}) + + vals, err := maputil.CastKeysToStrings(vals) + if err != nil { + return nil, err + } + return vals, nil +} +``` + +**Key Points:** +1. **Defaults** (root-level values:) are merged first (lowest priority) +2. **Values** (environment values + secrets) are merged second +3. **CLIOverrides** are merged last (highest priority) +4. **CLIOverrides** always use `ArrayMergeStrategyMerge` (element-by-element) +5. **Defaults/Values** use `ArrayMergeStrategySparse` (auto-detect nil values) + +### Deep Merge Behavior + +Helmfile uses the `dario.cat/mergo` library with `WithOverride` option: + +**Maps:** +```go +if err := mergo.Merge(&dest, &src, mergo.WithOverride); err != nil { + // handle error +} +``` + +This performs: +1. **Deep merge**: Nested maps are merged recursively +2. **Override**: Values from `src` override values in `dest` +3. **Type preservation**: Types are preserved during merge + +**Arrays (via maputil.MergeMaps):** +```go +func MergeMaps(a, b map[string]interface{}, opts ...MergeOptions) map[string]interface{} { + arrayStrategy := ArrayMergeStrategySparse // default + + // ... merging logic ... + + if vSlice != nil { // Array detected + if outSlice != nil { + out[k] = mergeSlices(outSlice, vSlice, arrayStrategy) + continue + } + } +} +``` + +**Three Array Merge Strategies:** + +1. **ArrayMergeStrategySparse** (default for Defaults & Values) + - Auto-detects based on nil values in override array + - If **any nil values**: merge element-by-element (sparse) + - If **no nil values**: replace entire array (complete) + +2. **ArrayMergeStrategyReplace** (not used in standard flow) + - Always replaces entire array + - Used for complete array replacement + +3. **ArrayMergeStrategyMerge** (used for CLIOverrides) + - Always merges element-by-element + - Preserves base array elements unless explicitly overridden + - Perfect for CLI `array[index]=value` syntax + +### Loading Order vs Merge Order + +**Important Distinction:** + +| Values Source | Loading Order | Merge Priority | +|------------------|----------------------|----------------| +| CLI Overrides | Loaded **first** (init) | Merged **last** (highest) | +| Root values: | Loaded **second** | Merged **first** (lowest) | +| Environment values| Loaded **third** | Merged **second** | +| Environment secrets| Loaded **fourth** | Merged **second** (with values)| + +**Why this matters:** +- CLI overrides are available early (for template rendering) +- But they don't override until the final merge +- This allows environment values to reference CLI overrides in templates +- But final merge ensures CLI values win + +## Key Takeaways + +1. **Values merge deeply** - nested maps are merged recursively, not replaced +2. **Arrays use smart merging** - by default, complete arrays replace, sparse arrays merge element-by-element +3. **Later values win** - the order of sources determines precedence +4. **Separate concerns** - state values (`.Values`) vs release values (passed to Helm) +5. **Use templates** - avoid repetition by using `templates:` and root-level `values:` +6. **Secrets have high priority** - non-HCL secrets are merged last, overriding environment values +7. **Prefer maps over arrays** - for configuration that needs incremental updates, use maps instead of arrays +8. **CLI overrides are special** - loaded first but merged last, always using element-by-element array merging + +## Troubleshooting + +### Value not being overridden + +Check the precedence order. A value defined in a base file might be overridden by environment values. Remember that **non-HCL secrets have the highest priority** among environment values. + +### Unexpected array behavior + +Arrays use **sparse auto-detection** by default: +- Arrays with `nil` values: merged element-by-element +- Arrays without `nil` values: **replaced entirely** +- Empty arrays `[]`: **replace entirely** (clears the base array) + +To merge arrays, either: +1. Use `null` (nil) values for indices you want to preserve from the base +2. Convert to maps for better control +3. Use `--state-values-set` CLI flags for element-by-element updates + +### Can't access value in template + +Ensure the value is defined in **state values** (not just in release values). State values are accessible via `{{ .Values.KEY }}`, while release values are only passed to Helm. + +### Values file not being templated + +Only files with `.gotmpl` extension are templated. Regular `.yaml` files are used verbatim. + +### Secrets not overriding values + +Non-HCL secrets are merged **last** in the environment values loading process, giving them the highest priority among environment values. If your secrets aren't overriding, check: +1. The secrets file is properly encrypted +2. The secrets are listed in the `secrets:` section (not `values:`) +3. You're using the correct helm-secrets plugin + +### CLI overrides not working + +CLI `--state-values-set` uses **element-by-element array merging**, which is different from file-based values. This means: +```bash +# This merges at index 0, preserving other indices +helmfile --state-values-set 'servers[0]=prod1.example.com' sync + +# This replaces the entire array +helmfile --state-values-file <(echo 'servers: [prod1.example.com]') sync +``` diff --git a/docs/writing-helmfile.md b/docs/writing-helmfile.md index 25475876..96c499b8 100644 --- a/docs/writing-helmfile.md +++ b/docs/writing-helmfile.md @@ -1,6 +1,8 @@ # The Helmfile Best Practices Guide -This guide covers the Helmfile’s considered patterns for writing advanced helmfiles. It focuses on how helmfile should be structured and executed. +This guide covers the Helmfile's considered patterns for writing advanced helmfiles. It focuses on how helmfile should be structured and executed. + +**Before diving into advanced patterns, we strongly recommend reading [Values Merging and Data Flow](values-and-merging.md)** to understand how Helmfile merges values from various sources. This foundational knowledge is essential for writing effective helmfiles. ## Helmfile .Values vs Helm .Values diff --git a/mkdocs.yml b/mkdocs.yml index 2378c9db..b4c92617 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -20,6 +20,8 @@ nav: - Templating Funcs: templating_funcs.md - HCL Funcs: hcl_funcs.md - Built-in Objects: builtin-objects.md + - Core Concepts: + - Values and Merging: values-and-merging.md - Advanced Features: - Best Practices Guide: writing-helmfile.md - Advanced Features: advanced-features.md