fix: array merge regression - layer arrays now replace defaults (#2367)

* fix: array merge regression - layer arrays now replace defaults (#2353)

PR #2288 introduced element-by-element array merging to fix #2281, but this
caused a regression where layer/environment arrays were merged instead of
replacing base arrays entirely.

This fix uses automatic sparse array detection:
- Arrays with nil values (from --state-values-set) merge element-by-element
- Arrays without nils (from layer YAML) replace entirely

This follows Helm's documented behavior where arrays replace rather than merge.

Signed-off-by: Aditya Menon <amenon@canarytechnologies.com>

* fix: use separate CLIOverrides field for element-by-element array merging

The previous approach using ArrayMergeStrategySparse detection didn't work
for --state-values-set array[0]=value because setting index 0 produces no
nils in the array.

This fix adds a CLIOverrides field to Environment that keeps CLI values
separate from layer values. CLI overrides are merged last using
ArrayMergeStrategyMerge (always element-by-element), while layer values
use the default strategy (arrays replace).

This ensures:
- --state-values-set array[0]=x only changes index 0, preserving other elements
- Layer/environment file arrays still replace base arrays entirely
- Issue #2281 fix is preserved (--state-values-set array[1].field=x works)

Signed-off-by: Aditya Menon <amenon@canarytechnologies.com>

* fix: correct comment about array merge strategy in test

Signed-off-by: Aditya Menon <amenon@canarytechnologies.com>

* fix: propagate Defaults in multi-part helmfiles and fix merge order

- Add Defaults field merging from ctxEnv to preserve base values across
  helmfile parts separated by ---
- Fix merge order: current part values now correctly override previous
  parts (was reversed, causing older values to win)
- Update 147 snapshot test files for new Environment log format with
  CLIOverrides field

This completes the fix for issue #2353 by ensuring:
1. Layer arrays replace entirely (not element-by-element merge)
2. CLI --state-values-set sparse arrays still merge element-by-element
3. Multi-part helmfiles properly inherit and override values

Signed-off-by: Aditya Menon <amenon@canarytechnologies.com>

* fix: address Copilot review comments

- Initialize EmptyEnvironment with empty maps to match New() constructor
- Update test comment to accurately describe ArrayMergeStrategySparse

Signed-off-by: Aditya Menon <amenon@canarytechnologies.com>

* fix: ensure templates access merged values via .Environment.Values

This commit fixes a regression in the CLIOverrides integration where
templates accessing .Environment.Values couldn't see CLI override values.

Changes:
- Remove CLIOverrides-into-Values merge from Merge() to keep proper
  layering order (Defaults → Values → CLIOverrides) in GetMergedValues()
- Update NewEnvironmentTemplateData to set envCopy.Values to the merged
  values, ensuring templates see the same values via both .Values and
  .Environment.Values

This ensures:
- Issue #2353: Layer arrays still replace entirely (Sparse strategy)
- Issue #2281: CLI sparse arrays still merge element-by-element
- Templates can access CLI overrides via .Environment.Values

Signed-off-by: Aditya Menon <amenon@canarytechnologies.com>

* docs: improve mergeSlices documentation per Copilot review

Address Copilot review comments on PR #2367:
- Document empty array edge case: explicitly setting [] clears base array
- Document recursive strategy propagation for nested map merging
- Add comprehensive behavior description for all array merge strategies

Signed-off-by: Aditya Menon <amenon@canarytechnologies.com>

* fix: use merged values when rendering environment value files

Environment value files (*.yaml.gotmpl) can reference CLI values via
.Values. Previously, only env.Values was passed to template rendering,
which didn't include CLIOverrides.

Now we call env.GetMergedValues() to get Defaults + Values + CLIOverrides
before rendering, so templates can access CLI values like:
  --state-values-set foo=bar

This fixes the state-values-set-cli-args-in-environments integration test.

Signed-off-by: Aditya Menon <amenon@canarytechnologies.com>

---------

Signed-off-by: Aditya Menon <amenon@canarytechnologies.com>

pkg/maputil/maputil.go

@@ -224,33 +224,40 @@ func typedVal(val string, st bool) any {
 	return val
 }
 
-func MergeMaps(a, b map[string]interface{}) map[string]interface{} {
+// MergeMaps merges two maps with special handling for nested maps and arrays.
+func MergeMaps(a, b map[string]interface{}, opts ...MergeOptions) map[string]interface{} {
+	arrayStrategy := ArrayMergeStrategySparse
+	if len(opts) > 0 {
+		arrayStrategy = opts[0].ArrayStrategy
+	}
+
 	out := make(map[string]interface{}, len(a))
-	// fill the out map with the first map
 	for k, v := range a {
 		out[k] = v
 	}
 	for k, v := range b {
 		if v == nil {
+			// If key doesn't exist in base, add nil (issue #1154).
+			// If key exists in base, don't overwrite with nil.
+			if _, exists := out[k]; !exists {
+				out[k] = nil
+			}
 			continue
 		}
 		if v, ok := v.(map[string]interface{}); ok {
 			if bv, ok := out[k]; ok {
 				if bv, ok := bv.(map[string]interface{}); ok {
-					// if b and out map has a map value, merge it too
-					out[k] = MergeMaps(bv, v)
+					out[k] = MergeMaps(bv, v, opts...)
 					continue
 				}
 			}
 		}
-		// Handle array merging element-by-element
 		vSlice := toInterfaceSlice(v)
 		if vSlice != nil {
 			if outVal, exists := out[k]; exists {
 				outSlice := toInterfaceSlice(outVal)
 				if outSlice != nil {
-					// Both are slices - merge element by element
-					out[k] = mergeSlices(outSlice, vSlice)
+					out[k] = mergeSlices(outSlice, vSlice, arrayStrategy)
 					continue
 				}
 			}
@@ -260,7 +267,6 @@ func MergeMaps(a, b map[string]interface{}) map[string]interface{} {
 	return out
 }
 
-// toInterfaceSlice converts various slice types to []interface{}
 func toInterfaceSlice(v any) []any {
 	if slice, ok := v.([]any); ok {
 		return slice
@@ -268,41 +274,82 @@ func toInterfaceSlice(v any) []any {
 	return nil
 }
 
-// mergeSlices merges two slices element by element
-// Elements from override (b) take precedence, but we preserve elements from base (a)
-// that don't exist in override
-func mergeSlices(base, override []any) []any {
-	// Determine the maximum length
+type ArrayMergeStrategy int
+
+const (
+	// ArrayMergeStrategySparse uses auto-detection: sparse arrays (with nils) merge
+	// element-by-element, complete arrays (no nils) replace entirely.
+	ArrayMergeStrategySparse ArrayMergeStrategy = iota
+	// ArrayMergeStrategyReplace always replaces arrays entirely.
+	ArrayMergeStrategyReplace
+	// ArrayMergeStrategyMerge always merges arrays element-by-element (for CLI overrides).
+	ArrayMergeStrategyMerge
+)
+
+type MergeOptions struct {
+	ArrayStrategy ArrayMergeStrategy
+}
+
+// mergeSlices merges two slices based on the strategy.
+//
+// Behavior by strategy:
+//   - ArrayMergeStrategyReplace: always returns override as-is
+//   - ArrayMergeStrategySparse: auto-detects based on nil values (see below)
+//   - ArrayMergeStrategyMerge: always merges element-by-element
+//
+// For Sparse strategy, auto-detection logic:
+//   - If override contains ANY nil values → treated as sparse → merge element-by-element
+//   - If override contains NO nil values → treated as complete → replace entirely
+//
+// Edge cases:
+//   - Empty array `[]` has no nils, so it replaces entirely. This is intentional:
+//     explicitly setting an empty array should clear the base array.
+//   - Explicit `[null, value]` in YAML is treated as sparse (rare but correct).
+//
+// Recursive behavior: When merging maps within array elements, the same strategy
+// is propagated to nested MergeMaps calls, maintaining consistent merge semantics.
+func mergeSlices(base, override []any, strategy ArrayMergeStrategy) []any {
+	if strategy == ArrayMergeStrategyReplace {
+		return override
+	}
+
+	if strategy == ArrayMergeStrategySparse {
+		isSparse := false
+		for _, v := range override {
+			if v == nil {
+				isSparse = true
+				break
+			}
+		}
+		if !isSparse {
+			return override
+		}
+	}
+
+	// Merge element-by-element (for ArrayMergeStrategyMerge or sparse arrays)
 	maxLen := len(base)
 	if len(override) > maxLen {
 		maxLen = len(override)
 	}
 
 	result := make([]interface{}, maxLen)
-
-	// First copy all elements from base
 	copy(result, base)
 
-	// Then merge/override with elements from override
 	for i := 0; i < len(override); i++ {
 		overrideVal := override[i]
-
-		// Skip nil values in override - they represent unset indices
 		if overrideVal == nil {
 			continue
 		}
 
-		// If both are maps, merge them recursively
 		if overrideMap, ok := overrideVal.(map[string]any); ok {
 			if i < len(base) {
 				if baseMap, ok := base[i].(map[string]any); ok {
-					result[i] = MergeMaps(baseMap, overrideMap)
+					result[i] = MergeMaps(baseMap, overrideMap, MergeOptions{ArrayStrategy: strategy})
 					continue
 				}
 			}
 		}
 
-		// Otherwise, override completely
 		result[i] = overrideVal
 	}