fix: skip cache refresh for shared cache paths to prevent race conditions (#2396)

* fix: skip cache refresh for shared cache paths to prevent race conditions

When multiple helmfile processes run in parallel (e.g., as ArgoCD plugin),
they share the same OCI chart cache in ~/.cache/helmfile. One process could
delete and re-download (refresh) a cached chart while another process was
still using it, causing "path not found" errors.

This fix:
- Adds isSharedCachePath() helper to detect shared cache paths
- Skips chart deletion/refresh for paths in the shared cache directory
- Users can force refresh by running `helmfile cache cleanup` first

Fixes #2387

Signed-off-by: yxxhero <aiopsclub@163.com>

* docs: document OCI chart caching behavior and multi-process safety

Add documentation for:
- OCI chart cache location and behavior
- How to force cache refresh with `helmfile cache cleanup`
- Multi-process safety when using shared cache
- Cache management commands (info, cleanup)

Signed-off-by: yxxhero <aiopsclub@163.com>

* fix: address review comments for shared cache handling

- Return error instead of chartActionDownload for corrupted shared cache
- Change refresh skip log from Debugf to Infof for user visibility
- Add t.Helper() to createTestLogger test helper

Signed-off-by: yxxhero <aiopsclub@163.com>

* fix: handle symlinks and add debug logging in isSharedCachePath

- Use filepath.EvalSymlinks to resolve symlinks before path comparison
- Add debug logging when filepath.Abs fails
- Add test case for symlink to shared cache directory

Signed-off-by: yxxhero <aiopsclub@163.com>

* fix: address copilot review comments

- Include underlying error in corrupted cache error message
- Add cleanup for test directories created in shared cache
- Clarify --skip-refresh flag documentation

Signed-off-by: yxxhero <aiopsclub@163.com>

* fix: handle edge case when chartPath equals sharedCacheDir

- isSharedCachePath now returns true for exact match with cache dir
- Add test case for exact match with shared cache directory

Signed-off-by: yxxhero <aiopsclub@163.com>

* test: add integration test for acquireChartLock shared cache behavior

Add TestAcquireChartLockSharedCacheSkipRefresh to verify that
acquireChartLock returns chartActionUseCached instead of
chartActionRefresh when the chart exists in the shared cache,
even when refresh is requested. This tests the core fix for
the race condition issue #2387.

Signed-off-by: yxxhero <aiopsclub@163.com>

---------

Signed-off-by: yxxhero <aiopsclub@163.com>
This commit is contained in:
yxxhero
2026-02-14 10:46:05 +08:00
committed by GitHub
parent c6b962dbbf
commit 58df057dcc
3 changed files with 234 additions and 5 deletions
+35
View File
@@ -665,6 +665,33 @@ The `helmfile init` sub-command checks the dependencies required for helmfile op
The `helmfile cache` sub-command is designed for cache management. Go-getter-backed remote file system are cached by `helmfile`. There is no TTL implemented, if you need to update the cached files or directories, you need to clean individually or run a full cleanup with `helmfile cache cleanup`
#### OCI Chart Cache
OCI charts are cached in the shared cache directory (`~/.cache/helmfile` by default, or `$HELMFILE_CACHE_HOME`). This cache is shared across all helmfile processes.
**Cache Behavior:**
- When a chart exists in the shared cache and is valid, it is reused without re-downloading
- The `--skip-refresh` flag can be used to skip checking for updates to cached charts stored in process-specific temporary directories (it does not affect charts already present in the shared cache)
- When running multiple helmfile processes in parallel (e.g., as an ArgoCD plugin), charts in the shared cache are not refreshed/deleted to prevent race conditions
**Forcing a Cache Refresh:**
To force a refresh of cached OCI charts, run:
```bash
helmfile cache cleanup
```
This will clear the shared cache, allowing the next helmfile command to re-download charts.
#### cache info
Display information about the cache directory.
#### cache cleanup
Remove all cached files from the cache directory.
### sync
The `helmfile sync` sub-command sync your cluster state as described in your `helmfile`. The default helmfile is `helmfile.yaml`, but any YAML file can be passed by specifying a `--file path/to/your/yaml/file` flag.
@@ -1867,6 +1894,14 @@ export MY_OCI_REGISTRY_USERNAME=spongebob
export MY_OCI_REGISTRY_PASSWORD=squarepants
```
### OCI Chart Caching
OCI charts are automatically cached in the shared cache directory (`~/.cache/helmfile` by default, or the directory specified by `HELMFILE_CACHE_HOME`). This improves performance by avoiding redundant downloads.
**Multi-Process Safety:** When running multiple helmfile processes in parallel (e.g., as an ArgoCD plugin), charts in the shared cache are not deleted or refreshed to prevent race conditions where one process might delete a chart that another is using. To force a cache refresh, run `helmfile cache cleanup` first.
See the [cache](#cache) section for more details on cache management.
## Attribution
We use: