* feat: add `helmfile doctor` command for AI-assisted diff analysis
`helmfile doctor` runs `helmfile diff` and asks an OpenAI-compatible LLM to
summarize the changes and flag risks (data loss, security exposure, breaking
changes, downtime, performance, best-practice issues).
Key design decisions:
- When no LLM is configured, doctor is equivalent to `helmfile diff` with
one exception: --show-secrets is always forced off (secrets never reach
stdout, even without an LLM).
- Secrets are ALWAYS redacted via two layers: (1) ShowSecrets() forced to
false so helm-diff emits <REDACTED> placeholders; (2) a defense-in-depth
text redactor strips residual secret-looking content (Secret YAML blocks,
sensitive key/value lines, base64 blobs, JWT tokens) before LLM transmission.
- LLM configuration precedence: env (HELMFILE_LLM_*) < helmfile.yaml (llm:)
< CLI flags (--llm-*).
- Supports any OpenAI-compatible backend (OpenAI, Azure, One-API, LiteLLM,
Ollama, etc.) with automatic response_format fallback for backends that
don't support JSON mode.
- Prompt injection defense: release names and environment values are
JSON-encoded before insertion into the LLM prompt.
- Exit codes: 0 (success/low-risk), 2 (high-risk gate, bypass with --force),
1 (other errors). Helm-diff's 'detected changes' exit-2 is swallowed.
New packages:
- pkg/agent/llm: OpenAI-compatible client with JSON response parsing, mock
client for testing, prompt builder with injection defense.
- pkg/agent/doctor: secret redactor (state machine + regex), report renderer
(markdown + JSON), config resolver (env < yaml < flag merge).
Testing: 70+ unit tests covering redaction patterns, prompt injection,
response_format fallback, JSON parsing, yaml roundtrip, concurrency safety,
panic recovery, and error propagation. go test -race passes.
Documentation: full doctor section in docs/cli.md, llm: block reference in
docs/configuration.md, updated skills/helmfile for AI agents.
Signed-off-by: yxxhero <aiopsclub@163.com>
* docs: fix doctor equivalence wording per PR review
Per review feedback (PR #2660): the docs claimed doctor is 'equivalent to
helmfile diff — same flags, same output, same exit codes' in the unconfigured
path, but this over-promises because:
1. doctor --output is the report format (not helm-diff's output format)
2. helm-diff's --output is exposed as --diff-output in doctor
3. --show-secrets is silently ignored
Updated all three locations (cli.md, cmd/doctor.go Long + godoc, pkg/app/doctor.go
godoc) to say 'falls back to helmfile diff with --show-secrets forced off' and
explicitly note the --output / --diff-output flag difference.
Signed-off-by: yxxhero <aiopsclub@163.com>
---------
Signed-off-by: yxxhero <aiopsclub@163.com>
* feat: parallel kubedog tracking with progress printer and safety valves
Rework kubedog integration so resource tracking runs in parallel with
helm upgrade/install, giving live progress output and recovering from
known helm/kubedog wedge conditions.
Core:
- kubedogTrackingHandle runs tracking in a background goroutine alongside
the helm subprocess (startBackgroundKubedogTracking); helm output is
buffered and replayed as a single block so it no longer interleaves
with progress ticks.
- Capture UID+generation baselines before handing off to helm so each
tracker waits until the resource actually changes (freshness gate).
Progress printer (pkg/kubedog/printer.go):
- Styled, auto-sized progress table with a heartbeat flusher, child (pod)
status roll-up, pre-ready pod-phase handling, multi-namespace support,
and optional color. PreviewBreakdown summarizes kept/filtered resources.
Safety valves (verify cluster state via the live API):
- Tracker-race valve (always on): when helm succeeds but a dyntracker
goroutine is wedged, poll the API and cancel the tracker so wait()
returns success instead of blocking until --track-timeout.
- Helm-stuck killer (opt-in via helmStuckGrace): if the cluster stays
converged while helm v4's hook waiter is wedged, SIGINT the helm
subprocess to recover.
- Failure watchdog (pkg/kubedog/watchdog.go): surface failing pods that
never made it into dyntracker's resource graph.
Options: trackFailedLogs, helmStuckGrace, trackTimeout, color
(Color/NoColor), and resource filtering (trackKinds/skipKinds/
trackResources). PersistentVolumeClaim support in resource classification.
Signed-off-by: Roman Mykhailiuk <romanm@cybellum.com>
* test: add unit tests for kubedog tracking
Cover the progress printer, resource classification, the failure
watchdog, helm-output trimming/dedup, the release hard-timeout helper,
and the color/track option plumbing.
Signed-off-by: Roman Mykhailiuk <romanm@cybellum.com>
* test: update golden logs, e2e snapshots, and values-id fixtures
Refresh pkg/app testapply/testdestroy golden logs, e2e template
snapshots, and the TestGenerateID values-id golden hashes for the new
kubedog progress output and the merged release struct layout.
Signed-off-by: Roman Mykhailiuk <romanm@cybellum.com>
* refactor: remove dead kubedog display code and dedupe tracker setup
Deep-review pass on the parallel kubedog tracking changes:
- Remove pkg/kubedog/display.go (308 lines) and display_test.go (453 lines).
These rendered progress for the legacy per-kind trackers that this PR
replaces; in the merged tree every function is unreferenced outside its
own tests. The new progressPrinter (printer.go) supersedes them.
TestMain (color.ForceColor for deterministic ANSI in tests) is preserved
in a new main_test.go.
- Dedupe trackWithKubedog: the post-helm fallback rebuilt the exact same
tracker options as buildReleaseTracker. Reuse buildReleaseTracker instead,
dropping ~40 lines of duplicated timeout/log/filter/tracker construction.
No behavior change; build, go vet, golangci-lint, and the kubedog/state
unit tests all pass.
Signed-off-by: yxxhero <aiopsclub@163.com>
* fix: stop waitForFreshness busy-looping the API after helm finishes
Once upstreamDoneCh closes it is always ready, so the select in
waitForFreshness stopped blocking on the ticker and re-ran probe() (a
live GET) as fast as the round-trip allowed for the whole 3s grace
window — hammering the API server once per tracked resource.
Track a local view of the channel and nil it out on first delivery so
the first hit records the timestamp (one fast retry, as intended) and
all subsequent polls are ticker-throttled. Functional behavior is
unchanged: return nil when fresh, errUpstreamDoneNoChange after grace.
Signed-off-by: yxxhero <aiopsclub@163.com>
* test: drop trailing blank line from helm4 OCI pull snapshots
The trailing-newline trim in helmexec.info() removes the blank line helm
prints after the OCI chart "Digest:" line. The helm3 (output.yaml)
snapshots never captured that line, but the helm4 (output-helm4.yaml)
snapshots for oci_chart_pull{,_direct,_once,_once2} and
issue_473_oci_chart_url_fetch still expected it, so they failed under
helm 4. Remove the blank line so the snapshots match the trimmed output.
Signed-off-by: yxxhero <aiopsclub@163.com>
* test: refresh diff-args integration goldens for styled release headers
DisplayAffectedReleases now emits a styled "========== Updated Releases
==========" header (matching the app/e2e goldens already updated by this
PR) and helmexec.info() trims the trailing blank after helm's install
status. Update the diff-args apply-stderr{,-helm4} and apply-live-
stderr{,-helm4} goldens accordingly so they match the actual stderr.
Signed-off-by: yxxhero <aiopsclub@163.com>
* test: drop trimmed blank line from v1-subhelmfile template golden
The trailing-newline trim in helmexec.info() removes the blank line helm
prints after '"incubator" has been added to your repositories'. Update
the v1-subhelmfile-multi-bases-with-array-values result and result-live
goldens so the template stdout comparison matches.
Signed-off-by: yxxhero <aiopsclub@163.com>
---------
Signed-off-by: Roman Mykhailiuk <romanm@cybellum.com>
Signed-off-by: yxxhero <aiopsclub@163.com>
Co-authored-by: Roman Mykhailiuk <romanm@cybellum.com>
* chore: bump helm release pins
* chore: align helm module metadata
* chore: finalize helm patch bumps
* fix: add --plain-http flag for Helm 3.21+ OCI push in tests
Helm 3.21.1 introduced stricter security checks that reject HTTP
scheme downgrades when pushing to OCI registries, with the error:
"blob upload Location downgrades scheme from https"
Previously only Helm 4 required --plain-http for HTTP-only OCI
registries. Now Helm 3.21+ also requires this flag.
Add a new requiresPlainHTTPForOCI() helper that returns true for
both Helm 4.x and Helm 3.21+, and use it in execHelmPush() instead
of isHelm4().
* fix: safe fallback in requiresPlainHTTPForOCI when version detection fails
Default to true (require --plain-http) when helm version detection
fails, since any Helm version that supports helm push also supports
the --plain-http flag. This avoids the inconsistent HELMFILE_HELM4
env var fallback which only covered Helm 4.
* fix: update snapshot tests for Helm 4.2.1 OCI pull output
Helm 4.2.1 now outputs additional 'Pulled:' and 'Digest: sha256:...'
lines after each OCI chart pull. The SHA256 digest is non-deterministic
because helm packages include build timestamps, so normalize it with
a regex placeholder.
- Add ociDigestRegex to normalize non-deterministic OCI digest values
- Create output-helm4.yaml for 5 tests that lacked Helm 4 snapshots
- Update output-helm4.yaml for oci_need and postrenderer to include
the new Pulled/Digest lines from Helm dependency pull operations
* fix: update ociDigestRegex to match empty digest in Helm 4.2.1 OCI pull output
Helm 4.2.1 outputs "Digest: sha256:" (empty hash) when pulling OCI charts.
The regex required at least one hex char ([0-9a-f]+), so it did not match
and the digest was not normalized to $DIGEST in snapshot tests.
Also fix the replacement string: Go regex ReplaceAllString interprets $DIGEST
as a capture group reference (resolving to empty). Use $$DIGEST to produce
a literal $DIGEST in the output.
Signed-off-by: yxxhero <aiopsclub@163.com>
---------
Signed-off-by: yxxhero <aiopsclub@163.com>
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: yxxhero <aiopsclub@163.com>
* fix: refresh Chart.lock after rewriting file:// dependencies
`rewriteChartDependencies` rewrites relative `file://` repository URLs in
Chart.yaml to absolute paths so chartify can resolve them from a temp
directory. That mutates the Chart.yaml dependencies block, which
invalidates the Chart.lock digest (helm computes it as
`sha256(json.Marshal([2][]Dependency{req, lock}))` over the dependencies).
Once the lock is out of sync, downstream `helm dependency build` errors
with "the lock file (Chart.lock) is out of sync with the dependencies
file (Chart.yaml)" and chartify falls back to `helm dependency update`.
`dep update` then re-resolves Chart.yaml's version constraints against
the chart repo, so any constraint that admits newer versions
(e.g. `version: "*"`, `~1.0`) silently picks up a newer dependency on
every render — even though Chart.lock pins a specific version.
Repro:
- Chart.yaml has `version: "*"` for some-dep, Chart.lock pins 4.1.0,
upstream now publishes 4.2.0.
- `helm template .` honors the cached `charts/some-dep-4.1.0.tgz`.
- `helmfile template` produces 4.2.0, because it triggered chartify
(via jsonPatches/strategic-merge/kustomize/etc), which copied the
chart, ran `dep build` against an out-of-sync lock, fell back to
`dep up`, and re-resolved the wildcard.
This commit refreshes Chart.lock alongside Chart.yaml in the temp copy:
- Mirror the rewritten file:// repository URLs onto matching entries in
Chart.lock's dependencies. Without this, `helm dep build` would resolve
the lock's relative `file://` paths against the temp chart directory
and fail with "directory ... not found".
- Recompute the digest using helm's resolver.HashReq algorithm
(`sha256(json.Marshal([2][]chart.Dependency{req, lock}))`). The
algorithm is small and stable; resolver.HashReq itself lives in an
internal package, so it's inlined here.
- Locked versions are preserved verbatim — only the repository URL is
updated and the digest recomputed. Chart.lock remains the source of
truth for which versions get installed.
- The original Chart.lock on disk is never modified; only the temp copy
is rewritten.
Adds TestRewriteChartDependencies_RefreshesChartLock covering digest
recomputation, file:// URL mirroring, version preservation, untouched
non-file:// deps, and original-on-disk integrity.
Signed-off-by: Shane Starcher <shane.starcher@gmail.com>
* fix: address Copilot review issues for Chart.lock refresh
- Map all helm Dependency fields (alias, condition, tags, import-values,
enabled) when building the request slice for digest computation, not
just name/version/repository. This ensures the recomputed digest
matches Helm's resolver.HashReq for all dependency shapes.
- Match lock entries by Name + Alias (not Name alone) to correctly
handle charts with duplicate dependency names distinguished by alias.
- Log a warning when reading Chart.lock fails with a non-NotExist error,
while still treating a missing Chart.lock as expected.
- Add test case exercising dependencies with alias, condition, tags, and
import-values fields, including same-name deps disambiguated by alias.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: Shane Starcher <shane.starcher@gmail.com>
* build(deps): bump github.com/helmfile/chartify to v0.26.4
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: Shane Starcher <shane.starcher@gmail.com>
* fix: normalize import-values for JSON marshaling and improve test coverage
- Normalize import-values using maputil.RecursivelyStringifyMapKey before
assigning to helmchart.Dependency.ImportValues. When go-yaml v2 decodes
nested maps (e.g. import-values entries with child/parent keys), they
become map[interface{}]interface{} which json.Marshal cannot encode.
This would silently prevent Chart.lock rewriting. The normalization
converts all map keys to strings, making the value JSON-safe.
- Improve TestRewriteChartDependencies_RefreshesChartLockWithExtraFields
to prove that extra fields (condition, tags, import-values) actually
affect the computed digest by comparing digests with and without those
fields and asserting they differ.
Signed-off-by: Shane Starcher <shane.starcher@gmail.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: normalize lock ImportValues and fix digest test isolation
- Normalize lock.Dependencies ImportValues via RecursivelyStringifyMapKey
before json.Marshal, preventing failures when go-yaml v2 decodes nested
maps as map[interface{}]interface{}.
- Fix TestRewriteChartDependencies_RefreshesChartLockWithExtraFields to use
a shared root directory so both chart variants resolve file:// paths to
the same absolute location, isolating digest differences to field content.
- Add TestRewriteChartDependencies_GoYamlV2ImportValues exercising the
HELMFILE_GO_YAML_V3=false path with import-values containing nested maps.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: Shane Starcher <shane.starcher@gmail.com>
* fix: add exact digest verification test against Helm's HashReq
Add TestRewriteChartDependencies_DigestMatchesHelmHashReq which computes
the expected digest independently using the same algorithm as Helm's
resolver.HashReq and asserts the rewritten Chart.lock matches exactly.
This guards against producing a digest that is "different" yet still
rejected by `helm dependency build`.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: Shane Starcher <shane.starcher@gmail.com>
---------
Signed-off-by: Shane Starcher <shane.starcher@gmail.com>
Co-authored-by: Shane Starcher <shane.starcher@gmail.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Update vals to v0.44.0 to fix YC_TOKEN json-string handling for
Yandex Lockbox secret decryption. Also bumps transitive dependencies
including yandex-cloud/go-genproto (0.69.0 -> 0.75.0) which contains
the actual fix.
Add replace directives to pin k8s.io/{api,apimachinery,client-go} to
v0.35.4 for compatibility with helm and kubedog dependencies that
still require removed k8s.io/api packages (scheduling/v1alpha1,
autoscaling/v2beta2).
Closes#2564
Signed-off-by: yxxhero <aiopsclub@163.com>
* build(deps): replace werf/kubedog-for-werf-helm with werf/kubedog
Replace the fork github.com/werf/kubedog-for-werf-helm with the upstream
github.com/werf/kubedog. The fork was a temporary compatibility shim; the
upstream repository now includes the necessary k8s API fixes.
Signed-off-by: yxxhero <aiopsclub@163.com>
* fix(kubedog): initialize InformerFactory to prevent nil pointer panic
The upstream werf/kubedog now requires an InformerFactory for its resource
trackers (deployment, statefulset, daemonset, job, canary), but the
multitrack layer still passes nil. Bypass the broken multitrack feed layer
by creating resource trackers directly with a properly initialized
InformerFactory.
Signed-off-by: yxxhero <aiopsclub@163.com>
* fix(lint): correct misspelling of canceled
Signed-off-by: yxxhero <aiopsclub@163.com>
---------
Signed-off-by: yxxhero <aiopsclub@163.com>
dependabot[bot]