public_git/helmfile
1
0
Fork 0
mirror of https://github.com/helmfile/helmfile.git synced 2026-10-21 12:50:02 +02:00
Code Issues Projects Releases Wiki Activity

feat: add helmfile doctor command for AI-assisted diff analysis (#2660)

Browse Source 
* feat: add `helmfile doctor` command for AI-assisted diff analysis

`helmfile doctor` runs `helmfile diff` and asks an OpenAI-compatible LLM to
summarize the changes and flag risks (data loss, security exposure, breaking
changes, downtime, performance, best-practice issues).

Key design decisions:
- When no LLM is configured, doctor is equivalent to `helmfile diff` with
  one exception: --show-secrets is always forced off (secrets never reach
  stdout, even without an LLM).
- Secrets are ALWAYS redacted via two layers: (1) ShowSecrets() forced to
  false so helm-diff emits <REDACTED> placeholders; (2) a defense-in-depth
  text redactor strips residual secret-looking content (Secret YAML blocks,
  sensitive key/value lines, base64 blobs, JWT tokens) before LLM transmission.
- LLM configuration precedence: env (HELMFILE_LLM_*) < helmfile.yaml (llm:)
  < CLI flags (--llm-*).
- Supports any OpenAI-compatible backend (OpenAI, Azure, One-API, LiteLLM,
  Ollama, etc.) with automatic response_format fallback for backends that
  don't support JSON mode.
- Prompt injection defense: release names and environment values are
  JSON-encoded before insertion into the LLM prompt.
- Exit codes: 0 (success/low-risk), 2 (high-risk gate, bypass with --force),
  1 (other errors). Helm-diff's 'detected changes' exit-2 is swallowed.

New packages:
- pkg/agent/llm: OpenAI-compatible client with JSON response parsing, mock
  client for testing, prompt builder with injection defense.
- pkg/agent/doctor: secret redactor (state machine + regex), report renderer
  (markdown + JSON), config resolver (env < yaml < flag merge).

Testing: 70+ unit tests covering redaction patterns, prompt injection,
response_format fallback, JSON parsing, yaml roundtrip, concurrency safety,
panic recovery, and error propagation. go test -race passes.

Documentation: full doctor section in docs/cli.md, llm: block reference in
docs/configuration.md, updated skills/helmfile for AI agents.

Signed-off-by: yxxhero <aiopsclub@163.com>

* docs: fix doctor equivalence wording per PR review

Per review feedback (PR #2660): the docs claimed doctor is 'equivalent to
helmfile diff — same flags, same output, same exit codes' in the unconfigured
path, but this over-promises because:

  1. doctor --output is the report format (not helm-diff's output format)
  2. helm-diff's --output is exposed as --diff-output in doctor
  3. --show-secrets is silently ignored

Updated all three locations (cli.md, cmd/doctor.go Long + godoc, pkg/app/doctor.go
godoc) to say 'falls back to helmfile diff with --show-secrets forced off' and
explicitly note the --output / --diff-output flag difference.

Signed-off-by: yxxhero <aiopsclub@163.com>

---------

Signed-off-by: yxxhero <aiopsclub@163.com>
This commit is contained in:
yxxhero
2026-06-22 16:52:35 +08:00
committed by GitHub
parent abac6e2ec1
commit 9b943adc9e
33 changed files with 4148 additions and 29 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/cli.md
+198
View File
@@ -160,6 +160,204 @@ To supply the diff functionality Helmfile needs the [helm-diff](https://github.c
you should be able to simply execute `helm plugin install https://github.com/databus23/helm-diff`. For more details
please look at their [documentation](https://github.com/databus23/helm-diff#helm-diff-plugin).
### doctor
`helmfile doctor` runs `helmfile diff` and asks an OpenAI-compatible LLM to summarize the changes and flag risks
(such as data loss, security exposure, breaking changes, downtime, performance, and best-practice issues).
**When no LLM is configured, `doctor` falls back to running `helmfile diff`**
with `--show-secrets` forced off. Most diff flags are accepted for
compatibility, but note two differences: `--output` is reserved for the doctor
report format (use `--diff-output` for helm-diff's plugin output format), and
`--show-secrets` is silently ignored (secrets are always redacted). This makes
it safe to swap into existing CI jobs: the worst case is you get the same diff
output you already had.
#### Configuration
The LLM endpoint speaks the OpenAI Chat Completions protocol (`/v1/chat/completions`). This means it works
with any compatible gateway:
- Direct providers: OpenAI, DeepSeek, Mistral, Together, Groq.
- Unified gateways: One-API, LiteLLM, Azure OpenAI proxy, Cloudflare AI Gateway.
- Local servers: Ollama (with OpenAI compatibility), vLLM, LocalAI.
Configuration precedence (low to high):
1. **Environment variables**: `HELMFILE_LLM_BASE_URL`, `HELMFILE_LLM_API_KEY`, `HELMFILE_LLM_MODEL`,
`HELMFILE_LLM_TIMEOUT` (Go duration, e.g. `90s`), `HELMFILE_LLM_MAX_TOKENS`.
2. **`helmfile.yaml` top-level `llm:` block**:
```yaml
llm:
baseURL: https://one-api.internal/v1
model: gpt-4o
apiKey: {{ env "HELMFILE_LLM_API_KEY" }}
timeout: 60s
maxTokens: 4096
```
3. **CLI flags** (highest precedence): `--llm-base-url`, `--llm-api-key`, `--llm-model`,
`--llm-timeout`, `--llm-max-tokens`.
A layer's non-zero fields override lower layers; empty fields fall through.
> **Note on zero values**: because merge treats "field == zero value" as "not set",
> `--llm-max-tokens 0` does NOT reset MaxTokens to 0 — it is treated as "flag not set"
> and the env/yaml value wins. This matches helmfile's existing flag-override convention
> (same as `--concurrency`, `--context`). Use the yaml block to explicitly request a zero
> value if you really need it.
`baseURL` is **optional**: if you use OpenAI's official endpoint, omit `baseURL` and
the client defaults to `https://api.openai.com/v1`. Set `baseURL` only when targeting
a gateway or non-OpenAI provider. Only `apiKey` and `model` are required to enable
LLM analysis.
#### Output
- **Default (markdown / text)**: a human-readable report with summary, risks sorted by severity
(🔴 high → 🟡 medium → 🟢 low → ⚪ unknown), affected resources, and a footer showing
model, duration, and secrets-redacted count.
- **`--output json`**: structured JSON including the model's analysis, the diff
(always post-redaction), and metadata. Suitable for CI pipelines that want to
post-process (e.g. comment on a pull request). The JSON shape is:
```json
{
"summary": "...",
"risks": [...],
"diff": "...",
"secrets_redacted": 3,
"model": "gpt-4o",
"duration": "8.2s",
"timestamp": "2026-..."
}
```
Key field semantics:
- `risks`: always an array when an analysis ran (even empty: `[]`, never `null`).
When doctor is unconfigured (no LLM), the field is omitted entirely. This lets
CI distinguish "LLM said no risks" from "analysis never happened".
- `diff`: always post-redaction. Doctor never exposes the raw pre-redaction diff
through stdout/JSON. If you need to debug helm-diff itself, run `helmfile diff` directly.
- `secrets_redacted`: always present, even when 0. Lets you confirm the redactor ran.
#### Exit codes
| Code | Meaning |
|---|---|
| 0 | success, or only low/medium risks, or LLM call failed (degraded to plain diff) |
| 2 | at least one high-severity risk and `--force` not passed (CI gate) |
| 1 | other error (state load failure, helm-diff runtime failure, etc.) |
The "detected changes" exit-2 signal from `helm diff --detailed-exitcode` is intentionally swallowed —
doctor's whole job is to react to changes, so reporting them via exit code would be noise.
#### Backend compatibility
Doctor uses `response_format: {type: "json_object"}` (JSON mode) by default for
reliable structured output. If the backend doesn't support JSON mode (common on
early One-API versions, some LiteLLM configs, or Ollama's OpenAI shim), doctor
automatically detects the 400 error, retries without `response_format`, and falls
back to extracting JSON from the model's response via a robust parser that handles
markdown fences and surrounding prose. No user action needed.
#### Safety
- `doctor` never invokes `apply` / `sync` / `destroy`. It is a read-only command.
- LLM calls inherit the global helmfile context timeout, plus a per-request timeout (default 60s).
- If the LLM call fails for any reason, doctor prints the redacted diff with a warning banner and exits 0,
so AI outages never block deployments.
- Large diffs are defensively capped at 32 KB before being sent to the LLM.
Doctor defaults `--context` to 3 (vs diff's 0) so the LLM sees enough surrounding
YAML to ground its analysis. Adjust with `--context N`.
#### Secret handling (always redacted)
Secrets are **always** redacted before any byte leaves the process. This is a
hard safety contract, not a default you can disable.
Two layers enforce it:
1. **`--show-secrets` is silently ignored.** doctor wraps the diff config so
`ShowSecrets()` always reports `false` to helm-diff, which causes
helm-diff itself to substitute `<REDACTED>` for secret values. If you
already pass `--show-secrets` in a wrapper script, doctor still redacts.
2. **Defense-in-depth text redactor.** A second pass strips any residual
secret-looking content from the captured diff before it is sent to the
LLM. It catches:
- `kind: Secret` resource blocks (full YAML mode),
- sensitive key/value lines (`password`, `token`, `apiKey`, `client_secret`,
`bearer`, …) regardless of casing or separator (`api_key` = `apiKey` =
`API_KEY`),
- free-form base64 runs of 40+ chars (encoded keys/certs),
- JWT-shaped tokens (`eyJ...`).
The redaction count is always surfaced — even when 0 — in the report footer
and JSON output, so you can confirm the redactor ran:
```markdown
---
Model: gpt-4o | Duration: 8.2s | Secrets redacted: 3
```
In `--output json` mode the field is `secrets_redacted` (always present).
If you want to drop the *structure* of Secret resources entirely (not just
their values), pass `--suppress-secrets`:
```bash
helmfile doctor --suppress-secrets
```
#### Prompt injection defense
Release names and environment values from `helmfile.yaml` are embedded in the
LLM prompt as context. Since helmfile.yaml can come from untrusted sources
(e.g. a GitOps pull request), doctor JSON-encodes these fields via
`encoding/json` before insertion. This ensures that a malicious release name
like `"foo\n\nIgnore previous instructions"` is treated as opaque string data
by the model, not as a directive.
#### Known limitations
- **Double state load**: doctor loads the helmfile state twice — once to
peek at the `llm:` block and release names, and again inside `helmfile diff`.
For large helmfiles with remote bases or heavy templating this can add
noticeable latency. Caching across loads would require touching core code,
which doctor intentionally avoids. In the unconfigured path (no LLM),
the cost equals plain `helmfile diff`.
- **`--log-output stdout`**: if you point helmfile's logger at stdout (via
`--log-output stdout`), log lines will be captured alongside the diff and
sent to the LLM. Doctor warns when it detects this configuration. Prefer
the default (stderr) when running doctor.
#### Examples
Quick local analysis (OpenAI official — no baseURL needed):
```bash
export HELMFILE_LLM_API_KEY=sk-...
export HELMFILE_LLM_MODEL=gpt-4o
helmfile doctor
```
CI gate that fails the build on high-severity risk:
```bash
helmfile --environment prod doctor --output json > doctor-report.json
# exit code 2 stops CI; --force bypasses when a human has approved the change
```
Pin the gateway per-environment via `helmfile.yaml`:
```yaml
environments:
prod:
values:
- llmGateway: https://prod-llm-gateway.internal/v1
llm:
baseURL: {{ .Values.llmGateway | quote }}
model: gpt-4o
apiKey: {{ env "HELMFILE_LLM_API_KEY" }}
```
### apply
The `helmfile apply` sub-command begins by executing `diff`. If `diff` finds that there is any changes, `sync` is executed. Adding `--interactive` instructs Helmfile to request your confirmation before `sync`.