diff --git a/CHANGELOG.md b/CHANGELOG.md index e2bc96e..2cd1bef 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -8,6 +8,12 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Added +- **`skills/ha-git-backup`** — an ops-companion skill for the engineer package: a two-circuit + Home Assistant backup system (real git in `/config` with a deploy key and a pre-commit secret + scanner + age-encrypted full backups in GitHub Releases), with install/sync/scan/offsite/restore + scripts, HA automations, a restore runbook and a mandatory monthly drill. Field-tested; + the secret scanner ships canary-tested (BusyBox `grep -e` lesson recorded). + - **Manufacturer names from the archive itself** (`appprog_parser.py`): `parse_devices_from_project` now reads vendor names from the `knx_master.xml` shipped inside every `.knxproj`/`.knxprod` (e.g. M-0073 → HDL, M-00B6 → Ekinex S.p.A.) instead of relying on a hardcoded id map that can diff --git a/README.md b/README.md index 9a23196..21ab944 100644 --- a/README.md +++ b/README.md @@ -160,6 +160,9 @@ spec encodes. KNX IoT (Turtle/RDF) semantic export. - Live control of the house stays in the official Home Assistant integration (layer 1) — this server only prepares its configuration. +- **Ops companion**: [`skills/ha-git-backup`](skills/ha-git-backup) — the life of your config + *after* deploy: a real git history of `/config` (deploy key + pre-commit secret scanner) plus + encrypted offsite backups in GitHub Releases, with a monthly restore drill. ### 🧩 The foundation — a growing device library diff --git a/README.ru.md b/README.ru.md index 12bf7dd..932d721 100644 --- a/README.ru.md +++ b/README.ru.md @@ -138,6 +138,9 @@ KNX Community в мае 2026 прямо просит такую интеграц семантический экспорт KNX IoT (Turtle/RDF). - Живое управление домом остаётся в официальной интеграции Home Assistant (слой 1) — этот сервер только готовит её конфигурацию. +- **Ops-компаньон**: [`skills/ha-git-backup`](skills/ha-git-backup) — жизнь конфига *после* + деплоя: настоящая git-история `/config` (deploy key + pre-commit сканер секретов) плюс + шифрованные offsite-бэкапы в GitHub Releases и ежемесячный restore drill. ### 🧩 Фундамент — растущая библиотека устройств diff --git a/skills/ha-git-backup/README.md b/skills/ha-git-backup/README.md new file mode 100644 index 0000000..947a279 --- /dev/null +++ b/skills/ha-git-backup/README.md @@ -0,0 +1,37 @@ +# ha-git-backup — the ops companion skill + +Our Scenario 3 ends with `generate_ha_package` handing you a Home Assistant KNX YAML. This +skill covers what happens **after you deploy it**: the life of your `/config`. + +**Two circuits, two different jobs:** + +1. **History** — a *real* git repo inside `/config`, pushed to a private GitHub repo over a + **deploy key** (blast radius: one repo). Every change is a meaningful commit + (`[daily] 3 file(s) | HA 2026.7.1`); a **pre-commit secret scanner** blocks any commit whose + diff contains a token, password or private key. Answers "what broke my automation last night" + in one `git diff`. +2. **Recovery** — full native HA backups (including `.storage`: entity registry, auth, UI + dashboards) **age-encrypted** and uploaded to GitHub Releases with rotation. The private age + key never touches the HA host — a compromised host cannot read its own backups. Answers + "the SSD died" with a 30-minute restore. + +Plus: HA automations (daily commit at 03:30, a pre-update tag, a sync button, a failure alert), +a health sensor, a restore runbook and a **mandatory monthly restore drill** — because a backup +you have never restored is a hypothesis, not a backup. + +## Quick start + +```text +1. Create an empty PRIVATE GitHub repo (e.g. ha-config). +2. Follow references/setup.md: deploy key → scripts to /config/.git-sync/bin → install.sh. +3. Add assets/configuration_snippet.yaml + assets/automations.yaml, restart HA (full restart). +4. Circuit 2: age key + fine-grained PAT (Contents RW, ONE repo) → backup_offsite.sh. +5. Run the first restore drill immediately (references/restore-runbook.md). +``` + +Start with [SKILL.md](SKILL.md) — it is written as a Claude skill (drop it into +`.claude/skills/` and Claude will walk you through installation, incidents and drills), but it +reads just as well as plain documentation for humans. + +> Field-tested: the secret scanner ships canary-tested (see LESSON-05 — a BusyBox grep quirk +> silently disabled the private-key pattern until a planted-secret test caught it). diff --git a/skills/ha-git-backup/SKILL.md b/skills/ha-git-backup/SKILL.md new file mode 100644 index 0000000..4e0be2d --- /dev/null +++ b/skills/ha-git-backup/SKILL.md @@ -0,0 +1,102 @@ +--- +name: ha-git-backup +description: >- + Two-circuit backup & GitHub sync system for Home Assistant. Circuit 1 — real git inside + /config with a deploy key, a pre-commit secret scanner and meaningful commits; Circuit 2 — + full encrypted HA backups in GitHub Releases. Use EVERY time the task involves: backing up + Home Assistant, syncing its config, git in /config, restoring a config, rolling back a YAML + change, "what broke my automation", a leaked secret, setting up a deploy key, migrating HA + to new hardware, GitHub Releases for backups — and BEFORE any HA Core/OS update or AFTER any + incident with config loss. Also trigger on: backup, restore, gitleaks, secrets.yaml, 3-2-1. + The skill ships ready-made scripts (install, sync, secret-scan, offsite, restore) and a + mandatory protocol — do NOT reinvent sync logic, use scripts/. +--- + +# HA Git Backup — a two-circuit system + +Philosophy: **git gives you history, a backup gives you recovery. These are different jobs — +different circuits solve them.** Projects like GithubConfigSync mix the two and implement git +on top of the Contents API — we use real git. + +## Architecture + +``` +CIRCUIT 1: CONFIG HISTORY (what changed and when) +/config (git repo) ──deploy key──▶ GitHub private repo (ha-config) + ├── triggers: daily 03:30 / pre-update / button / HA restart + ├── secret protection: .gitignore → pre-commit scan → age mirror + └── commit: "[trigger] N files | HA 2026.7.1" + file list + +CIRCUIT 2: RECOVERY (full state, including .storage) +HA native backup (.tar) ──age──▶ GitHub Releases (ha-config) + ├── schedule: weekly + before updates + ├── rotation: 8 releases + └── restore drill: 1st of the month (mandatory) +``` + +The 3-2-1 rule: the config lives (1) on the HA host, (2) in the GitHub git repo, (3) as a full +tarball in Releases — plus a local NAS copy if you have one. + +## IRON RULES + +1. **Deploy key, not a PAT.** Fine-grained WRITE access to ONE repository. A classic token with + the `repo` scope = access to all your private repos → forbidden. +2. **A secret in the diff blocks the commit.** The pre-commit hook runs `secret_scan.sh`. + Bypassing with `--no-verify` — only deliberately, with the reason logged. +3. **`.storage/` does NOT go to git** (auth, tokens), except the `lovelace*` whitelist — + dashboards. The full `.storage` lives only in the encrypted Circuit-2 tarball. +4. **A backup without a restore drill is not a backup.** Monthly: download the release → + `age -d` → `tar -t` → verify `.storage/core.config_entries` is inside. +5. **Do not rewrite the scripts from scratch.** The logic (lock, retry, notify, rotation) is + already in `scripts/` — read and use them. + +## Quick start (fresh install) + +1. Create an empty private repo `ha-config` on GitHub. +2. Read `references/setup.md` — step by step: deploy key, install via the SSH add-on. +3. Run `scripts/install.sh` inside HA — it does git init, .gitignore, hook, remote, first dry-run. +4. Add the contents of `assets/configuration_snippet.yaml` to configuration.yaml and + `assets/automations.yaml` to your automations. **A full HA restart is required** + (`shell_command` and `command_line` are NOT loaded by a quick reload). +5. Circuit 2: `scripts/backup_offsite.sh` needs a fine-grained PAT + an age key. + See `references/setup.md` §4. +6. Run the first restore drill IMMEDIATELY — before the system is ever needed. + +## Symptom → action map + +| Symptom | Action | +|---|---| +| "Worked yesterday, broken today" | `git log --oneline -10`, `git diff HEAD~1` in /config — see what changed | +| Broke YAML, HA won't start | `git checkout -- ` or `git reset --hard `, restart | +| Secret leaked into the repo | `references/incident-secret-leak.md` — rotate the secret FIRST, history second | +| Push silent/failing | `/config/.git-sync/log`; typical: read-only deploy key, stale known_hosts | +| Migrating to new hardware | `scripts/restore.sh` — the Circuit-2 tarball, NOT the git repo (git has no .storage) | +| Sync sensor = error | Check the log; the automation already sends a persistent_notification | +| Repo bloated | Check binaries (db, tar) are gitignored; history is cleaned with `git gc` | + +## What's in the skill + +- `scripts/install.sh` — bootstrap the git repo in /config +- `scripts/ha_git_sync.sh` — the sync engine: lock → scan → commit → push with retry → status file +- `scripts/secret_scan.sh` — diff-based secret scanner (keys, tokens, private keys, passwords not via !secret) +- `scripts/backup_offsite.sh` — Circuit 2: age encryption + upload to GitHub Releases + rotation +- `scripts/restore.sh` — interactive restore with a checklist +- `assets/gitignore.template`, `assets/configuration_snippet.yaml`, `assets/automations.yaml` +- `references/setup.md` — full installation from scratch (deploy key, age, PAT) +- `references/architecture.md` — design rationale, comparison with alternatives +- `references/incident-secret-leak.md` — the secret-leak incident protocol +- `references/restore-runbook.md` — restore runbook and the monthly drill + +## LESSONS (recorded — do not repeat) + +- **LESSON-01**: the GithubConfigSync approach (files one by one via the Contents API) = a commit + per file, rate limits, snapshot duplicates in the git repo. Real git solves all of that for free. +- **LESSON-02**: .gitignore is NOT protection. A file added before the rule keeps being tracked. + Hence the mandatory pre-commit scan. +- **LESSON-03**: a config git repo ≠ a backup: without .storage a restore gives you a bare HA + with no entity registry, no auth, no UI dashboards. +- **LESSON-04**: a restore drill in calm times costs 10 minutes. A first-ever restore during an + outage without a drill costs an evening and your nerves. +- **LESSON-05**: BusyBox grep treats patterns starting with `-` as options — always pass scan + patterns with `grep -E -e "$pattern"`. The scanner was silently skipping its private-key + pattern until this was caught by a canary test. Test your scanner with planted secrets. diff --git a/skills/ha-git-backup/SKILL.ru.md b/skills/ha-git-backup/SKILL.ru.md new file mode 100644 index 0000000..8a20d0f --- /dev/null +++ b/skills/ha-git-backup/SKILL.ru.md @@ -0,0 +1,76 @@ +# HA Git Backup — двухконтурная система (русская версия; каноническая — [SKILL.md](SKILL.md)) + +Философия: **git даёт историю, бэкап даёт восстановление. Это разные задачи — их решают разные +контуры.** Проекты типа GithubConfigSync смешивают их и реализуют git поверх Contents API — мы +используем настоящий git. + +## Архитектура + +``` +КОНТУР 1: ИСТОРИЯ КОНФИГА (что изменилось и когда) +/config (git repo) ──deploy key──▶ GitHub private repo (ha-config) + ├── триггеры: daily 03:30 / pre-update / кнопка / рестарт HA + ├── защита секретов: .gitignore → pre-commit scan → age-зеркало + └── коммит: "[trigger] N files | HA 2026.7.1" + список файлов + +КОНТУР 2: ВОССТАНОВЛЕНИЕ (полное состояние, включая .storage) +HA native backup (.tar) ──age──▶ GitHub Releases (ha-config) + ├── расписание: еженедельно + перед обновлениями + ├── ротация: 8 релизов + └── restore drill: 1-е число месяца (обязательный) +``` + +Правило 3-2-1: конфиг живёт (1) на хосте HA, (2) в git-репо GitHub, (3) полный тарбол в +Releases + локальный NAS при наличии. + +## ЖЕЛЕЗНЫЕ ПРАВИЛА + +1. **Deploy key, не PAT.** Fine-grained доступ на ЗАПИСЬ в ОДИН репозиторий. Classic-токен со + scope `repo` = доступ ко всем приватным репо → запрещён. +2. **Секрет в diff = коммит блокируется.** Pre-commit hook вызывает `secret_scan.sh`. Обход + через `--no-verify` — только осознанно и с записью причины в лог. +3. **`.storage/` НЕ идёт в git** (auth, токены), кроме whitelist `lovelace*` — дашборды. + Полное `.storage` живёт только в шифрованном тарболе Контура 2. +4. **Бэкап без restore drill — не бэкап.** Раз в месяц: скачать релиз → `age -d` → `tar -t` → + проверить наличие `.storage/core.config_entries`. +5. **Не переписывать скрипты с нуля.** Логика (lock, retry, notify, ротация) уже в `scripts/` + — читать и использовать их. + +## Быстрый старт (новая установка) + +1. Создать приватный репо `ha-config` на GitHub (пустой). +2. Прочитать `references/setup.md` — там пошагово: deploy key, установка через SSH add-on. +3. Запустить `scripts/install.sh` внутри HA — он делает git init, .gitignore, hook, remote, + первый dry-run. +4. Добавить `assets/configuration_snippet.yaml` в configuration.yaml и `assets/automations.yaml` + в автоматизации. **Нужен ПОЛНЫЙ перезапуск HA** (quick-reload не грузит + `shell_command`/`command_line`). +5. Контур 2: `scripts/backup_offsite.sh` требует fine-grained PAT + age-ключ. + См. `references/setup.md` §4. +6. Провести первый restore drill СРАЗУ — до того, как система понадобится. + +## Карта симптомов → действия + +| Симптом | Действие | +|---|---| +| «Вчера работало, сегодня нет» | `git log --oneline -10`, `git diff HEAD~1` в /config | +| Сломал YAML, HA не стартует | `git checkout -- ` или `git reset --hard `, рестарт | +| Утёк секрет в репо | `references/incident-secret-leak.md` — ротация секрета ПЕРВОЙ | +| Push молчит/падает | `/config/.git-sync/log`; типовое: deploy key read-only, протух known_hosts | +| Переезд на новое железо | `scripts/restore.sh` — тарбол Контура 2, НЕ git-репо (в git нет .storage) | +| Sync-сенсор = error | Смотреть лог; автоматизация шлёт persistent_notification | +| Репо распух | Бинарники (db, tar) в .gitignore; история чистится `git gc` | + +## УРОКИ (зафиксированы, не повторять) + +- **УРОК-01**: GithubConfigSync-подход (файлы по одному через Contents API) = коммит на файл, + rate limits, снапшоты-дубли. Настоящий git решает всё это бесплатно. +- **УРОК-02**: .gitignore — НЕ защита. Файл, добавленный до правила, продолжает трекаться. + Отсюда обязательный pre-commit scan. +- **УРОК-03**: git-репо конфига ≠ бэкап: без .storage восстановление даёт голый HA без entity + registry, авторизаций и дашбордов из UI. +- **УРОК-04**: restore drill в спокойное время стоит 10 минут. Первое восстановление во время + аварии без drill стоит вечер и нервы. +- **УРОК-05**: BusyBox-grep принимает паттерн, начинающийся с `-`, за опцию — паттерны сканера + передавать только как `grep -E -e "$pattern"`. Сканер молча пропускал проверку приватных + ключей, пока это не поймал тест с подсадной уткой. Проверяй сканер подсадными секретами. diff --git a/skills/ha-git-backup/assets/automations.yaml b/skills/ha-git-backup/assets/automations.yaml new file mode 100644 index 0000000..3aee0ab --- /dev/null +++ b/skills/ha-git-backup/assets/automations.yaml @@ -0,0 +1,70 @@ +# ha-git-backup: automations. Add to automations.yaml or via the UI. + +- alias: "GitSync - daily commit" + triggers: + - trigger: time + at: "03:30:00" + actions: + - action: shell_command.git_sync_daily + +- alias: "GitSync - button" + triggers: + - trigger: state + entity_id: input_button.git_sync_now + actions: + - action: shell_command.git_sync_manual + +- alias: "GitSync - tag before a Core update" + triggers: + - trigger: state + entity_id: update.home_assistant_core_update + to: "on" + actions: + - action: shell_command.git_sync_pre_update + +- alias: "GitSync - commit after restart (capture the surviving state)" + triggers: + - trigger: homeassistant + event: start + actions: + - delay: "00:03:00" + - action: shell_command.git_sync_manual + +- alias: "GitSync - alert on failure" + triggers: + - trigger: state + entity_id: sensor.git_sync_status + conditions: + - condition: template + value_template: "{{ 'error' in states('sensor.git_sync_status') }}" + actions: + - action: persistent_notification.create + data: + title: "⚠️ Git Sync FAILED" + message: "{{ states('sensor.git_sync_status') }} — see /config/.git-sync/log" + # + add your notify.mobile_app_* here + +- alias: "Backup - weekly full + offsite" + triggers: + - trigger: time + at: "04:00:00" + conditions: + - condition: time + weekday: [sun] + actions: + - action: backup.create # native HA backup (including .storage) + - delay: "00:10:00" + - action: shell_command.backup_offsite + +- alias: "Backup - restore drill reminder" + triggers: + - trigger: time + at: "10:00:00" + conditions: + - condition: template + value_template: "{{ now().day == 1 }}" + actions: + - action: persistent_notification.create + data: + title: "📋 Restore drill" + message: "1st of the month: verify recovery (restore.sh full, DRILL mode)." diff --git a/skills/ha-git-backup/assets/configuration_snippet.yaml b/skills/ha-git-backup/assets/configuration_snippet.yaml new file mode 100644 index 0000000..b17d73d --- /dev/null +++ b/skills/ha-git-backup/assets/configuration_snippet.yaml @@ -0,0 +1,21 @@ +# ha-git-backup: add to configuration.yaml +# Paths assume scripts live in /config/.git-sync/bin — adjust otherwise. +# NOTE: shell_command and command_line load only on a FULL HA restart. + +shell_command: + git_sync_manual: "/config/.git-sync/bin/ha_git_sync.sh manual" + git_sync_daily: "/config/.git-sync/bin/ha_git_sync.sh daily" + git_sync_pre_update: "/config/.git-sync/bin/ha_git_sync.sh pre-update" + backup_offsite: "GH_REPO={{ 'YOUR_USER/ha-config' }} /config/.git-sync/bin/backup_offsite.sh" + +# Health sensor: reads the status file written by ha_git_sync.sh +command_line: + - sensor: + name: git_sync_status + command: "cat /config/.git-sync/last_status 2>/dev/null || echo 'never'" + scan_interval: 300 + +input_button: + git_sync_now: + name: "Git Sync Now" + icon: mdi:source-branch-sync diff --git a/skills/ha-git-backup/assets/gitignore.template b/skills/ha-git-backup/assets/gitignore.template new file mode 100644 index 0000000..e34cad6 --- /dev/null +++ b/skills/ha-git-backup/assets/gitignore.template @@ -0,0 +1,43 @@ +# ha-git-backup: echelon 1 of protection. Do NOT loosen without reading SKILL.md +# --- secrets and state --- +secrets.yaml +*.pem +*.key +known_devices.yaml +.storage/* +!.storage/lovelace* +.cloud/ +.google.token + +# --- databases and runtime junk (no place in git; Circuit 2 covers them) --- +*.db +*.db-* +*.log +*.log.* +home-assistant.log* +OZW_Log.txt +deps/ +tts/ +image/ +www/community/ +__pycache__/ +.cache/ +.ha_run.lock + +# --- backups are not stored in the git repo --- +backups/ +*.tar +*.tar.age + +# --- embedded git repos keep their own history (content is covered by Circuit 2) --- +esphome/ + +# --- this system's own runtime files --- +.git-sync/log +.git-sync/last_status +.git-sync/last_offsite +.git-sync/gh_token +.git-sync/age_recipient +.git-sync/deploy_key +.git-sync/deploy_key.pub +.HA_VERSION diff --git a/skills/ha-git-backup/references/architecture.md b/skills/ha-git-backup/references/architecture.md new file mode 100644 index 0000000..1dc8b9f --- /dev/null +++ b/skills/ha-git-backup/references/architecture.md @@ -0,0 +1,41 @@ +# Architecture: design rationale + +## Why two circuits, not one +Git answers "what changed and when" (diffs, blame, file rollback in seconds). A backup answers +"how do I bring the system up from nothing" (.storage: entity registry, auth, config entries, +the zigbee network). Mixing the two jobs produces a system that does both badly — the main +lesson from analysing GithubConfigSync. + +## Comparison with alternatives +| Solution | History | Full recovery | Secrets | Verdict | +|---|---|---|---|---| +| **This system** | real git | Releases + age | 3 echelons | ✅ | +| GithubConfigSync | pseudo-git (Contents API, commit per file) | no (.storage excluded) | .gitignore only | history — yes, restore — no | +| Git Pull add-on (official) | pull only | no | — | a different job (deploy) | +| Google Drive Backup add-on | no | yes | HA encryption | great as an ADDITION to Circuit 2 (second offsite copy = full 3-2-1) | +| Bare cron+git | yes | no | manual | that IS Circuit 1, but without scan/notify/lock | + +## Key decisions +1. **Deploy key instead of a PAT for git** — blast radius of one repo. +2. **Fine-grained PAT only for the Releases API** (Contents RW, one repo) — Releases can't be + pushed over SSH. +3. **The pre-commit scan is mandatory**: .gitignore does not protect against a secret written + INTO automations.yaml as a literal (a real, frequent case: a Telegram bot token in a + rest_command). +4. **Whitelist `.storage/lovelace*`**: dashboards created in the UI live in .storage — without + them the config history is incomplete. The rest of .storage holds auth → encrypted Circuit 2 + only. +5. **age, not GPG**: a single binary, no keyring state, the key is one line. There is NO private + key on the HA host — a compromised host ≠ readable backups. +6. **GitHub Releases, not git-lfs and not files in the repo**: binaries in git bloat history + forever; Releases is flat storage with a 2 GB/file limit and trivial rotation. +7. **A status file + command_line sensor, not push notifications from the script**: the script + does not depend on HA API tokens; observability is done by HA itself. +8. **Locking via flock**: an HA restart plus the daily trigger cannot race. + +## System boundaries (deliberate) +- The 2 GB Releases limit: exclude media/large DBs from the native backup (partial backup). +- The git repo in /config is visible to every add-on with config access — threat model: + trusted add-ons. +- Git history keeps deleted files forever: a leaked secret requires the + incident-secret-leak.md protocol, not a simple delete. diff --git a/skills/ha-git-backup/references/incident-secret-leak.md b/skills/ha-git-backup/references/incident-secret-leak.md new file mode 100644 index 0000000..68f815b --- /dev/null +++ b/skills/ha-git-backup/references/incident-secret-leak.md @@ -0,0 +1,32 @@ +# Incident: a secret landed in the git repo + +The order is STRICT. Step 1 comes before everything else. + +## 1. ROTATE THE SECRET (immediately) +A leaked token/password/key counts as compromised from the moment of push, even in a private +repo. Revoke and reissue it at the provider (GitHub token → revoke; a service API key → +regenerate; a password → change). Only then move on to cleanup. + +## 2. Assess the exposure +- `git log --all -S '' --oneline` — which commits carry it. +- Is the repo private? Any forks/clones? Who had access? + +## 3. Rewrite history +```bash +# on your workstation, not on HA: +git clone git@github.com:YOUR_USER/ha-config.git && cd ha-config +pip install git-filter-repo +git filter-repo --replace-text <(echo '==>REDACTED') +git push --force --all && git push --force --tags +``` +On HA afterwards: `cd /config && git fetch && git reset --hard origin/main`. + +## 4. GitHub specifics +A force-push does NOT delete objects instantly: commits stay reachable by SHA via cache/API. +For a full purge — GitHub Support (a request to delete unreachable objects). One more reason +step 1 is primary: history may never be fully scrubbed; rotation closes the risk with certainty. + +## 5. Close the gap +- Why didn't the scan catch it? → add the pattern to `secret_scan.sh` PATTERNS. +- The value should have lived in secrets.yaml behind `!secret` — move it. +- Record a LESSON in SKILL.md. diff --git a/skills/ha-git-backup/references/restore-runbook.md b/skills/ha-git-backup/references/restore-runbook.md new file mode 100644 index 0000000..211540e --- /dev/null +++ b/skills/ha-git-backup/references/restore-runbook.md @@ -0,0 +1,38 @@ +# Restore runbook + +## Scenario A: a broken file/config (frequent) — Circuit 1 +```bash +cd /config +git log --oneline -10 # find the last known-good commit +git diff --stat # what changed +git checkout -- automations.yaml # surgical rollback +ha core check && ha core restart +``` +Time: minutes. If HA won't start at all — same commands via the SSH add-on (it lives +independently of Core). + +## Scenario B: dead SD/SSD/hardware — Circuit 2 +1. Fresh HA OS install on the new medium. +2. On your workstation: download the latest `backup/*` release from `ha-config`, decrypt: + `age -d -i age_key.txt -o restored.tar ha-backup-*.tar.age` +3. New HA onboarding → "Restore from backup" → upload restored.tar. +4. After start verify: entity registry intact, auth alive, dashboards present, ESPHome nodes + reconnected, zigbee/z-wave network up. +5. Verify Circuit 1: `cd /config && git status` (the repo arrives inside the backup, .git and all). + +## Scenario C: need a year-old config +Circuit 1: `git log --before="2025-07-01" -1` → checkout the needed file from that SHA. + +## The monthly DRILL (1st of the month, reminder ships in automations.yaml) +The goal — prove the "Releases → age → tar" chain is alive WITHOUT a real restore: +1. Download the latest release. +2. `age -d` with the private key (this doubles as "the key isn't lost" check!). +3. `tar -tf restored.tar | grep -c .` — the archive reads. +4. `tar -tf restored.tar | grep core.config_entries` — .storage is inside. +5. `date +%F > /config/.git-sync/last_drill` +Any step failing = an incident: fix now, not when the SSD burns. + +## System health metrics +- `sensor.git_sync_status` = ok, age < 26 h +- `.git-sync/last_offsite` — age < 8 days +- `.git-sync/last_drill` — age < 35 days diff --git a/skills/ha-git-backup/references/setup.md b/skills/ha-git-backup/references/setup.md new file mode 100644 index 0000000..fc33a91 --- /dev/null +++ b/skills/ha-git-backup/references/setup.md @@ -0,0 +1,61 @@ +# Installation from scratch + +## §1. Prerequisites +- HA OS / Supervised with the **Advanced SSH & Web Terminal** add-on (it ships git and ssh). +- A private repo `ha-config` on GitHub (empty, no README). +- Copy the skill's scripts to `/config/.git-sync/bin/` and the gitignore template to + `/config/.git-sync/assets/`, then `chmod +x` the scripts. +- Note: with `sftp: false` in the add-on config, `scp` won't work — use a tar pipe: + `tar cz scripts | ssh 'cd /tmp && tar xz'`. + +## §2. Deploy key (Circuit 1) — NOT a PAT +```bash +ssh-keygen -t ed25519 -f /config/.git-sync/deploy_key -N "" -C "ha-git-backup" +cat /config/.git-sync/deploy_key.pub +``` +GitHub → repo `ha-config` → Settings → Deploy keys → Add key → **Allow write access**. + +SSH config so git picks exactly this key: +```bash +mkdir -p ~/.ssh && cat >> ~/.ssh/config <<'EOF' +Host github.com + IdentityFile /config/.git-sync/deploy_key + IdentitiesOnly yes +EOF +``` +Why a deploy key: it grants access to ONE repo. A compromised key ≠ a compromised GitHub +account (unlike a classic PAT with the `repo` scope). + +## §3. Circuit 1: install +```bash +/config/.git-sync/bin/install.sh git@github.com:YOUR_USER/ha-config.git +# review the dry-run file list for secrets, then: +/config/.git-sync/bin/ha_git_sync.sh manual +``` +Add `assets/configuration_snippet.yaml` and `assets/automations.yaml`, **fully restart HA** +(quick reload does not load `shell_command`/`command_line`), check `sensor.git_sync_status`. + +## §4. Circuit 2: offsite to GitHub Releases +1. **age key** (generate NOT on HA but on your workstation): + `age-keygen -o age_key.txt` → store the private key offline (password manager / paper). + Only the public half goes to HA: + `echo "age1..." > /config/.git-sync/age_recipient` +2. **Fine-grained PAT**: GitHub → Settings → Developer settings → Fine-grained tokens → + access to `ha-config` only, permission **Contents: Read and write**. 1-year expiry, + calendar reminder to rotate. + `echo "github_pat_..." > /config/.git-sync/gh_token && chmod 600 /config/.git-sync/gh_token` +3. `age` inside the SSH add-on container: add `age` to the add-on's `packages:` list (persists + across restarts) or `apk add age`. +4. Test: `GH_REPO=YOUR_USER/ha-config /config/.git-sync/bin/backup_offsite.sh` + +## §5. Optional: an age mirror of secrets.yaml (echelon 3) +To make the git repo self-sufficient for restoring the YAML part: +```bash +age -r "$(cat /config/.git-sync/age_recipient)" -o /config/secrets.yaml.age /config/secrets.yaml +``` +`secrets.yaml.age` is NOT gitignored → it goes to git. Add the command to ha_git_sync.sh before +`git add` if you enable this. Decryption requires only the private key, which is not on HA. + +## §6. Final — the mandatory first drill +Right after installation run `restore.sh full` in DRILL mode (steps 1–4). Record the date in +`/config/.git-sync/last_drill`. diff --git a/skills/ha-git-backup/scripts/backup_offsite.sh b/skills/ha-git-backup/scripts/backup_offsite.sh new file mode 100755 index 0000000..f91e376 --- /dev/null +++ b/skills/ha-git-backup/scripts/backup_offsite.sh @@ -0,0 +1,56 @@ +#!/usr/bin/env bash +# backup_offsite.sh — Circuit 2: freshest native HA backup → age encryption → +# GitHub Release in the ha-config repo. Rotation: keep the KEEP latest releases. +# Requires: age, curl, a fine-grained PAT (Contents RW of ONE repo) in +# /config/.git-sync/gh_token, the public age key in /config/.git-sync/age_recipient +set -euo pipefail + +CONFIG_DIR="${CONFIG_DIR:-/config}" +BACKUP_DIR="${BACKUP_DIR:-/backup}" # native backup path on supervised/HAOS +REPO="${GH_REPO:?export GH_REPO=user/ha-config}" +KEEP="${KEEP:-8}" +TOKEN="$(cat "$CONFIG_DIR/.git-sync/gh_token")" +RECIPIENT="$(cat "$CONFIG_DIR/.git-sync/age_recipient")" +API="https://api.github.com/repos/$REPO" +AUTH=(-H "Authorization: Bearer $TOKEN" -H "Accept: application/vnd.github+json") +LOG="$CONFIG_DIR/.git-sync/log" +log(){ echo "$(date '+%F %T') [offsite] $*" >> "$LOG"; } + +# 1. The freshest native backup (creating it is the HA automation's job, before this call) +LATEST="$(ls -t "$BACKUP_DIR"/*.tar 2>/dev/null | head -1)" +[ -n "$LATEST" ] || { log "FAIL: no .tar in $BACKUP_DIR"; exit 1; } + +# 2. Encrypt (the HA tarball may already be password-encrypted — age on top is fine) +STAMP="$(date +%Y-%m-%d_%H%M)" +ENC="/tmp/ha-backup-$STAMP.tar.age" +age -r "$RECIPIENT" -o "$ENC" "$LATEST" +SIZE=$(du -m "$ENC" | cut -f1) +[ "$SIZE" -lt 1900 ] || { log "FAIL: $SIZE MB > GitHub Releases limit (2GB) — exclude media from the backup"; rm -f "$ENC"; exit 1; } + +# 3. Create the release +TAG="backup/$STAMP" +REL_JSON="$(curl -sf "${AUTH[@]}" -X POST "$API/releases" \ + -d "{\"tag_name\":\"$TAG\",\"name\":\"HA backup $STAMP\",\"body\":\"age-encrypted, $SIZE MB\"}")" +REL_ID="$(echo "$REL_JSON" | grep -o '"id": *[0-9]*' | head -1 | grep -o '[0-9]*')" +[ -n "$REL_ID" ] || { log "FAIL: release create"; rm -f "$ENC"; exit 1; } + +# 4. Upload the asset +curl -sf "${AUTH[@]}" -H "Content-Type: application/octet-stream" \ + --data-binary @"$ENC" \ + "https://uploads.github.com/repos/$REPO/releases/$REL_ID/assets?name=$(basename "$ENC")" >/dev/null \ + || { log "FAIL: asset upload"; rm -f "$ENC"; exit 1; } +rm -f "$ENC" +log "OK: $TAG uploaded ($SIZE MB)" + +# 5. Rotation: delete backup/* releases beyond KEEP (and their tags) +curl -sf "${AUTH[@]}" "$API/releases?per_page=100" \ + | grep -oE '"tag_name": *"backup/[^"]*"|"id": *[0-9]*' \ + | paste - - | grep 'backup/' | sort -r | tail -n +$((KEEP+1)) \ + | while read -r idline tagline; do + RID="$(echo "$idline" | grep -o '[0-9]*')" + RTAG="$(echo "$tagline" | sed 's/.*"backup/backup/;s/"//g')" + curl -sf "${AUTH[@]}" -X DELETE "$API/releases/$RID" >/dev/null || true + curl -sf "${AUTH[@]}" -X DELETE "$API/git/refs/tags/$RTAG" >/dev/null || true + log "rotated out: $RTAG" + done +echo "ok: offsite $TAG" > "$CONFIG_DIR/.git-sync/last_offsite" diff --git a/skills/ha-git-backup/scripts/ha_git_sync.sh b/skills/ha-git-backup/scripts/ha_git_sync.sh new file mode 100755 index 0000000..bf880f9 --- /dev/null +++ b/skills/ha-git-backup/scripts/ha_git_sync.sh @@ -0,0 +1,65 @@ +#!/usr/bin/env bash +# ha_git_sync.sh — Circuit 1: git sync /config → GitHub +# Usage: ha_git_sync.sh [trigger] trigger: manual|daily|pre-update|restart +# Exit: 0 = ok/nothing-to-do, 1 = error (recorded in the status file) +set -uo pipefail + +CONFIG_DIR="${CONFIG_DIR:-/config}" +SYNC_DIR="$CONFIG_DIR/.git-sync" +LOG="$SYNC_DIR/log" +STATUS="$SYNC_DIR/last_status" # read by the HA command_line sensor +LOCKFILE="/tmp/ha_git_sync.lock" +TRIGGER="${1:-manual}" +BRANCH="${GIT_BRANCH:-main}" +RETRIES=3 + +mkdir -p "$SYNC_DIR" +log() { echo "$(date '+%F %T') [$TRIGGER] $*" >> "$LOG"; } +fail() { echo "error: $*" > "$STATUS"; log "FAIL: $*"; exit 1; } +ok() { echo "ok: $*" > "$STATUS"; log "OK: $*"; exit 0; } + +# --- lock: no parallel runs --------------------------------------------------- +exec 9>"$LOCKFILE" +flock -n 9 || fail "another sync is running" + +cd "$CONFIG_DIR" || fail "no $CONFIG_DIR" +[ -d .git ] || fail "not a git repo — run install.sh first" + +# --- any changes? -------------------------------------------------------------- +git add -A +if git diff --cached --quiet; then + ok "clean, nothing to commit" +fi + +# --- secret scanner (echelon 2) ----------------------------------------------- +SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)" +if ! "$SCRIPT_DIR/secret_scan.sh"; then + git reset -q # unstage everything, commit nothing + fail "SECRET DETECTED in staged diff — commit blocked, see $LOG" +fi + +# --- a meaningful commit -------------------------------------------------------- +HA_VER="unknown" +[ -f .HA_VERSION ] && HA_VER="$(cat .HA_VERSION)" +NFILES=$(git diff --cached --name-only | wc -l | tr -d ' ') +SUMMARY="[$TRIGGER] $NFILES file(s) | HA $HA_VER" +BODY="$(git diff --cached --stat | tail -20)" + +git commit -q -m "$SUMMARY" -m "$BODY" || fail "commit failed" +log "committed: $SUMMARY" + +# --- tag before an update ------------------------------------------------------- +if [ "$TRIGGER" = "pre-update" ]; then + TAG="pre-update/$(date +%Y-%m-%d_%H%M)" + git tag -f "$TAG" && log "tagged $TAG" +fi + +# --- push with retry and exponential backoff ------------------------------------ +n=0 +until git push -q origin "$BRANCH" --tags 2>>"$LOG"; do + n=$((n+1)) + [ $n -ge $RETRIES ] && fail "push failed after $RETRIES attempts (commit is local, next run will push it)" + sleep $((5 * 2 ** n)) +done + +ok "$SUMMARY pushed" diff --git a/skills/ha-git-backup/scripts/install.sh b/skills/ha-git-backup/scripts/install.sh new file mode 100755 index 0000000..d2df300 --- /dev/null +++ b/skills/ha-git-backup/scripts/install.sh @@ -0,0 +1,57 @@ +#!/usr/bin/env bash +# install.sh — Circuit 1 bootstrap: a git repo in /config. +# Run ONCE from the SSH add-on. Requires: git, ssh, a created private repo and +# a deploy key (see references/setup.md §2). Idempotent. +set -euo pipefail + +CONFIG_DIR="${CONFIG_DIR:-/config}" +REMOTE="${1:-}" # git@github.com:YOUR_USER/ha-config.git +BRANCH="${GIT_BRANCH:-main}" +SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)" + +[ -n "$REMOTE" ] || { echo "Usage: install.sh git@github.com:YOUR_USER/ha-config.git"; exit 1; } +command -v git >/dev/null || { echo "git not found — use the Advanced SSH & Web Terminal add-on"; exit 1; } + +cd "$CONFIG_DIR" +mkdir -p .git-sync + +# 1. The repository +if [ ! -d .git ]; then + git init -q -b "$BRANCH" + echo "[+] git init ($BRANCH)" +fi +git config user.name "ha-git-backup" +git config user.email "ha-git-backup@config.local" +git remote get-url origin >/dev/null 2>&1 && git remote set-url origin "$REMOTE" || git remote add origin "$REMOTE" + +# 2. .gitignore (echelon 1) — never overwrite an existing one +if [ ! -f .gitignore ]; then + cp "$SCRIPT_DIR/../assets/gitignore.template" .gitignore + echo "[+] .gitignore installed" +else + echo "[i] .gitignore already exists — diff it against assets/gitignore.template manually" +fi + +# 3. pre-commit hook (echelon 2) +mkdir -p .git/hooks +cat > .git/hooks/pre-commit </dev/null || true +chmod +x "$SCRIPT_DIR"/*.sh +echo "[+] pre-commit secret scan wired up" + +# 4. SSH access check (deploy key) +if ssh -o BatchMode=yes -o StrictHostKeyChecking=accept-new -T git@github.com 2>&1 | grep -q "successfully authenticated"; then + echo "[+] deploy key works" +else + echo "[!] SSH to GitHub failed — check the deploy key (references/setup.md §2)" +fi + +# 5. First pass: show WHAT would enter the repo, before any real commit +echo "[i] DRY-RUN — these files would land in the first commit:" +git add -A -n | head -50 +echo "..." +echo "Review the list above for secrets. If it looks right, run:" +echo " $SCRIPT_DIR/ha_git_sync.sh manual" diff --git a/skills/ha-git-backup/scripts/restore.sh b/skills/ha-git-backup/scripts/restore.sh new file mode 100755 index 0000000..214885d --- /dev/null +++ b/skills/ha-git-backup/scripts/restore.sh @@ -0,0 +1,35 @@ +#!/usr/bin/env bash +# restore.sh — recovery. Two scenarios: +# restore.sh file — roll back files from git (Circuit 1) +# restore.sh full — full restore from a GitHub Release (Circuit 2) +# Also used for the monthly restore drill (references/restore-runbook.md) +set -euo pipefail +MODE="${1:-help}" + +case "$MODE" in +file) + cd "${CONFIG_DIR:-/config}" + echo "Recent commits:"; git log --oneline -15 + echo + echo "Roll back one file: git checkout -- " + echo "Roll back everything: git reset --hard (DESTRUCTIVE: wipes uncommitted changes)" + echo "What changed: git diff --stat" + echo "After rollback: validate the config (ha core check) and restart HA." + ;; +full) + REPO="${GH_REPO:?export GH_REPO=user/ha-config}" + KEYFILE="${AGE_KEY:?export AGE_KEY=/path/to/age_key.txt}" + echo "=== FULL RESTORE (Circuit 2) ===" + echo "1. List backups: gh release list -R $REPO (or the /releases API)" + echo "2. Download asset: gh release download -R $REPO -D /tmp" + echo "3. Decrypt: age -d -i $KEYFILE -o restored.tar /tmp/ha-backup-*.tar.age" + echo "4. Verify: tar -tf restored.tar | grep -c . && tar -tf restored.tar | grep core.config_entries" + echo "5. HAOS: place restored.tar in /backup and restore via the UI" + echo " (Settings → System → Backups), or onboarding-restore on a fresh install." + echo "6. After start: verify entity registry, auth, dashboards, ESPHome nodes." + echo + echo "DRILL mode: run steps 1-4 WITHOUT step 5, record the date in .git-sync/last_drill" + ;; +*) + echo "Usage: restore.sh file|full"; exit 1;; +esac diff --git a/skills/ha-git-backup/scripts/secret_scan.sh b/skills/ha-git-backup/scripts/secret_scan.sh new file mode 100755 index 0000000..8410f5a --- /dev/null +++ b/skills/ha-git-backup/scripts/secret_scan.sh @@ -0,0 +1,56 @@ +#!/usr/bin/env bash +# secret_scan.sh — echelon 2 of protection: scans the STAGED diff for secrets. +# Exit: 0 = clean, 1 = secret found (the commit must be blocked). +# Also installed as the pre-commit hook (install.sh wires it up). +set -uo pipefail + +CONFIG_DIR="${CONFIG_DIR:-/config}" +LOG="$CONFIG_DIR/.git-sync/log" +ALLOWLIST="$CONFIG_DIR/.git-sync/scan_allowlist" # exclusion lines (one per line) + +# Only ADDED diff lines, without headers +DIFF="$(git diff --cached --unified=0 | grep -E '^\+' | grep -vE '^\+\+\+' || true)" +[ -z "$DIFF" ] && exit 0 + +# Apply the allowlist (false positives are recorded THERE, not by disabling the scanner) +if [ -f "$ALLOWLIST" ]; then + while IFS= read -r line; do + [ -n "$line" ] && DIFF="$(echo "$DIFF" | grep -vF "$line" || true)" + done < "$ALLOWLIST" +fi + +PATTERNS=( + # private keys + '-----BEGIN (RSA |EC |OPENSSH |DSA |PGP )?PRIVATE KEY' + # cloud / API tokens of known formats + 'AKIA[0-9A-Z]{16}' # AWS access key + 'ghp_[A-Za-z0-9]{36}' # GitHub classic PAT + 'github_pat_[A-Za-z0-9_]{22,}' # GitHub fine-grained PAT + 'xox[baprs]-[A-Za-z0-9-]{10,}' # Slack + 'AIza[0-9A-Za-z_-]{35}' # Google API key + 'sk-[A-Za-z0-9_-]{20,}' # OpenAI/Anthropic-style + 'eyJhbGciOi[A-Za-z0-9_-]{20,}' # JWT / HA long-lived token + # a secret-looking field assigned a literal value NOT via !secret + '(password|api_key|token|client_secret|bearer)["'"'"']?\s*[:=]\s*["'"'"']?[A-Za-z0-9_\-\.\+/=]{8,}' +) + +HIT="" +for p in "${PATTERNS[@]}"; do + # NOTE: "-e" is mandatory — BusyBox grep treats a pattern starting with "-" as an + # option and silently skips it otherwise (LESSON-05, caught by a canary test). + M="$(echo "$DIFF" | grep -inE -e "$p" | grep -viE '!secret|REDACTED|example|changeme|password_here' || true)" + [ -n "$M" ] && HIT="$HIT +[$p] +$M" +done + +if [ -n "$HIT" ]; then + { + echo "$(date '+%F %T') SECRET-SCAN BLOCKED COMMIT:" + echo "$HIT" | head -40 + echo "--- False positive? Add the exact line to $ALLOWLIST" + } >> "$LOG" 2>/dev/null || true + echo "SECRET DETECTED — commit blocked. Details: $LOG" >&2 + exit 1 +fi +exit 0