diff --git a/CHANGELOG.md b/CHANGELOG.md index 1fb99ed..ff880dc 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -8,6 +8,30 @@ See each [GitHub Release](https://github.com/TheGrandWazoo/truenas-proxmox/relea --- +## [Unreleased] — 4.0.3 + +### Fixed +- **WebSocket calls failed after the connection sat idle in a PVE daemon** + (#303). TrueNAS closes an idle WebSocket after 30–60 s (measured on SCALE + 25.04 and 25.10). Long-lived `pvedaemon` workers reused their cached + connection after that and died with `closed unexpectedly during call + 'pool.dataset.query'`. That failed VM creates (once leaving a VM in + `lock: create`), returned HTTP 500 on content listings, and made `status()` + report SCALE storages as inactive at random. The CLI was unaffected + because each `qm`/`pvesm` is a fresh process. + - Before every reuse, a cached connection must have nothing pending on + the socket, and if it has been idle for more than 10 s it must answer + `core.ping` within 5 s. Otherwise the plugin reconnects. A ping takes + 2–4 ms, compared with 1–4 s for a reconnect and login. + - Read-only calls (`*.query`, `*.config`) that still hit a closed socket + reconnect and retry once. Writes are never retried. + +### Verified +- Read-only idle probe on pve01: 4.0.2 fails after 60 s and 90 s of idle + time on SCALE 25.04 and 25.10. A long-lived process with the fix ran + `status` plus `list_images` every 45 s for 5 minutes against both hosts, + 14/14 OK. + ## [4.0.2] — 2026-10-13 Volume lifecycle fixes. Every change is backward compatible and `storage.cfg` is unchanged. diff --git a/README.md b/README.md index 2d278ca..7f5cb13 100644 --- a/README.md +++ b/README.md @@ -648,6 +648,27 @@ ls /etc/pve/nodes/*/qemu-server/.conf 2>/dev/null # no output → VM no lon To free an orphan, use `pvesm free :`. On 4.0.1 and earlier, use the `vdisk_free` one-liner from the entry above. Freeing a VM's last disk also removes its now-empty per-VM target. +### "WebSocket connection ... closed unexpectedly", or a SCALE storage flapping to inactive (4.0.0–4.0.2) + +**Symptom:** on TrueNAS SCALE 25.04+, requests through the Proxmox API, GUI or Terraform fail at random. Retrying sometimes helps. The error looks like this: + +``` +TrueNAS WebSocket connection to closed unexpectedly during call 'pool.dataset.query' +``` + +Other symptoms: +- VM creation fails, sometimes leaving the VM in `lock: create`. +- Content listings return HTTP 500. +- The storage shows `active: 0` for some requests and `active: 1` for others. +- `qm`/`pvesm` on the command line work fine. + +**Cause:** TrueNAS closes an idle WebSocket after 30–60 s. Each long-lived `pvedaemon` worker kept its cached connection and reused it after TrueNAS had closed it. **Fixed in v4.0.3** ([#303](https://github.com/TheGrandWazoo/freenas-proxmox/issues/303)). Now a cached connection is checked before every reuse and pinged if it has been idle, and read-only calls reconnect and retry once. TrueNAS CORE and SCALE < 25.04 (REST) were never affected. + +**On 4.0.0–4.0.2:** +- Upgrade. +- If you can't upgrade yet, `systemctl restart pvedaemon` clears the stale connections, but only until they go idle again. +- A VM left in `lock: create` with no disks can be removed with `qm unlock && qm destroy `. + ### Filing a Bug Report Please use the [GitHub issue tracker](https://github.com/TheGrandWazoo/truenas-proxmox/issues) and include the [bug report template](.github/ISSUE_TEMPLATE/bug_report.md). Include relevant log lines from `syslog` (search for `FreeNAS::`).