Full alloc_image/activate_volume/path/deactivate_volume/free_image
cycle against .92 with zero code changes to TrueNASMultipath.pm --
real dm-multipath device came up with 2 active paths. Also notes a
practical finding: .92 has only one TrueNAS portal object (0.0.0.0)
rather than two per-IP ones, and iscsiadm still produced two genuine
independent paths logging into different IPs against it.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Ran the full read/write/snapshot suite against .90 (TrueNAS CORE,
real production box) to confirm _transport()'s dispatcher wrapper
doesn't regress the REST path for hosts that never touch WebSocket.
Zero regressions, confirmed back to original state afterward.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Every read and write API call the WebSocket transport makes is
live-verified against real TrueNAS SCALE hosts. Kevin's sign-off.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Every API call the WebSocket transport makes is now live-verified
against a real TrueNAS SCALE host with zero param-shape issues
remaining. Left as Draft pending Kevin's own sign-off, per his
explicit instruction earlier.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
alloc_image/path/volume_size_info/free_image confirmed working
end-to-end against .92 after fixing the delete param-shape bugs.
pool.dataset.create/delete both confirmed along the way. Only
zfs.snapshot.* remains entirely untested against the WS transport.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
_api_ws/_ws_connect/_ws_call/_transport landed on next, live-verified
against .92 (25.04.2.6) through the real plugin module. Also notes
the /system/product_type bug caught during integration (returns
license tier, not product family -- fixed by parsing /system/version
magnitude instead).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Kevin rebuilt .92 from scratch specifically to TrueNAS-25.04.2.6 -- the
exact SCALE minor version NAS-135643 was originally filed against, two
patch levels later. iscsi.target.query returned a real target cleanly,
no error. This was the one item ADR-012 flagged as an acknowledged gap
rather than a completed check; no remaining live-verification gaps for
this ADR now. Also documents a minor auth-flow gotcha found along the
way: retrying auth.login_ex on the same connection after a failed
attempt threw an unhandled RuntimeError on this fresh install,
reinforcing the login_with_api_key recommendation.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
release/3.x -> main, release/4.x -> next, master -> archive/v2-legacy
(retired, was byte-identical to release/2.x). Branch identity is now
role-based, not version-numbered -- main and next are fixed names
that never get renamed at a major-version cutover; release/N.x is
reserved exclusively for already-superseded majors, created only at
the moment they're archived.
This directly fixes the structural cause of the bug found 2026-08-30:
build.yml hardcoded literal branch-name comparisons for the
testing/beta channel, and ADR-006's original "master -> beta channel"
rule was never superseded when release/3.x took over as the active
branch, leaving master as a silent, stale duplicate of release/2.x
that CI still treated as a legitimate publish source. Rewrote the
version-resolution logic to match roles instead: refs/heads/main,
refs/heads/next, and a refs/heads/release/*.x pattern that auto-
catches any future archived major with zero code changes needed.
Also tightened the push/PR triggers from a bare wildcard to an
explicit allowlist, and switched the release-notes CHANGELOG link to
a tag-relative permalink instead of a floating branch reference.
Research (DEP-14, openedx/frontend-base #273, semantic-release's
branch-pattern config) is cited in ADR-013; the openedx precedent in
particular fixed this identical bug the same way.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
.91 was upgraded straight to TrueNAS-25.10.6 (skipping the planned
Fangtooth stop) and re-tested clean: all 8 method-mapping families
work against real production-scale iSCSI config (2 targets, 9
extents), and NAS-135643 still doesn't reproduce. Also corrects the
auth.login_ex conclusion from the .92-only pass -- it works fine given
the right username (.91's key IS root, .92's isn't), so the
login_with_api_key recommendation stands for a different reason: no
username field exists in storage.cfg to use it correctly, not because
login_ex is broken. Notes the resulting gap (no 25.04.x node left in
the lab) and the revised lab-node role split.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Not committed to any release yet -- filed as a backlog idea alongside
today's #243/ADR-012 work, since a dual REST/WebSocket transport
reality in v4.0.0 makes visible version mismatches more useful to
surface.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Two things surfaced since this ADR was last updated: confirmed
TrueNASMultipath.pm needs no separate WebSocket work (it calls the
shared _api() helper directly), and release/4.x now exists as the
implementation branch, including the dormant CI bug it exposed and
the fix for it.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Multipath (truenas-proxmox-multipath, shipped v3.2.0/#256) had zero
coverage in README.md and was actively described as unsupported/future
work in docs/architecture.md and docs/migration-advanced.md. Added:
- README: Prerequisites, Installation, and Configuration sections for
the multipath package and its truenas-multipath storage type
- architecture.md: new §9 covering what TrueNASMultipath.pm overrides
vs. inherits, and the live failover behavior confirmed 2026-08-29
- migration-advanced.md: points to the multipath package instead of
calling it unsupported
- truenas-storage-help.html: single-path note pointing to the
multipath alternative
Also fixed README content that was stale from the v3.0 beta era: the
PVE 9 blockdev issue (#266) was fixed in v3.1.0 but still read as
open, "v3.0 (beta)" headings never updated to v3.x, a broken internal
anchor link, and a note claiming WebSocket support was "v3.0.0" scope
when ADR-009 placed it at v4.0.0.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The "Publish to GitHub Pages testing dist" step ran for both 'testing'
and 'development' channel builds and does rm -rf pool/testing before
copying in the new .deb — so any push to a release/* branch other than
release/3.x silently overwrote the real testing dist with its own alpha
build. Confirmed happening 2026-08-30 when release/4.x was created and
its 3.2.5~alpha+d0ea761 build replaced release/3.x's legitimate beta
build there. Restricted the step to channel == 'testing' only.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Ran a diagnostic against the .92 TrueNAS SCALE lab node (25.10.3.1) to
check ADR-012's open assumptions. Confirms auth.login_with_api_key over
auth.login_ex (the latter needs a username field this plugin doesn't
collect), verifies all 8 REST->JSON-RPC method mappings live, and finds
NAS-135643 no longer reproduces on this version. Also documents a real
Perl gotcha: JSON's decode_json returns a blessed JSON::PP::Boolean for
true/false, not a plain 1/0.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Scopes the transport-abstraction design called for in #243: confirms the
JSON-RPC 2.0 method mapping for all 8 REST endpoint families the plugin
uses, decides on AnyEvent::WebSocket::Client (apt-packaged, no CPAN-only
deps) after finding an independent prior implementation, and defers the
#249 per-variant file split to a future v4.1/v5.0. Still Draft pending
live verification against the lab's Fangtooth/Goldeneye TrueNAS nodes.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Four of the six tracked issues (#228, #234, #256, #260) had shipped
since this list was last touched, and #250 was closed too — the
instructions Claude reads at session start were actively wrong.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Debian trixie recommends deb822-style .sources files over the one-liner
.list format, and PVE 9 ships on trixie. Show deb822 as the primary form
in README (stable, v2 track, Cloudsmith migration, testing) and
getting-started.md/upgrade-paths.md, keeping the one-liner .list format
as a fallback for PVE 8 / bookworm users.
Fixes#285
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Same root cause as #179: postinst/postrm only restarted pvedaemon/pveproxy,
never pvestatd/pve-ha-lrm/pve-ha-crm, so HA-managed VMs on truenas-multipath
storage could fail to start after install/upgrade. Mirrors the #179 fix —
bare statements so set -e aborts on a real failure, HA daemons restarted
via systemctl (they don't support a restart subcommand). Extended
test-restart-services.sh to cover both multipath scripts with the same
stub approach.
Fixes#287
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
CHANGELOG.md stopped at 3.0.0 despite five releases since. Backfilled
from git/tag history so the changelog is a reliable release record again.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
README and getting-started.md still described the legacy v2.x "ZFS over
iSCSI" storage type and pre-25.04 TrueNAS SCALE UI locations. Update both
to match the actual v3 "TrueNAS (ZFS/iSCSI)" panel fields, document the
SCALE 25.04+ API key location and the throwaway-share workaround for the
now-missing standalone portal/initiator-group screen, and sync the help
HTML per the same-commit doc-sync rule.
Fixes#286
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
core.fileMode=false on this checkout (NTFS-mounted /mnt/c) meant the
previous commit recorded tests/stubs/* and tests/fail-stubs/* as 100644
despite being executable locally, so CI failed with "Permission denied"
trying to exec them via PATH lookup.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
pvedaemon/pveproxy picked up the patched PVE::Storage state, but
pve-ha-lrm/pve-ha-crm kept running against the pre-install state, so
HA-managed VMs on TrueNAS storage failed to start. They don't support
a `restart` subcommand (only start/stop/status), so they're restarted
via systemctl instead. Also switched every restart from `cmd && log
...` to bare statements — set -e does not fire on the left side of
&&, so a genuine restart failure was previously swallowed silently
instead of aborting the script.
Adds tests/test-restart-services.sh: sources the real postinst/postrm
against stub PVE binaries in CI to catch both classes of regression
(wrong subcommand, silently-swallowed failure) without needing a live
PVE cluster. Real HA-cluster validation is documented as a manual
runbook (RUNBOOK-001) since GitHub Actions can't run pve-ha-lrm/crm.
Bumps $VERSION to 3.2.4.
Fixes#179
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
pvedaemon/pveproxy picked up the patched PVE::Storage state, but
pve-ha-lrm/pve-ha-crm kept running against the pre-install state,
so HA-managed VMs on TrueNAS storage failed to start. pve-ha-lrm
and pve-ha-crm don't support a `restart` subcommand (only
start/stop/status), so they're restarted via systemctl instead.
Confirmed root cause by mir07/deamonkai on #179; verified the full
restart_pve_services() sequence end-to-end on pve01-hq before commit.
Fixes#179
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
SCALE 25.04 tightened the iSCSI portal PUT schema — sending port in the
listen array now returns "Extra inputs are not permitted". Added §6.9 in
getting-started.md with the correct curl example, and a matching row in
the help panel troubleshooting table.
Closes#280
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Replace bare pveversion with pveversion -v (includes full package list
for better triage). Replace grep on /var/log/syslog with journalctl
(syslog doesn't exist on Debian 12+/PVE 9).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Users who had the v3 repo configured before v3.2.2 see an apt security
prompt when the Codename field changed from unset to 'error'. Document
the --allow-releaseinfo-change one-liner in README, getting-started.md
(new section 6.8), and upgrade-paths.md (codename column + callout).
Closes#284.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
PVE 9.2.3 sets APIVER=15 in PVE::Storage. Our plugin was returning 14,
triggering "implementing an older storage API" on every pvedaemon/pveproxy
restart. No functional change — plugin loaded and worked correctly at 14.
Closes#282
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
dists/error was never written to gh-pages because the codename alias
code was added after the v3.2.0 tag. The stable publish job only runs
on v*.*.* tags, so all subsequent pushes to release/3.x only touched
dists/testing. Bumping to v3.2.1 triggers stable publish to create it.
Also fix: cp -r src dst when dst already exists puts src inside dst
rather than replacing it. Add rm -rf before cp -r to guarantee a clean
replace on every stable release.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Codename was added to Release metadata but dists/<codename>/ directories were
never created on GitHub Pages, so apt would 404 on InRelease when using the
codename as the dist name. Copy each dist dir under its codename alias after
the main loop so both Suite and Codename values work interchangeably.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
v2/main → limelight (Rush), v3 → error (The Warning),
v4 → rivendell (Lord of the Rings), testing stays as testing.
Suite field unchanged — codename is an alias, both work in sources.list.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Multipath tested and confirmed working on TrueNAS CORE 13.x (FreeBSD/ALUA)
and SCALE 24.10 + 25.04 (Linux LIO). Failover and path recovery validated
on both platforms. Closes#256.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
If /etc/multipath.conf already had a manual devices{} block before package
install, postinst would append a second identical block, causing multipathd
to emit "duplicate keyword: devices" warnings. Strip any unmanaged devices
block in the append path to keep the file clean.
Fixes#279. Discovered 2026-06-20 on pve01-hq during SCALE 25.04 multipath testing.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Old multipath testing debs were never pruned, causing apt to see multiple
3.1.0~beta+<sha> versions and pick whichever sha sorts highest — not newest.
Mirror the existing pool/testing cleanup that already did this correctly.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Without a registered JS panel the truenas-multipath type was invisible in
the PVE Datacenter → Storage → Add dropdown. Adds truenas-multipath.js
with all base TrueNAS fields plus the required truenas_portals field, and
wires it into the multipath package install/remove scripts and CI build.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
On install: if /etc/multipath.conf already exists, append a tagged
block rather than skipping silently (old) or overwriting (dangerous).
On upgrade: replace only the tagged block, leaving the rest untouched.
On remove/purge: strip the tagged block; rest of the file preserved.
This means admins who added their own stanzas during manual testing
(e.g. before the package existed) won't lose config on install or
upgrade. Closes the silent "file already exists" footgun.
Also fixes pre-existing SC2015 shellcheck warnings in postrm.
Refs #256
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
TrueNAS.pm::qemu_blockdev_options returns an iscsi:// blockdev — wrong for
multipath where path() returns /dev/mapper/<wwid>. Simply removing the override
caused the parent's iscsi driver to be inherited instead.
Fix: explicitly call PVE::Storage::Plugin::qemu_blockdev_options (the
grandparent). It calls path(), sees /dev/mapper/<wwid> starts with '/',
stats it as S_ISBLK, and returns { driver => 'host_device', filename => ... }.
QEMU receives: "driver":"host_device","filename":"/dev/mapper/<wwid>"
Tested: VM 103 started with multipath disk (vm-103-disk-0 on TrueNAS-Scale01-MP).
QEMU confirmed using host_device driver on /dev/mapper/36589cfc000000ad...
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
TrueNAS.pm: expose naa field from _find_extent return hash — needed by
TrueNASMultipath.pm to build the /dev/mapper/<wwid> path.
TrueNASMultipath.pm: use multipathd reconfigure + fallback multipath -v0
instead of bare multipath(8) which conflicts with a running multipathd
daemon. Tested end-to-end on pve01-hq against TrueNAS SCALE:
- Two sessions (172.31.69.91 + 192.168.69.91) both established
- /dev/mapper/36589cfc000000eb2adcd24154021d376 appears as block device
- deactivate_volume cleanly flushes and logs out
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
PVE::SectionConfig registers properties globally across all plugin types.
Re-declaring the inherited TrueNAS.pm properties (truenas_host, truenas_pool,
etc.) caused a "duplicate property" error at pvedaemon startup when both
plugins are loaded. TrueNASMultipath::properties() now returns only the
new truenas_portals property; all inherited properties remain registered
by TrueNAS.pm.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
TrueNASMultipath.pm inherits from TrueNAS.pm which lives in perl5/. The
-I/tmp/pve-stub flag alone doesn't make that path searchable, so the
'use base' failed at compile time. Adding -Iperl5 fixes it.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Adds TrueNASMultipath.pm — a separate PVE storage type 'truenas-multipath'
that inherits all REST API and iSCSI resource management from TrueNAS.pm but
uses iscsiadm + dm-multipath for block device access:
path() → /dev/mapper/<wwid> (NAA from TrueNAS extent)
activate_volume() → iscsiadm login on each portal in truenas_portals;
waits for /dev/mapper device to appear
deactivate_volume()→ multipath flush + iscsiadm logout
qemu_blockdev_options → returns undef (path() used instead)
Packaging:
packaging/DEBIAN-multipath/ — control.j2, postinst, postrm,
changelog.Debian, multipath.conf.example
Distributed via the 'multipath' apt component (ADR-011)
Depends: truenas-proxmox >= 3.1.0, open-iscsi, multipath-tools
CI (build.yml):
- Third staging dir (dist-multipath) and .deb output
- multipath component pool under pool/multipath/v<major>
- Both stable and testing publish steps updated to generate
main + multipath components; Release lists both
- Lint checks TrueNASMultipath.pm and multipath packaging scripts
PoC confirmed 2026-06-07 on pve01-hq against TrueNAS SCALE 24.10:
- Two sessions (172.31.69.91 + 192.168.69.91), both active
- dm-multipath aggregated 3 disks × 2 paths each
- /dev/mapper/mpath* devices appeared correctly
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
PVE allocates a vm-<vmid>-state-<snapname> volume when a snapshot is taken
with "Include RAM" selected. parse_volname didn't recognise this pattern,
causing a crash during both setup and cleanup of the state volume.
Added state volume pattern — treated as raw images, same vmid extraction
as disk volumes.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Subroutines::ProhibitExplicitReturnUndef — volume_snapshot,
volume_snapshot_delete, volume_snapshot_rollback all used return undef
where bare return is the correct Perl idiom.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Implements the full PVE storage plugin snapshot interface backed by native
ZFS snapshots on TrueNAS, via the v2.0 REST API:
volume_has_feature — advertises snapshot support for raw volumes
volume_snapshot_needs_fsfreeze — returns 1 (request guest IO freeze)
volume_snapshot — POST /zfs/snapshot {dataset, name}
volume_snapshot_delete — DELETE /zfs/snapshot/id/{uri_escaped_id}
volume_snapshot_rollback — POST /zfs/snapshot/rollback {id, options}
volume_rollback_is_possible — enforces ZFS "most recent snap only" rule
volume_snapshot_info — GET /zfs/snapshot?dataset=... → {name: {id, ctime}}
Snapshot ID format: pool/volname@snapname
DELETE path requires full URI encoding (uri_escape) — unlike the dataset
endpoint, the snapshot API does not accept bare slashes in the path segment.
Tested on pve01-hq (PVE 9.2.3) against TrueNAS CORE 13.0-U6:
- create / list / delete: OK
- rollback_is_possible ordering: correctly rejects non-latest snapshots
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The ACCESS_TOKEN secret was created in 2022 and has expired. Switch both
gh-pages publish steps (stable and testing dist) to github.token, which is
generated fresh per run and never expires. Add permissions: contents: write
to the publish job so GITHUB_TOKEN can push to gh-pages.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
build.yml: testing dist publish condition used 'beta'/'alpha' but the
workflow channel values are 'testing'/'development' — step was always
skipped. Corrected to match actual channel names.
README: badge URLs referenced TheGrandWazoo/truenas-proxmox (repo not yet
renamed); corrected to freenas-proxmox. Workflow badge now includes
branch=release/3.x so it tracks the default branch explicitly.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Adds packaging/changelog.Debian and packaging/DEBIAN-transitional/changelog.Debian
and wires them into the build staging steps. The gzipped changelog lands at
/usr/share/doc/truenas-proxmox/changelog.Debian.gz and
/usr/share/doc/freenas-proxmox/changelog.Debian.gz respectively, satisfying
Debian policy §12.7 and silencing the 'apt changelog' failure.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
