# TrueNAS ZFS-over-iSCSI Plugin for Proxmox VE [![License: AGPL v3](https://img.shields.io/badge/License-AGPL_v3-blue.svg)](https://www.gnu.org/licenses/agpl-3.0) [![GitHub release (latest SemVer)](https://img.shields.io/github/v/release/TheGrandWazoo/freenas-proxmox?sort=semver)](https://github.com/TheGrandWazoo/freenas-proxmox/releases/latest) [![GitHub Workflow Status](https://img.shields.io/github/actions/workflow/status/TheGrandWazoo/freenas-proxmox/build.yml?branch=release%2F3.x&label=build)](https://github.com/TheGrandWazoo/freenas-proxmox/actions/workflows/build.yml) [![GitHub issues](https://img.shields.io/github/issues/TheGrandWazoo/freenas-proxmox)](https://github.com/TheGrandWazoo/freenas-proxmox/issues) [![GitHub Sponsors](https://img.shields.io/github/sponsors/TheGrandWazoo?label=Sponsors)](https://github.com/sponsors/TheGrandWazoo) > This plugin is maintained by one person and has 4 million downloads — used in production at ISPs and MSPs worldwide. If it saves you time or headaches, [please consider sponsoring](https://github.com/sponsors/TheGrandWazoo) to keep it actively maintained and funded. A Proxmox VE storage plugin that manages ZFS-over-iSCSI volumes on TrueNAS (CORE and SCALE) through the TrueNAS REST API — no SSH-based LUN management, no `iscsiadm` scripting. --- ## Table of Contents - [How It Works](#how-it-works) - [Compatibility](#compatibility) - [Prerequisites](#prerequisites) - [Installation](#installation) - [Configuration](#configuration) - [Upgrading](#upgrading) - [Uninstalling](#uninstalling) - [Troubleshooting](#troubleshooting) - [Contributing](#contributing) - [Support the Project](#support-the-project) - [License](#license) --- ## How It Works This plugin is a native `PVE::Storage::Custom` type — Proxmox discovers it automatically and treats it like any other storage backend. When you create a VM disk, the plugin calls the **TrueNAS API** (REST v2.0 for CORE and SCALE < 25.04, WebSocket JSON-RPC 2.0 for SCALE 25.04+, chosen automatically per host) to provision a ZFS volume (zvol) and a dedicated per-VM iSCSI target, then hands Proxmox an `iscsi://` path. QEMU opens that path directly — no SSH, no `iscsiadm`, no kernel iSCSI sessions on the Proxmox host. - Bearer Token (API key) authentication — no username/password, no SSH keys - Per-VM iSCSI targets (`proxmox-vm-`) — removing one disk never affects other VMs - Automatic rollback when operations fail (no dangling iSCSI extents) - Disk resize, including on a running VM. Cloud-init drives are supported (v4.0.2+). - Supports TrueNAS CORE 13.x and TrueNAS SCALE 24.10–25.10 --- ## Compatibility ### Version Matrix | Plugin Version | Proxmox VE | TrueNAS CORE | TrueNAS SCALE | Status | |:--------------:|:----------:|:------------:|:-------------:|:------:| | **4.x** *(stable)* | 9.x ✅ | 13.0-U6+ ✅ (REST) | Electric Eel (24.10) (REST), Fangtooth (25.04) ✅ (WebSocket), Goldeneye (25.10) ✅ (WebSocket) | Stable | | **3.x** | 8.4.x ✅ (3.0.0 only — v3.1+ needs PVE 9), 9.x ✅ | 13.0-U6+ ✅ | Electric Eel (24.10) ✅, Fangtooth (25.04) ✅, Goldeneye (25.10) ✅ | Maintenance (`release/3.x`) | | **2.x** *(stable)* | 7.x ⚠️, 8.0–8.3 ✅, 8.4.x ✅ | 11.3+ | 22.02+ | Active | | **1.x** *(legacy)* | 5.x, 6.x | 11.x | — | Unsupported | Tested combinations are marked ✅. Untested combinations may work but are not validated. Check the [Releases page](https://github.com/TheGrandWazoo/freenas-proxmox/releases) for the specific versions tested against each release. ### Important Version Notices > **Proxmox VE 7 users** > > v2.x is the **last release series that supports PVE 7**. PVE 7 support is best-effort only — no new patches will be developed for it. > > v3.x requires Proxmox VE 8 or later. Stay on the latest v2.x release. > **Proxmox VE 8 users** > > v3.x is tested and working on PVE 8.4.x. > > Proxmox VE 8 reached **end-of-life on 2026-08-31**. Plan your upgrade to PVE 9 — v3.x is already tested and fully supported there. > > v4.x requires PVE 9. It installs on PVE 8 but will not load there. v3.1+ also requires PVE 9 (ADR-009). On PVE 8 the only release that loads is **3.0.0**, from the `v3` track, pinned with `apt install truenas-proxmox=3.0.0-1 && apt-mark hold truenas-proxmox`. Unhold it when the node moves to PVE 9. > **Proxmox VE 9+ users** > > v3.x is tested and fully supported on PVE 9.x (tested: 9.2.3). The earlier LUN type mismatch in QEMU's blockdev layer ([#266](https://github.com/TheGrandWazoo/freenas-proxmox/issues/266)) was fixed in v3.1.0 — no known issues on PVE 9. > > v2.x is not supported on PVE 9. Upgrade to v3.x if you're moving to PVE 9. > **Proxmox VE 5 or 6 users** > > These versions are not supported. PVE 5 reached end-of-life in 2019, PVE 6 in 2022. Please upgrade your Proxmox VE installation. Check the [Releases page](https://github.com/TheGrandWazoo/truenas-proxmox/releases) for the specific Proxmox and TrueNAS versions tested against each release. --- ## Prerequisites ### v3.x v3.x is a fully API-driven custom storage plugin. No SSH keys required. 1. On **TrueNAS**, ensure the iSCSI service is running and an iSCSI **portal** and **initiator group** are configured. The plugin creates per-VM iSCSI targets automatically — you do not need to pre-create a target. > **TrueNAS SCALE 25.04+:** the UI no longer exposes a standalone portal/initiator-group configuration screen — only the full iSCSI **Share wizard** (*Shares → iSCSI → Add*). Run the wizard once to create a throwaway share (this leaves a usable portal and initiator group behind), then delete the zvol/share the wizard created. The portal and initiator group persist after that deletion, and the plugin takes over from there. See [docs/getting-started.md §2.3](docs/getting-started.md#23-iscsi-service-on-truenas) for the full walkthrough. 2. Generate a TrueNAS API key: - TrueNAS SCALE 25.04+: *Credentials → API Keys → Add* - TrueNAS SCALE (pre-25.04): *System Settings → API Keys → Add* - TrueNAS CORE 13: *gear icon (top-right) → API Keys → Add* Copy the key — you will need it during storage configuration in Proxmox. > **Known limitation — TPM state disks** > > `tpmstate0` (virtual TPM) disks **cannot** be stored on v3.0 iSCSI storage. The `swtpm` backend requires a local filesystem path and cannot use the `iscsi://` URIs that v3.0 provides to QEMU. > > If your VM uses Secure Boot or TPM, store the `tpmstate0` disk on a separate storage (e.g. `local-lvm` or NFS). All other disk types — virtio, scsi, IDE, EFI — work normally. > > For live VM migration with TPM, the TPM state disk must also be on shared filesystem storage (NFS or CephFS), not on this plugin's storage. ### Multipath (optional) By default, the plugin uses QEMU's `iscsi://` path directly — a single network path to TrueNAS. If that path goes down, disk I/O stops until it recovers; QEMU does not fail over to a second portal on its own. For sites with two or more independent network paths to TrueNAS, the separate **`truenas-proxmox-multipath`** package adds a second storage type (`truenas-multipath`) backed by the kernel's `iscsiadm` + `dm-multipath` instead of `iscsi://` — QEMU accesses a `/dev/mapper/` block device that stays up as long as any one path is healthy. See [Installation → Multipath](#multipath-optional-1) below. > **Requires:** two or more TrueNAS iSCSI portals on separate network paths reachable from every Proxmox node, and `truenas-proxmox` v3.1.0+ already installed. Not needed if you only have a single network path to TrueNAS. ### v2.x (legacy) > v2.x is no longer the recommended version. See [Migrating from v2.x](docs/migrating-from-v2.md). 1. **SSH keys** configured between each Proxmox node and TrueNAS — required by the Proxmox core for ZFS pool listing (see the [Proxmox wiki](https://pve.proxmox.com/wiki/Storage:_ZFS_over_iSCSI), section starting with `mkdir /etc/pve/priv/zfs`). 2. On **TrueNAS**, a pre-created iSCSI **target** and **initiator group**. 3. A TrueNAS API key (SCALE: *System Settings → API Keys*; CORE 13: *gear icon → API Keys*). --- ## Installation ### Stable Release (v4.x) v4.x requires Proxmox VE 9. On PVE 8, the only release that loads is v3.0.0. Use the `v3` track, i.e. replace `v4` with `v3` below, and install with `apt install truenas-proxmox=3.0.0-1 && apt-mark hold truenas-proxmox` (see [PVE 8 users](#important-version-notices)). ```bash # Import the GPG key curl -fsSL https://thegrandwazoo.github.io/freenas-proxmox/public.gpg.key \ | gpg --dearmor \ | sudo tee /etc/apt/keyrings/truenas-proxmox.gpg > /dev/null # Add the v4 repository track (deb822 format — recommended for PVE 9 / Debian trixie) cat << 'SOURCES' | sudo tee /etc/apt/sources.list.d/truenas-proxmox.sources Types: deb URIs: https://thegrandwazoo.github.io/freenas-proxmox Suites: v4 Components: main Signed-By: /etc/apt/keyrings/truenas-proxmox.gpg SOURCES # Install sudo apt update && sudo apt install truenas-proxmox ```
One-liner .list format ```bash echo "deb [signed-by=/etc/apt/keyrings/truenas-proxmox.gpg] \ https://thegrandwazoo.github.io/freenas-proxmox v4 main" \ | sudo tee /etc/apt/sources.list.d/truenas-proxmox.list ```
`apt upgrade` delivers v4.x point releases automatically. Packages never move between major versions on their own. A v3 install stays on v3 until you change the dist track. ### Upgrading from v3.x The upgrade is in place: switch the dist track from `v3` to `v4` and run `apt upgrade`. `storage.cfg` does not change, disks are not moved, and running VMs are not restarted. Every node must be on PVE 9 first. See [docs/upgrade-paths.md → v3.x → v4.x](docs/upgrade-paths.md#v3-to-v4). > **Existing users — one-time prompt after v3.2.2** > If `apt update` shows `changed its 'Codename' value from 'v3' to 'error'`, run: > ```bash > apt update --allow-releaseinfo-change > ``` > This happened once when codename aliases were added in v3.2.2. It will not recur. > Both `v3` (Suite) and `error` (Codename) continue to work in `sources.list`. ### Upgrading from v2.x **v3.0 is a breaking change.** Do not run `apt upgrade` until you have read the migration guide — your storage configuration will need to be updated. See [docs/migrating-from-v2.md](docs/migrating-from-v2.md). ### Staying on v2.x If you are on v2.x and want to continue receiving v2.x point releases without risking a v3 upgrade, switch to the `main` dist track (v2.x only): ```bash # Import the GPG key (if not already done) curl -fsSL https://thegrandwazoo.github.io/freenas-proxmox/public.gpg.key \ | gpg --dearmor \ | sudo tee /etc/apt/keyrings/truenas-proxmox.gpg > /dev/null # Add the v2 repository track (deb822 format — recommended for PVE 9 / Debian trixie) cat << 'SOURCES' | sudo tee /etc/apt/sources.list.d/truenas-proxmox.sources Types: deb URIs: https://thegrandwazoo.github.io/freenas-proxmox Suites: main Components: main Signed-By: /etc/apt/keyrings/truenas-proxmox.gpg SOURCES sudo apt update ```
PVE 8 / Debian bookworm — one-liner .list format ```bash echo "deb [signed-by=/etc/apt/keyrings/truenas-proxmox.gpg] \ https://thegrandwazoo.github.io/freenas-proxmox main main" \ | sudo tee /etc/apt/sources.list.d/truenas-proxmox.list ```
### Migrating from Cloudsmith > Cloudsmith is being phased out. GitHub Pages is the permanent home for this project's packages. If you installed from the Cloudsmith apt repo, swap your source to GitHub Pages. Your installed package is not affected — this only changes where future updates come from. **On v3.x (Cloudsmith stable):** ```bash # Remove the old Cloudsmith source sudo rm -f /etc/apt/sources.list.d/truenas-proxmox*.list sudo rm -f /usr/share/keyrings/ksatechnologies-truenas-proxmox*.gpg # Import the GitHub Pages GPG key curl -fsSL https://thegrandwazoo.github.io/freenas-proxmox/public.gpg.key \ | gpg --dearmor \ | sudo tee /etc/apt/keyrings/truenas-proxmox.gpg > /dev/null # Point to the v3 dist track (deb822 format — recommended for PVE 9 / Debian trixie) cat << 'SOURCES' | sudo tee /etc/apt/sources.list.d/truenas-proxmox.sources Types: deb URIs: https://thegrandwazoo.github.io/freenas-proxmox Suites: v3 Components: main Signed-By: /etc/apt/keyrings/truenas-proxmox.gpg SOURCES sudo apt update ```
PVE 8 / Debian bookworm — one-liner .list format ```bash echo "deb [signed-by=/etc/apt/keyrings/truenas-proxmox.gpg] \ https://thegrandwazoo.github.io/freenas-proxmox v3 main" \ | sudo tee /etc/apt/sources.list.d/truenas-proxmox.list ```
**On v2.x (Cloudsmith stable) — staying on v2:** ```bash # Remove the old Cloudsmith source sudo rm -f /etc/apt/sources.list.d/freenas-proxmox*.list /etc/apt/sources.list.d/truenas-proxmox*.list sudo rm -f /usr/share/keyrings/ksatechnologies-truenas-proxmox*.gpg # Import the GitHub Pages GPG key curl -fsSL https://thegrandwazoo.github.io/freenas-proxmox/public.gpg.key \ | gpg --dearmor \ | sudo tee /etc/apt/keyrings/truenas-proxmox.gpg > /dev/null # Point to the v2 dist track (deb822 format — recommended for PVE 9 / Debian trixie; v2.x only, you will never receive a v3 package) cat << 'SOURCES' | sudo tee /etc/apt/sources.list.d/truenas-proxmox.sources Types: deb URIs: https://thegrandwazoo.github.io/freenas-proxmox Suites: main Components: main Signed-By: /etc/apt/keyrings/truenas-proxmox.gpg SOURCES sudo apt update ```
PVE 8 / Debian bookworm — one-liner .list format ```bash echo "deb [signed-by=/etc/apt/keyrings/truenas-proxmox.gpg] \ https://thegrandwazoo.github.io/freenas-proxmox main main" \ | sudo tee /etc/apt/sources.list.d/truenas-proxmox.list ```
For a full upgrade path matrix (v2, v3 and v4) see [docs/upgrade-paths.md](docs/upgrade-paths.md). --- ### Testing / Beta Release For early access to new features (may be unstable). Beta builds are published on every push to `main` (the current stable major's development branch) — always the latest build, not accumulated history. ```bash # Import the GPG key (skip if already done for the stable track) curl -fsSL https://thegrandwazoo.github.io/freenas-proxmox/public.gpg.key \ | gpg --dearmor \ | sudo tee /etc/apt/keyrings/truenas-proxmox.gpg > /dev/null # Add the testing dist track (deb822 format — recommended for PVE 9 / Debian trixie) cat << 'SOURCES' | sudo tee /etc/apt/sources.list.d/truenas-proxmox.sources Types: deb URIs: https://thegrandwazoo.github.io/freenas-proxmox Suites: testing Components: main Signed-By: /etc/apt/keyrings/truenas-proxmox.gpg SOURCES sudo apt update && sudo apt install truenas-proxmox ```
PVE 8 / Debian bookworm — one-liner .list format ```bash echo "deb [signed-by=/etc/apt/keyrings/truenas-proxmox.gpg] \ https://thegrandwazoo.github.io/freenas-proxmox testing main" \ | sudo tee /etc/apt/sources.list.d/truenas-proxmox.list ```
To switch back to stable, replace `Suites: testing` with `Suites: v4` in `truenas-proxmox.sources` (or `testing` with `v4` in `truenas-proxmox.list` if using the one-liner format) and run `apt update`. --- ### Multipath (optional) Ships as a separate package so single-path installs never pull in `iscsiadm`/`multipath-tools` they don't need. Requires `truenas-proxmox` already installed (see [Stable Release](#stable-release-v4x) above) and at least two TrueNAS iSCSI portals on separate network paths — see [Prerequisites → Multipath](#multipath-optional) for the full requirement. Add the `multipath` component to your existing `truenas-proxmox.sources` (or `.list`) file — same dist track (`v4`/`v3`/`testing`/etc.) you're already using for the base package: ```bash # deb822 format — add "multipath" alongside "main" in Components sudo sed -i 's/^Components: main$/Components: main multipath/' \ /etc/apt/sources.list.d/truenas-proxmox.sources sudo apt update && sudo apt install truenas-proxmox-multipath ```
One-liner .list format ```bash echo "deb [signed-by=/etc/apt/keyrings/truenas-proxmox.gpg] \ https://thegrandwazoo.github.io/freenas-proxmox v4 main multipath" \ | sudo tee /etc/apt/sources.list.d/truenas-proxmox.list sudo apt update && sudo apt install truenas-proxmox-multipath ```
This installs `open-iscsi` and `multipath-tools` as dependencies and registers the `truenas-multipath` storage type. From v4.0.2 on, the package requires the **exact same version** of `truenas-proxmox`, so apt always upgrades (or holds) the two together ([#299](https://github.com/TheGrandWazoo/freenas-proxmox/issues/299)). See [Configuration → Multipath Storage](#multipath-storage) for the extra field it needs, and [docs/architecture.md](docs/architecture.md) for how it differs from the default single-path `iscsi://` model. --- ## Configuration After installation, **refresh your browser** to load the updated Proxmox UI. Then add a new storage: 1. Navigate to **Datacenter → Storage → Add → TrueNAS (ZFS/iSCSI)** 2. Fill in the storage fields — see the table below ### Fields | Field | Value | |-------|-------| | ID | A short name, e.g. `truenas-vms`. Cannot be changed later. | | TrueNAS Host | IP or hostname of your TrueNAS server. Also used as the iSCSI portal address unless Portal IP is set separately. | | API Key | Paste the TrueNAS API key you generated. Leave blank at creation time only if you plan to use a [keyfile](#securing-the-api-token-recommended-for-production) instead. | | Pool / Dataset Path | The ZFS pool or dataset path where VM disks are created, e.g. `tank` or `tank/proxmox/vdisks`. | | Sub-dataset | Optional — extra sub-path appended below Pool / Dataset Path. Leave blank in most cases. | | Shared | Enabled (required for clusters). | | Use SSL | Enabled (recommended). | | Verify SSL Certificate | Disabled unless TrueNAS has a valid CA-signed certificate (most homelab setups use self-signed certs). | | Portal IP | Optional — leave blank unless the TrueNAS management IP differs from the iSCSI data IP. | | Target IQN | Optional — leave blank; auto-discovered from existing iSCSI targets. | Authentication is **Bearer token only** — username/password auth was removed in v3.0. If you're still on v2.x, see [Migrating from v2.x](docs/migrating-from-v2.md). ### Multipath Storage If you installed `truenas-proxmox-multipath` (see [Installation → Multipath](#multipath-optional-1)), add storage as **Datacenter → Storage → Add → TrueNAS Multipath (ZFS/iSCSI)** instead. All the fields above apply, plus one more: | Field | Value | |-------|-------| | Portals | Comma-separated list of iSCSI portal IPs, one per network path — e.g. `172.31.69.91,192.168.69.91`. Each is logged into separately; `dm-multipath` aggregates the sessions into one block device. | A `truenas-multipath` storage cannot be edited into a plain `truenas` storage (or vice versa) after creation — pick the right storage type up front, or remove and re-add it. ### Securing the API Token (Recommended for Production) By default the API token is stored in `/etc/pve/storage.cfg`, which is replicated in plaintext across all cluster nodes. For production deployments, move it into a private keyfile that only root can read and that is not replicated: ```bash # Run on each Proxmox node — replace 'truenas-vms' with your actual storage ID STORAGEID="truenas-vms" KEYFILE="/etc/pve/priv/truenas-${STORAGEID}.key" echo -n "your-api-token-here" > "$KEYFILE" chmod 600 "$KEYFILE" pvesm set "$STORAGEID" --truenas_api_key "" ``` The plugin checks `/etc/pve/priv/truenas-.key` automatically; if it exists, it's used and `truenas_api_key` in `storage.cfg` is ignored. The keyfile must exist on **every Proxmox node** — `/etc/pve/priv/` is not replicated via pmxcfs, so copy it manually. ### ZFS Block Size v3.0's storage panel has no blocksize field — newly created zvols inherit the **default volblocksize of the parent pool/dataset** on TrueNAS. If you need a specific block size (e.g. 16k on TrueNAS SCALE, which requires a 16k minimum), set it as the default on the pool or dataset in the TrueNAS UI before creating disks through the plugin; it does not need to be reconfigured in Proxmox. > This section previously described a Proxmox-side **ZFS Blocksize** field — that field belongs to the legacy v2.x built-in "ZFS over iSCSI" storage type, not the v3 plugin. See [Migrating from v2.x](docs/migrating-from-v2.md) if you're still running v2.x. --- ## Upgrading ### v3.x → v4.x (in place) This is an in-place upgrade on PVE 9: switch the dist track to `v4` and run `apt upgrade`. There are no config changes and no Move Disk, and VMs keep running. Pre-checks, multipath, verification and rollback are covered in [docs/upgrade-paths.md → v3.x → v4.x](docs/upgrade-paths.md#v3-to-v4). Coming from v2.x, do the v2→v3 Move Disk migration below first ([v2.x → v4.x](docs/upgrade-paths.md#v2-to-v4)). ### v2.x → v2.x (patch / minor upgrade) Standard apt upgrade — the package re-applies any patches needed after a Proxmox VE update automatically: ```bash apt update && apt full-upgrade ``` ### v2.x → v3.0 (migration) v3.0 is a different storage plugin type (`PVE::Storage::Custom::TrueNAS`) and uses a different architecture (per-VM iSCSI targets, `iscsi://` paths). There is no in-place upgrade — disk data stays on TrueNAS and you move VM disks across using Proxmox's built-in **Move Disk** function. > **Tested migration path (confirmed in lab):** > Migrating live VM disks from a v2.x storage to a v3.0 storage via Move Disk was validated on Proxmox VE 8.4 with TrueNAS CORE 13.0-U6 and TrueNAS SCALE 24.10. VMs remained running throughout the migration. #### Step-by-step 1. **Install v3.x** on all Proxmox nodes (see [Installation](#installation)). 2. **Add a new v3.0 storage** in Proxmox (*Datacenter → Storage → Add → TrueNAS (ZFS/iSCSI)*). Use the same TrueNAS pool as your existing v2.x storage. Give it a distinct ID (e.g. `truenas-v3`). 3. **For each VM**, move its disks from the v2.x storage to the new v3.0 storage: - Select the VM → **Hardware** tab - Select the disk → **Move Disk** - Choose the new v3.0 storage as the target - Check **Delete source** if you want the old zvol removed after the move Repeat for each disk (including EFI disk if present). VMs can remain running during Move Disk. 4. **Verify** the VM boots and its disks are accessible after migration. 5. **Remove the v2.x storage** from Proxmox once all VMs are migrated (*Datacenter → Storage → Remove*). The iSCSI targets and extents that belonged to the old storage will need to be cleaned up from TrueNAS manually if they were not auto-removed. > **Note:** EFI disks (`efidisk0`) and data disks can be migrated with Move Disk. TPM state disks (`tpmstate0`) must stay on local-lvm or NFS — see the [TPM limitation](#v3x) in Prerequisites. --- ## Uninstalling ```bash apt remove truenas-proxmox ``` If you also installed the multipath package, remove it too (before or after — order doesn't matter, but do both if you're fully removing the plugin): ```bash apt remove truenas-proxmox-multipath ``` This removes the plugin and reverses all patches, returning your Proxmox VE installation to its unmodified state. Any storage configurations using this plugin should be removed from Proxmox before uninstalling. --- ## Troubleshooting ### Understanding what this plugin does — and does not — do Before reporting an issue, it helps to know which layer the error is coming from. This plugin is an **orchestrator**: it calls the TrueNAS REST API to provision resources (create zvols, iSCSI extents, and target mappings). It does not carry data. The actual disk I/O path — QEMU reading and writing blocks — runs directly between Proxmox and TrueNAS over iSCSI, with no involvement from this plugin. There are three distinct layers where errors can appear: **Layer 1 — Plugin (TrueNAS REST API)** The plugin called the TrueNAS API and got an error, or the API was unreachable. These errors come from the plugin code. On v3.0, they appear in syslog/journalctl with a `TrueNASPlugin:` prefix (v2.x used `freenas-proxmox:`). Common causes: - Wrong API host IP or hostname - HTTP vs HTTPS mismatch (toggle **Use SSL**) - API token expired, revoked, or not entered correctly - TrueNAS iSCSI service not running - TrueNAS API service not reachable from the Proxmox node Example log line (v3.0): ``` TrueNASPlugin: Unable to connect to the TrueNAS API at '192.168.1.10' using HTTPS (500) ``` **Layer 2 — iSCSI / QEMU data path** The VM's QEMU process (v3.0) or the Proxmox kernel iSCSI stack (v2.x) failed to connect or lost its session to TrueNAS. These errors come from QEMU or `iscsiadm` — not from this plugin. In **v3.0**, look for QEMU log lines referencing `iscsi://` paths. In **v2.x**, look for `iscsiadm` lines in syslog. Common causes: - TrueNAS iSCSI service stopped while a VM was running - Initiator group does not permit the Proxmox node's IP - Network path to the iSCSI portal is down - CHAP authentication configured on the TrueNAS target but not in Proxmox (note: this plugin does not configure CHAP — it must be set to None or configured separately) Example log line (v2.x): ``` iscsiadm: No active sessions. ``` **Layer 3 — Proxmox core storage stack** Errors from PVE's own storage subsystem — pvedaemon, pvestatd, or storage.cfg parsing. Common causes: - `storage.cfg` syntax error or stale config keys (especially after migrating from v2.x — see [migration guide](docs/migrating-from-v2.md)) - `pvedaemon` or `pvestatd` service crashed **Quick triage:** | Symptom | Likely layer | First check | |---------|-------------|-------------| | Disk creation fails, API error in task log | Plugin (Layer 1) | API key, SSL setting, TrueNAS API reachable | | Disk created on TrueNAS but Proxmox reports error | Plugin (Layer 1) | syslog for `truenas-proxmox:` lines | | VM won't start, iSCSI session error | iSCSI/QEMU (Layer 2) | TrueNAS iSCSI service, initiator group ACL | | Storage shows unavailable | Proxmox core (Layer 3) | `pvedaemon` service, storage.cfg | --- ### After install, "TrueNAS (ZFS/iSCSI)" is not visible in the Add Storage dropdown Refresh your browser (force-refresh with Ctrl+Shift+R or Cmd+Shift+R). The Proxmox UI JavaScript is cached aggressively. ### Storage shows as unavailable / API connection fails Check `journalctl -f` or `/var/log/syslog` on the Proxmox node — the plugin logs all API calls and errors with `[FreeNAS::API::]` prefixes. Common causes: - Wrong API host or portal IP - SSL mismatch (try toggling SSL on/off) - API token expired or revoked - TrueNAS iSCSI service not running ### "volblocksize is less than the default minimum block size" warning on disk creation This warning appears on TrueNAS SCALE when a zvol is created with a blocksize below 16k. **v2.4.0 and later:** the plugin automatically detects the correct blocksize from the TrueNAS API and corrects it. If you see a line like `freenas-proxmox: blocksize 8192 < recommended 16384 -- correcting storage '...'` in the task log, the correction was applied and the disk was created correctly. The storage config is also updated automatically so subsequent disk creations will be silent. **v2.3.x and earlier:** see [ZFS Block Size](#zfs-block-size) — manually set the blocksize to `16k` for SCALE storages. ### API key stops working after upgrading TrueNAS SCALE to 25.04 TrueNAS SCALE 25.04 **revokes all existing API keys** that were created with whitelisted methods during the upgrade. If you are using `truenas_token_auth` and your storage shows as unavailable after a SCALE 25.04 upgrade, your API key was revoked. **Fix:** 1. Log into the TrueNAS SCALE web UI 2. Go to **Credentials → API Keys** and generate a new API key 3. In Proxmox, edit the affected storage (**Datacenter → Storage → Edit**) and paste the new key into the **API Secret / Token** field **Additionally**, TrueNAS SCALE 25.04 enforces HTTPS for API key authentication — keys transmitted over plain HTTP are automatically revoked. Ensure **Use SSL** is enabled in your Proxmox storage config when using token auth. > **Note:** TrueNAS SCALE 25.04 also deprecates the REST API that v3.x uses. It still works through at least SCALE 25.10, and full removal is planned for SCALE 26.x. TrueNAS CORE has no plans to remove REST. **v4.0+** uses the WebSocket JSON-RPC 2.0 API automatically for SCALE 25.04+ hosts. Upgrade before moving to SCALE 26.x: see [docs/upgrade-paths.md → v3.x → v4.x](docs/upgrade-paths.md#v3-to-v4), [ADR-012](.claude/cos/adrs/ADR-012-websocket-transport-v4.md) and [issue #243](https://github.com/TheGrandWazoo/truenas-proxmox/issues/243). ### Disk size in Proxmox shows larger than what I entered This is expected. Proxmox creates disks in **GiB** (gibibytes, base-2) but displays them in **GB** (gigabytes, base-10) in some views. | Unit | Base | 1 unit = | |------|------|----------| | GiB (gibibyte) | 2¹⁰ = 1024 | 1,073,741,824 bytes | | GB (gigabyte) | 10³ = 1000 | 1,000,000,000 bytes | When you enter **80 GiB** in the Proxmox disk creation dialog, the zvol is created as exactly 85,899,345,920 bytes. TrueNAS reports that in decimal: **85.90 GB**. The disk inside the VM is still exactly 80 GiB — nothing is lost or added. A quick reference: | Entered in Proxmox | TrueNAS / decimal display | |--------------------|---------------------------| | 10 GiB | 10.74 GB | | 32 GiB | 34.36 GB | | 80 GiB | 85.90 GB | | 100 GiB | 107.37 GB | | 500 GiB | 536.87 GB | | 1 TiB | 1,099.51 GB (≈ 1.10 TB) | ### Dangling extents on TrueNAS after a failed operation If you see iSCSI extents in TrueNAS that are not associated with any target, they can be safely deleted from the TrueNAS UI. v2.3.0 and later automatically roll back and clean up after a failed LUN creation. ### `pvesm free` / GUI Remove fails with "has no iSCSI extent" **Symptom:** removing a disk fails with `Volume 'vm-101-disk-0' has no iSCSI extent on . Was it created via this plugin?`. The zvol still exists on TrueNAS, but its iSCSI extent is gone. Typical causes are a failed `alloc_image`, an extent deleted by hand, or a leftover from v2.x. **Fixed in v4.0.2** ([#292](https://github.com/TheGrandWazoo/freenas-proxmox/issues/292)). `path()` now returns a placeholder for these orphans, so Proxmox can delete them normally. The task log shows `path: has no mapped iSCSI extent — returning placeholder path`. Starting a VM on such a disk still fails at `activate_volume`, as it should. **On v4.0.1 and earlier:** first confirm that no VM config references the disk (`grep -r '' /etc/pve/nodes/*/qemu-server/`). Then call `vdisk_free` directly, which skips the `path()` lookup: ```bash perl -MPVE::Storage -e 'PVE::Storage::vdisk_free(PVE::Storage::config(), "STORAGE:vm-101-disk-0")' ``` ### Resize, disk info or cloud-init operations fail on 4.0.1 and earlier These errors come from plugin gaps that are **fixed in v4.0.2** ([#294](https://github.com/TheGrandWazoo/freenas-proxmox/issues/294)): | Error | Cause | Workaround before upgrading | |---|---|---| | `can't resize this image format` (`qm disk resize`, GUI Resize, Terraform) | No `volume_resize` | Grow the zvol in TrueNAS (Datasets → zvol → Edit → Size). Then run `qm rescan --vmid ` so Proxmox picks up the new size. | | `volume_size_info on '' failed - no format` (content API, `bpg/proxmox` Terraform refresh) | `volume_size_info` returned only a size | None. Upgrade. | | `unable to parse TrueNAS volume name 'vm--cloudinit'` (`qm destroy`, cloud-init regenerate) | `parse_volname` rejected cloud-init names | Remove the cloud-init drive from the VM. Then delete `vm--cloudinit` in TrueNAS along with its iSCSI extent (and the `proxmox-vm-` target if it's now empty). | Resizing only grows the disk. You still need to grow the partition and filesystem inside the guest yourself (cloud-init's `growpart` does this on next boot). ### `qm destroy` left multipath disks behind on TrueNAS **Symptom:** you destroy a VM whose disks are on a `truenas-multipath` storage. The task reports `OK`, but the zvols, extents and the `proxmox-vm-` target are still on TrueNAS. The journal has no `free_image` line for those disks. **Cause:** in multipath versions up to and including 4.0.1, `path()` returned the LUN id where Proxmox expects the owning VMID. `qm destroy` skips any disk whose owner doesn't match, so it skipped all of them without a warning. **Fixed in v4.0.2** ([#291](https://github.com/TheGrandWazoo/freenas-proxmox/issues/291)). The base `truenas` storage type was never affected. **Finding orphans:** list the storage's volumes and look for VMIDs that have no config: ```bash pvesm list # every vm--* volume on the storage ls /etc/pve/nodes/*/qemu-server/.conf 2>/dev/null # no output → VM no longer exists ``` To free an orphan, use `pvesm free :`. On 4.0.1 and earlier, use the `vdisk_free` one-liner from the entry above. Freeing a VM's last disk also removes its now-empty per-VM target. ### "WebSocket connection ... closed unexpectedly", or a SCALE storage flapping to inactive (4.0.0–4.0.2) **Symptom:** on TrueNAS SCALE 25.04+, requests through the Proxmox API, GUI or Terraform fail at random. Retrying sometimes helps. The error looks like this: ``` TrueNAS WebSocket connection to closed unexpectedly during call 'pool.dataset.query' ``` Other symptoms: - VM creation fails, sometimes leaving the VM in `lock: create`. - Content listings return HTTP 500. - The storage shows `active: 0` for some requests and `active: 1` for others. - `qm`/`pvesm` on the command line work fine. **Cause:** TrueNAS closes an idle WebSocket after 30–60 s. Each long-lived `pvedaemon` worker kept its cached connection and reused it after TrueNAS had closed it. **Fixed in v4.0.3** ([#303](https://github.com/TheGrandWazoo/freenas-proxmox/issues/303)). Now a cached connection is checked before every reuse and pinged if it has been idle, and read-only calls reconnect and retry once. TrueNAS CORE and SCALE < 25.04 (REST) were never affected. **On 4.0.0–4.0.2:** - Upgrade. - If you can't upgrade yet, `systemctl restart pvedaemon` clears the stale connections, but only until they go idle again. - A VM left in `lock: create` with no disks can be removed with `qm unlock && qm destroy `. ### Filing a Bug Report Please use the [GitHub issue tracker](https://github.com/TheGrandWazoo/truenas-proxmox/issues) and include the [bug report template](.github/ISSUE_TEMPLATE/bug_report.md). Include relevant log lines from `syslog` (search for `FreeNAS::`). --- ## Contributing Contributions are welcome. Please read [CONTRIBUTING.md](CONTRIBUTING.md) before opening a pull request. For significant changes, open an issue first to discuss the approach. --- ## Support the Project This plugin is built and maintained by one person. Sponsorship directly funds the hardware lab used to test every release against real Proxmox and TrueNAS nodes before it ships to you. **[Become a sponsor on GitHub](https://github.com/sponsors/TheGrandWazoo)** — GitHub Sponsors has no platform fee and goes straight to development. Sponsor tiers: - **$5/month** — Supporter. You keep this maintained. - **$10/month** — Backer. Listed in DONORS.md. - **$25/month** — Sustainer. Priority issue responses. Donor support has funded a 4-node Proxmox cluster and TrueNAS test lab used for development and validation. See [DONORS.md](DONORS.md) for a full list of donors. --- ## License Copyright (c) 2020 KSA Technologies, LLC This program is free software: you can redistribute it and/or modify it under the terms of the [GNU Affero General Public License](LICENSE) as published by the Free Software Foundation, version 3. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.