docs: v2.x → v3.0 migration guides — beginner, advanced, troubleshooting (#228)
Three new docs covering the full migration path from freenas-proxmox v2.x to truenas-proxmox v3.0: - migrating-from-v2.md: step-by-step beginner guide with pre-migration checklist, TPM detection, capacity notes, Delete source semantics, FAQ - migration-advanced.md: architecture diff, config field mapping, HA/portal failure warnings, bulk migration planning, rollback procedures - migration-troubleshooting.md: Move Disk failure recovery matrix, dangling resource cleanup, API key expiration, common 422 errors Reviewed by beginner and advanced user minions; incorporated feedback on concept explanations, safety-critical gaps (HA, SPOF portal, concurrent cutover staggering), and production edge cases. Closes #228 Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
Kevin Adams
parent
c7ce39d8ae
commit
0a17a30838
|
|
@ -0,0 +1,190 @@
|
|||
# Migrating from v2.x to v3.0
|
||||
|
||||
This guide covers moving from the old `freenas-proxmox` v2.x plugin to the new `truenas-proxmox` v3.0 plugin.
|
||||
|
||||
v3.0 is a clean break from v2.x — a completely rewritten native Proxmox storage plugin. You cannot upgrade in place. The migration involves installing the new plugin, adding a new storage entry, and moving your VM disks across using Proxmox's built-in Move Disk function. **VMs can stay running throughout.**
|
||||
|
||||
---
|
||||
|
||||
## What's a zvol?
|
||||
|
||||
A **zvol** (ZFS volume) is TrueNAS's name for a dedicated chunk of storage carved out of a pool — think of it as a virtual hard drive sitting inside TrueNAS. Each VM disk in this plugin is its own zvol. When you "move a disk," Proxmox creates a new zvol, copies the data to it, and deletes the old one.
|
||||
|
||||
---
|
||||
|
||||
## What changed from v2.x
|
||||
|
||||
**Authentication:** v2.x used a username and password (or a secret token). v3.0 uses a TrueNAS API key only — username/password is gone.
|
||||
|
||||
**iSCSI connection model:** v2.x ran `iscsiadm` on the Proxmox host to create kernel iSCSI sessions. You could see block devices like `/dev/disk/by-path/ip-...` on your Proxmox node. In v3.0, QEMU connects directly to TrueNAS via `iscsi://` — no kernel sessions, no block devices on the host. **This is expected and correct.** If you look in `/dev/` and don't see the disk, that's fine — QEMU holds the connection, not the host.
|
||||
|
||||
**Per-VM targets:** v2.x shared iSCSI targets across VMs. v3.0 creates a dedicated iSCSI target for each VM (`proxmox-vm-<vmid>`). This means removing one disk never affects other VMs.
|
||||
|
||||
**Package name:** `freenas-proxmox` is now a transitional stub that installs `truenas-proxmox`. Both can coexist during migration.
|
||||
|
||||
---
|
||||
|
||||
## Pre-migration checklist
|
||||
|
||||
Do these checks **before** you start:
|
||||
|
||||
```
|
||||
[ ] All Proxmox nodes running PVE 8.0 or newer
|
||||
pveversion | grep pve-manager
|
||||
|
||||
[ ] TrueNAS reachable from each Proxmox node (API + iSCSI)
|
||||
curl -k https://<truenas_host>/api/v2.0/system/info
|
||||
nc -zv <truenas_host> 3260
|
||||
|
||||
[ ] iSCSI service enabled on TrueNAS
|
||||
Sharing → iSCSI → Target Global Configuration: check "Enable"
|
||||
|
||||
[ ] TrueNAS pool is healthy (no resilvering, no scrub running)
|
||||
TrueNAS → Storage → Pools: all pools show "Healthy"
|
||||
|
||||
[ ] TrueNAS pool has free space ≥ your largest single disk
|
||||
(only one copy is needed at a time if you move one disk at a time)
|
||||
|
||||
[ ] Check each VM for tpmstate0 disks (see TPM section below)
|
||||
Proxmox → VM → Hardware: look for tpmstate0
|
||||
|
||||
[ ] Back up any VMs you're not comfortable losing (optional but recommended)
|
||||
```
|
||||
|
||||
**Ports required from Proxmox to TrueNAS:**
|
||||
|
||||
| Port | Protocol | Used for |
|
||||
|------|----------|---------|
|
||||
| 443 | HTTPS | Plugin API calls (REST v2.0) |
|
||||
| 3260 | TCP | iSCSI disk I/O (QEMU to TrueNAS) |
|
||||
| 22 | SSH | Not needed — removed in v3.0 |
|
||||
|
||||
---
|
||||
|
||||
## Check for TPM state disks first
|
||||
|
||||
Before you do anything else, check if any of your VMs have a TPM state disk. QEMU requires TPM state on a local filesystem path — it cannot use `iscsi://` URIs. You cannot migrate `tpmstate0` to v3.0 iSCSI storage.
|
||||
|
||||
**How to check:** In Proxmox, click each VM → **Hardware** tab. Look for a disk labeled `tpmstate0`. If you see one, you need to move it to `local-lvm` or NFS storage **before** migrating that VM's other disks.
|
||||
|
||||
> **Note for HA users:** If the VM is managed by Proxmox HA, `local-lvm` is not suitable for `tpmstate0` (it won't be accessible on other nodes if the VM is restarted). Use NFS or another shared filesystem storage for `tpmstate0` instead.
|
||||
|
||||
---
|
||||
|
||||
## Step 1 — Install v3.0 on every Proxmox node
|
||||
|
||||
Run this on **every node in the cluster** before adding the new storage:
|
||||
|
||||
```bash
|
||||
# Add the Cloudsmith repo (if not already added)
|
||||
curl -1sLf 'https://dl.cloudsmith.io/public/ksatechnologies/truenas-proxmox/setup.deb.sh' | bash
|
||||
|
||||
# Install
|
||||
apt update && apt install truenas-proxmox
|
||||
```
|
||||
|
||||
Verify the install on each node:
|
||||
```bash
|
||||
dpkg -l truenas-proxmox
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Step 2 — Create an API key on TrueNAS
|
||||
|
||||
**TrueNAS SCALE:**
|
||||
1. Log in to the TrueNAS web UI
|
||||
2. Go to **System Settings → API Keys**
|
||||
3. Click **Add**, give it a name (e.g. `proxmox-v3`), click **Generate Key**
|
||||
4. Copy the key — you will only see it once
|
||||
|
||||
**TrueNAS CORE:**
|
||||
1. Log in to the TrueNAS web UI
|
||||
2. Click the gear icon (top right) → **API Keys**
|
||||
3. Click **Add**, give it a name, click **Submit**
|
||||
4. Copy the key
|
||||
|
||||
Keep the key somewhere safe — you will paste it into Proxmox next.
|
||||
|
||||
---
|
||||
|
||||
## Step 3 — Add the v3.0 storage in Proxmox
|
||||
|
||||
> Keep your existing v2.x storage entry active — **do not remove it yet**. Both entries can coexist while you migrate.
|
||||
|
||||
1. In the Proxmox web UI, go to **Datacenter → Storage → Add → TrueNAS**
|
||||
2. Fill in the fields:
|
||||
|
||||
| Field | What to enter |
|
||||
|-------|--------------|
|
||||
| ID | A new name, e.g. `truenas-v3` — must be different from your v2 storage ID |
|
||||
| TrueNAS Host | Same IP/hostname as your v2 storage |
|
||||
| API Key | The key you just created |
|
||||
| Pool | Same ZFS pool path as before (e.g. `tank/proxmox/disks`) |
|
||||
| Nodes | Select all cluster nodes |
|
||||
|
||||
3. Click **Add**. If the storage turns green and shows free/used space, you are connected.
|
||||
|
||||
**What is "Portal IP"?** Leave it blank for a typical setup with one TrueNAS box on one network. Only fill this in if your TrueNAS has separate management and iSCSI data network interfaces and you want iSCSI traffic on a specific IP.
|
||||
|
||||
---
|
||||
|
||||
## Step 4 — Move your VM disks
|
||||
|
||||
For each VM disk, use Proxmox's Move Disk function. VMs can stay powered on. Each move copies the disk data, then cuts over — you may notice a brief I/O pause (typically under a second) at the moment of cutover. For database or latency-sensitive workloads, consider scheduling moves during low-traffic windows.
|
||||
|
||||
1. Click the VM → **Hardware** tab
|
||||
2. Click a disk (e.g. `scsi0`) → **Move Disk** in the top toolbar
|
||||
3. Set **Target Storage** to `truenas-v3`
|
||||
4. **Delete source:** if checked, Proxmox will automatically delete the old zvol immediately after the copy finishes and the VM switches to the new disk. If unchecked, both zvols remain — you can delete the old one manually from TrueNAS later. When in doubt, leave unchecked and delete manually after verifying the VM works.
|
||||
5. Click **Move Disk** and wait for the task to complete before starting the next move
|
||||
|
||||
**Capacity note:** Each move requires free pool space equal to the size of the disk being moved. If you move one disk at a time and check "Delete source," you only need space for one disk. If you run moves in parallel, multiply accordingly.
|
||||
|
||||
**Timing:** A 100 GB disk over a 1 Gbps link takes roughly 10–15 minutes. Plan accordingly for large disks.
|
||||
|
||||
Repeat for every disk on every VM. EFI disks, CloudInit drives, and data disks all follow the same process. Skip `tpmstate0` disks — handle those separately as described above.
|
||||
|
||||
---
|
||||
|
||||
## Step 5 — Verify and clean up
|
||||
|
||||
After moving all disks for a VM:
|
||||
|
||||
1. **Check the Hardware tab** — all disks should reference `truenas-v3`
|
||||
2. **Restart the VM** (or stop/start) and confirm it boots normally
|
||||
3. **Check TrueNAS** — go to Sharing → iSCSI → Targets. You should see one target named `proxmox-vm-<vmid>` for each migrated VM, with the correct number of LUNs
|
||||
|
||||
Once all VMs are migrated:
|
||||
|
||||
1. Confirm no disks reference the old v2.x storage: run this on any Proxmox node and expect no output:
|
||||
```bash
|
||||
grep -r 'old-storage-id:' /etc/pve/nodes/*/qemu-server/ /etc/pve/nodes/*/lxc/ 2>/dev/null
|
||||
```
|
||||
*(Replace `old-storage-id` with your actual v2 storage ID)*
|
||||
|
||||
2. Go to **Datacenter → Storage**, select the old v2.x storage, click **Remove**
|
||||
|
||||
3. Clean up any leftover iSCSI extents or targets on TrueNAS that were not auto-removed (see [migration-troubleshooting.md](migration-troubleshooting.md))
|
||||
|
||||
---
|
||||
|
||||
## Frequently asked questions
|
||||
|
||||
**Can I run v2.x and v3.0 at the same time?**
|
||||
Yes — both storage entries coexist in Proxmox during migration. Remove the v2.x entry only after all disks are moved.
|
||||
|
||||
**What if something goes wrong — can I go back?**
|
||||
Yes. The original disk is not deleted until the move fully completes (and only if you checked "Delete source"). If a move fails, the VM is still running on its original disk. You can also move disks back from v3.0 to v2.x storage using the same Move Disk process in reverse.
|
||||
|
||||
**Will my VMs have downtime?**
|
||||
No shutdown required. Move Disk with a running VM uses live storage migration. Expect a brief I/O pause (typically under a second) at cutover.
|
||||
|
||||
**What happened to my block devices in `/dev/`?**
|
||||
In v3.0, QEMU talks to iSCSI directly — no kernel sessions are created, so no block devices appear on the Proxmox host. This is correct behavior. Tools like `iscsiadm` and `lsscsi` will not show your VM disks.
|
||||
|
||||
**What about snapshots on my old disks?**
|
||||
Proxmox-level snapshots taken via the v2.x plugin will not transfer. If you have important snapshot data, revert to a snapshot before migrating. ZFS-level snapshots on TrueNAS are unaffected.
|
||||
|
||||
**My old storage still shows orphaned extents on TrueNAS after removal.**
|
||||
See [Dangling resources after removal](migration-troubleshooting.md#dangling-resources-after-removal) in the troubleshooting guide.
|
||||
|
|
@ -0,0 +1,210 @@
|
|||
# Migration Advanced Reference: v2.x to v3.0
|
||||
|
||||
This document is for administrators who want a full understanding of what changes between v2.x and v3.0, how to plan large-scale migrations, and how to handle edge cases. For a step-by-step walkthrough, start with the [Migration Guide](migrating-from-v2.md).
|
||||
|
||||
---
|
||||
|
||||
## What changed architecturally
|
||||
|
||||
### v2.x: Host-managed iSCSI sessions
|
||||
|
||||
v2.x used `iscsiadm` on the Proxmox host to establish kernel-level iSCSI sessions. All VMs on a storage shared a small number of iSCSI targets, with each VM disk assigned to a different LUN number. Device nodes like `/dev/disk/by-path/ip-...-lun-0` appeared on the Proxmox host and were passed to QEMU as block devices.
|
||||
|
||||
**Consequences:**
|
||||
- iSCSI sessions visible on the host (`iscsiadm -m session`)
|
||||
- Block device nodes in `/dev/` on the Proxmox host
|
||||
- LUN limit per target (256 max)
|
||||
- Removing one disk from a running VM could disrupt other VMs on the same target
|
||||
- Depended on SSH keys from Proxmox nodes to TrueNAS for pool enumeration
|
||||
|
||||
### v3.0: Per-VM targets, QEMU-native iSCSI
|
||||
|
||||
v3.0 creates a dedicated iSCSI target for each VM (`proxmox-vm-<vmid>`) and passes an `iscsi://` URI directly to QEMU. QEMU's built-in libiscsi library opens the session — no kernel iSCSI daemon involved.
|
||||
|
||||
**Consequences:**
|
||||
- No iSCSI sessions visible on the Proxmox host
|
||||
- No block device nodes in `/dev/` — this is expected and correct
|
||||
- Each VM has exactly one target; all its disks are LUNs on that target
|
||||
- Removing a disk from one VM never affects another VM's connection
|
||||
- No SSH keys required — the plugin communicates only via TrueNAS REST API
|
||||
|
||||
The `iscsi://` URI format stored in the VM config:
|
||||
```
|
||||
iscsi://<portal_ip>/<iqn>:<target_name>/<lun_id>
|
||||
```
|
||||
Example:
|
||||
```
|
||||
iscsi://192.168.10.50/iqn.2005-10.org.freenas.ctl:proxmox-vm-100/0
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Configuration field mapping
|
||||
|
||||
| v2.x field | v3.0 field | Notes |
|
||||
|-----------|-----------|-------|
|
||||
| `type freenas` / `iscsiprovider freenas` | `type truenas` | New storage plugin type |
|
||||
| `freenas_user` | *(removed)* | Basic auth no longer supported |
|
||||
| `freenas_password` | *(removed)* | Basic auth no longer supported |
|
||||
| `truenas_secret` | `truenas_api_key` | Same concept; renamed |
|
||||
| `portal` | `truenas_portal_ip` | Optional; defaults to `truenas_host` |
|
||||
| `target` | `truenas_target` | Optional; base IQN for auto-discovery |
|
||||
| `pool` | `truenas_pool` | Same ZFS pool path |
|
||||
| — | `truenas_dataset` | New: optional sub-dataset within the pool |
|
||||
| `truenas_ssl` | `truenas_ssl` | Unchanged (default: 1) |
|
||||
| `truenas_ssl_verify` | `truenas_ssl_verify` | Unchanged (default: 0) |
|
||||
|
||||
A v2.x storage config block like:
|
||||
```
|
||||
freenas: old-storage
|
||||
freenas_user admin
|
||||
freenas_password secret
|
||||
portal 192.168.10.50
|
||||
pool tank/proxmox/disks
|
||||
truenas_ssl 1
|
||||
truenas_ssl_verify 0
|
||||
```
|
||||
|
||||
Becomes in v3.0:
|
||||
```
|
||||
truenas: truenas-v3
|
||||
truenas_host 192.168.10.50
|
||||
truenas_api_key 1-AbCdEf...
|
||||
truenas_pool tank/proxmox/disks
|
||||
truenas_ssl 1
|
||||
truenas_ssl_verify 0
|
||||
nodes pve01,pve02,pve03
|
||||
shared 1
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## HA (High Availability) considerations
|
||||
|
||||
If any VM is managed by Proxmox HA, disable HA for that VM before migrating its disks:
|
||||
|
||||
1. Datacenter → HA → Resources → select the VM → Remove (or set to disabled)
|
||||
2. Migrate the disks
|
||||
3. Re-enable HA after migration is complete
|
||||
|
||||
If a node fences (reboots unexpectedly) while Move Disk is in progress, the HA manager may attempt to restart the VM on another node. The VM config may still reference the old disk at that point. Check the VM config after recovery and re-run the migration if needed. The original disk is always preserved until Move Disk fully completes.
|
||||
|
||||
---
|
||||
|
||||
## Single portal — single point of failure
|
||||
|
||||
v3.0 does not support iSCSI multipath. QEMU's libiscsi opens one connection to the portal IP and retries on failure but does not fail over to a second IP.
|
||||
|
||||
If your TrueNAS has multiple network interfaces or portal IPs, `truenas_portal_ip` selects exactly one. **All Proxmox nodes must be able to reach that IP on port 3260.** A failure of that interface will cause all VMs on that storage to lose disk access.
|
||||
|
||||
Options for resilience:
|
||||
- Use a floating/virtual IP at the network layer (bond, VRRP/CARP)
|
||||
- Ensure the portal IP is on a highly available network segment
|
||||
- Multipath support is tracked for a future release
|
||||
|
||||
---
|
||||
|
||||
## Running v2.x and v3.0 in parallel
|
||||
|
||||
Both storage entries can coexist in `/etc/pve/storage.cfg`. Proxmox routes disk allocation to whichever storage is specified — the two plugins do not interfere with each other. This allows a rolling, VM-by-VM migration without any maintenance window.
|
||||
|
||||
Recommended approach for clusters with many VMs:
|
||||
1. Install v3.0 on all nodes first (required before adding the new storage type)
|
||||
2. Add the v3.0 storage entry
|
||||
3. Migrate VMs in batches — test one or two first before proceeding
|
||||
4. Remove the v2.x storage entry only after confirming the last VM is migrated
|
||||
|
||||
---
|
||||
|
||||
## Planning for large deployments
|
||||
|
||||
**Capacity:** Move Disk creates a full copy of the zvol before deleting the old one. You need free pool space equal to the size of one disk at a time (or however many moves run concurrently). If free space is tight, migrate and verify one disk at a time with "Delete source" checked before starting the next.
|
||||
|
||||
**Concurrency:** Proxmox allows multiple Move Disk operations in parallel, but each operation is I/O-intensive. Limit concurrency to 2–3 simultaneous moves and **stagger start times by 5–10 minutes** — if multiple moves reach the cutover phase at the same time, multiple VMs will see simultaneous I/O pauses. Monitor TrueNAS pool load during migrations: `zpool iostat 1`. If the pool is above 60–70% utilization, reduce concurrency.
|
||||
|
||||
**Network:** Move Disk transfers data through the Proxmox host, not directly on TrueNAS. Traffic path: TrueNAS → iSCSI → Proxmox host RAM → iSCSI → TrueNAS. For a 1 Gbps link, budget ~10–15 minutes per 100 GB.
|
||||
|
||||
**Verification checklist per VM:**
|
||||
|
||||
- [ ] All disks listed in Hardware tab reference the new storage
|
||||
- [ ] VM boots cleanly from the new storage
|
||||
- [ ] `qm config <vmid>` shows no references to the old storage ID
|
||||
- [ ] TrueNAS: one iSCSI target named `proxmox-vm-<vmid>` exists with the correct number of LUNs
|
||||
- [ ] TrueNAS: old zvols from v2.x storage are gone (if Delete source was checked)
|
||||
|
||||
**Verification for the whole cluster when done:**
|
||||
|
||||
```bash
|
||||
# Find any VM configs still referencing the old storage ID (replace 'old-storage' with your v2 ID)
|
||||
grep -r 'old-storage:' /etc/pve/nodes/*/qemu-server/ /etc/pve/nodes/*/lxc/ 2>/dev/null
|
||||
```
|
||||
|
||||
No output means all disks are migrated.
|
||||
|
||||
---
|
||||
|
||||
## Why there is no in-place rename path
|
||||
|
||||
An in-place rename would require:
|
||||
1. Renaming each zvol on TrueNAS to match the new naming convention
|
||||
2. Rebuilding the iSCSI extent, target, and target-extent association under v3.0's per-VM model
|
||||
3. Rewriting the disk path in the Proxmox VM config from a block device to an `iscsi://` URI
|
||||
|
||||
Steps 2 and 3 require the VM to be offline and involve several API calls that can fail partway through. The Move Disk path is safer because Proxmox manages the cutover atomically and the VM can stay running. An offline rename script may be provided in a future release for large deployments where copying data is impractical.
|
||||
|
||||
---
|
||||
|
||||
## Rollback
|
||||
|
||||
If a Move Disk operation fails partway through:
|
||||
|
||||
1. The original disk is untouched — it is not deleted until the move completes successfully and you confirm
|
||||
2. The new (partial) zvol and any iSCSI objects created on TrueNAS may be left behind — clean these up manually:
|
||||
- On TrueNAS: **Sharing → iSCSI → Extents** — delete the orphaned extent
|
||||
- On TrueNAS: **Sharing → iSCSI → Targets** — delete the orphaned target (if no other disks for that VM exist)
|
||||
- On TrueNAS: **Storage → Pools → Browse** — delete the orphaned zvol
|
||||
3. The VM's config in Proxmox still references the original disk — it is unaffected
|
||||
|
||||
If you decide to roll back entirely to v2.x after partially migrating:
|
||||
- Disks still on v2.x storage will continue to work on v2.x
|
||||
- Disks already moved to v3.0 storage must stay on v3.0 storage — moving back is the same Move Disk process in reverse
|
||||
|
||||
---
|
||||
|
||||
## TPM state disks
|
||||
|
||||
VMs with vTPM enabled have a `tpmstate0` disk. QEMU requires this to be a local filesystem path — the `iscsi://` URI model that v3.0 uses is incompatible.
|
||||
|
||||
**If your v2.x storage used the kernel iSCSI path model (block device), tpmstate0 may have worked there.** In v3.0 it will not. Before migrating a VM with TPM:
|
||||
|
||||
1. Move the `tpmstate0` disk to `local-lvm` or NFS storage first
|
||||
2. Then migrate the remaining disks to v3.0 storage as normal
|
||||
|
||||
For live VM migration across cluster nodes (if needed), the `tpmstate0` disk must also be on shared filesystem storage (NFS, CephFS) — local-lvm does not support live migration.
|
||||
|
||||
---
|
||||
|
||||
## API key security
|
||||
|
||||
v2.x stored credentials in `/etc/pve/storage.cfg` in plain text. v3.0 supports the same (`truenas_api_key` in the config file), but also supports keyfiles stored outside the cluster config:
|
||||
|
||||
```bash
|
||||
# Store on each node (not synced across the cluster)
|
||||
mkdir -p /etc/pve/priv/truenas
|
||||
echo "1-YourKeyHere" > /etc/pve/priv/truenas/truenas-v3.key
|
||||
chmod 600 /etc/pve/priv/truenas/truenas-v3.key
|
||||
```
|
||||
|
||||
The plugin looks for `/etc/pve/priv/truenas/<storeid>.key` and prefers it over the config file value. This keeps the API key out of the cluster config database (which is readable by all cluster members).
|
||||
|
||||
---
|
||||
|
||||
## Cluster node considerations
|
||||
|
||||
The storage config in `/etc/pve/storage.cfg` is cluster-wide. The plugin must be installed on every node that has the storage in its `nodes` list. If a node is missing the plugin, `pvesm status` will show an error for that storage on that node.
|
||||
|
||||
Install order:
|
||||
1. Install `truenas-proxmox` on all nodes first
|
||||
2. Then add the storage entry (or it may fail on nodes that don't have the plugin yet)
|
||||
|
||||
The API key and any keyfiles must be present on every node individually — keyfiles in `/etc/pve/priv/` are not synced across the cluster.
|
||||
|
|
@ -0,0 +1,187 @@
|
|||
# Migration Troubleshooting
|
||||
|
||||
This guide covers problems specific to migrating from v2.x to v3.0. For general post-migration errors, see the troubleshooting section in the [README](../README.md).
|
||||
|
||||
---
|
||||
|
||||
## Move Disk fails immediately
|
||||
|
||||
**Symptom:** Task errors within seconds of starting Move Disk.
|
||||
|
||||
**Check the task log** (click the failed task in Proxmox):
|
||||
|
||||
| Error | Cause | Fix |
|
||||
|-------|-------|-----|
|
||||
| `TrueNAS API token not configured` | API key missing from the v3.0 storage config | Edit the storage in Datacenter → Storage and add the API key |
|
||||
| `Pool dataset '...' not found` | `truenas_pool` path doesn't exist on TrueNAS | Verify the pool path in the storage config matches exactly what TrueNAS shows |
|
||||
| `422 Unprocessable Entity` | API field validation error | Check TrueNAS version compatibility; enable logging (`journalctl -u pvedaemon -f`) for detail |
|
||||
| `SSL: certificate verify failed` | `truenas_ssl_verify 1` but certificate is self-signed | Set `truenas_ssl_verify 0` in storage config, or install a valid certificate |
|
||||
| `Connection refused` | Wrong host/IP or TrueNAS API not reachable | Verify host from the Proxmox node: `curl -k https://<host>/api/v2.0/system/info` |
|
||||
|
||||
---
|
||||
|
||||
## Move Disk hangs or times out
|
||||
|
||||
**Symptom:** Task starts, progress bar moves, but eventually fails with a timeout or connection error.
|
||||
|
||||
- **Network interruption:** iSCSI traffic is sensitive to packet loss. Check switch/VLAN configuration between Proxmox and TrueNAS.
|
||||
- **TrueNAS pool under stress:** If the pool is rebuilding (resilver in progress) or highly loaded, I/O will be slow. Check TrueNAS → Storage → Pools for any active operations.
|
||||
- **Large disk:** A 500 GB disk at 1 Gbps takes ~70 minutes. Proxmox's default task timeout may trigger. Check if the task actually completed on TrueNAS despite the Proxmox error.
|
||||
|
||||
After a failed move, check TrueNAS for a partial zvol:
|
||||
```
|
||||
Sharing → iSCSI → Extents # look for an extent with the new disk name
|
||||
Storage → Pools → Browse # look for a partial zvol
|
||||
```
|
||||
Delete any partial resources before retrying.
|
||||
|
||||
---
|
||||
|
||||
## VM won't start after Move Disk
|
||||
|
||||
**Symptom:** VM was running before migration, fails to start after the disk move completes.
|
||||
|
||||
**Check the VM config:**
|
||||
```bash
|
||||
qm config <vmid>
|
||||
```
|
||||
Look for the moved disk's line. It should show `iscsi://...` as the path, not a block device path.
|
||||
|
||||
**Check the iSCSI target exists on TrueNAS:**
|
||||
1. TrueNAS → Sharing → iSCSI → Targets — confirm `proxmox-vm-<vmid>` is present
|
||||
2. TrueNAS → Sharing → iSCSI → Extents — confirm the extent for each disk is present
|
||||
3. TrueNAS → Sharing → iSCSI → Associated Targets — confirm each extent is linked to the target at the correct LUN
|
||||
|
||||
If any of these are missing, the disk cannot be accessed. Re-run a Move Disk operation to recreate the iSCSI objects, or use the Proxmox task log to identify which step failed.
|
||||
|
||||
**Check the iSCSI portal is reachable:**
|
||||
```bash
|
||||
# From the Proxmox node, using qm's built-in check:
|
||||
qm start <vmid>
|
||||
# Read the full error in the task log — it often includes the iscsi:// URI
|
||||
```
|
||||
|
||||
If the error is `Failed to open drive ... iscsi://...`: the QEMU process cannot reach the iSCSI portal. Check:
|
||||
- `truenas_portal_ip` in storage config — should be the IP that Proxmox can reach TrueNAS on
|
||||
- Firewall rules between Proxmox and TrueNAS on port 3260
|
||||
- iSCSI service is running on TrueNAS (Sharing → iSCSI → Target Global Configuration → Enable)
|
||||
|
||||
---
|
||||
|
||||
## Disk shows wrong size or "undefined" after move
|
||||
|
||||
**Symptom:** The disk appears in the Hardware tab but shows "undefined" or an incorrect size.
|
||||
|
||||
This is usually transient — the Proxmox UI polls before the iSCSI setup is fully complete. Wait 30–60 seconds and refresh.
|
||||
|
||||
If it persists:
|
||||
|
||||
1. Check `volume_size_info` is returning a value:
|
||||
```bash
|
||||
pvesm volume_info <storeid> <volid>
|
||||
```
|
||||
Example: `pvesm volume_info truenas-v3 truenas-v3:vm-100-disk-0`
|
||||
|
||||
2. If this returns an error, the plugin cannot find the zvol via the TrueNAS API. Verify the zvol exists:
|
||||
```bash
|
||||
curl -sk -H "Authorization: Bearer <apikey>" \
|
||||
https://<truenas_host>/api/v2.0/pool/dataset/id/<pool>%2F<volname>
|
||||
```
|
||||
|
||||
3. If the zvol exists but `volume_size_info` fails, check for SSL or API connectivity issues in the daemon log:
|
||||
```bash
|
||||
journalctl -u pvedaemon --since '5 minutes ago' | grep TrueNAS
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Dangling resources after removal
|
||||
|
||||
**Symptom:** After removing the v2.x storage from Proxmox, TrueNAS still has old iSCSI extents, targets, or zvols that weren't cleaned up.
|
||||
|
||||
The v2.x plugin may not have removed these automatically. They are harmless but waste space and clutter the iSCSI config.
|
||||
|
||||
**To clean up manually on TrueNAS:**
|
||||
|
||||
1. **Extents** (Sharing → iSCSI → Extents): Delete any extents whose name matches old VM disk names
|
||||
2. **Target-Extent associations** (Sharing → iSCSI → Associated Targets): Delete associations for removed targets
|
||||
3. **Targets** (Sharing → iSCSI → Targets): Delete targets that no longer have any Proxmox VMs using them
|
||||
4. **Zvols** (Storage → Pools → Browse): Delete zvols that were not cleaned up (confirm the VM config no longer references them first)
|
||||
|
||||
Always verify a zvol is truly unused before deleting it — check `qm config` for all VMs.
|
||||
|
||||
---
|
||||
|
||||
## Old v2.x storage can't be removed from Proxmox
|
||||
|
||||
**Symptom:** "Remove" is greyed out or fails with "storage is still in use."
|
||||
|
||||
Proxmox blocks storage removal if any VM config still references it. Find the VMs:
|
||||
```bash
|
||||
grep -r 'old-storage:' /etc/pve/nodes/*/qemu-server/ /etc/pve/nodes/*/lxc/ 2>/dev/null
|
||||
```
|
||||
|
||||
For each match, move or remove the disk first. Common leftovers:
|
||||
- **Unused disks** (detached but not deleted): VM → Hardware tab, look for `unused0`, `unused1` etc. referencing the old storage
|
||||
- **CloudInit drives**: Move or delete the CloudInit drive (it will be regenerated)
|
||||
- **EFI disks**: Move via Move Disk the same as any other disk
|
||||
|
||||
---
|
||||
|
||||
## iSCSI service stops working after migration
|
||||
|
||||
**Symptom:** VMs that were running fine start failing with iSCSI errors after migration.
|
||||
|
||||
This is not a migration problem — it means the iSCSI service on TrueNAS is unhealthy. Check:
|
||||
|
||||
1. **TrueNAS → Sharing → iSCSI** — confirm the service shows as running
|
||||
2. **TrueNAS System Logs** — look for any iSCSI service errors around the time of failure
|
||||
3. **Proxmox node logs:**
|
||||
```bash
|
||||
dmesg | grep -i iscsi
|
||||
journalctl -u pvedaemon --since '30 minutes ago' | grep TrueNAS
|
||||
```
|
||||
|
||||
If TrueNAS rebooted or the iSCSI service was restarted, running VMs with iSCSI disks will lose connectivity. QEMU will retry the connection — VMs typically recover within 30–60 seconds if the service comes back quickly. For production workloads, configure TrueNAS to start the iSCSI service automatically on boot.
|
||||
|
||||
---
|
||||
|
||||
## Move Disk failure recovery matrix
|
||||
|
||||
If a Move Disk operation fails partway through, use this table to determine what to clean up and whether it is safe to retry:
|
||||
|
||||
| What failed | State on TrueNAS | VM state | Action |
|
||||
|-------------|-----------------|----------|--------|
|
||||
| Zvol copy (mid-copy) | Partial zvol exists | Running on original disk | Delete partial zvol on TrueNAS, retry Move Disk |
|
||||
| iSCSI extent creation | New zvol exists, no extent | Running on original disk | Delete new zvol on TrueNAS, retry Move Disk |
|
||||
| iSCSI target creation | New zvol + extent exist | Running on original disk | Delete extent and new zvol, retry Move Disk |
|
||||
| targetextent association | All iSCSI objects exist | Running on original disk | Delete targetextent, extent, new zvol, retry Move Disk |
|
||||
| VM config rewrite | All iSCSI objects exist, config not updated | Running on original disk | Verify VM is still on old disk (`qm config`); delete new iSCSI objects and zvol, retry |
|
||||
| Zvol deletion (Delete source) | New disk active, old zvol still present | Running on new disk | Verify VM config references new disk; delete old zvol manually with `zfs destroy` |
|
||||
| Node crash mid-copy | Partial zvol may exist | Will recover to original disk | After node recovers, check TrueNAS for partial zvol; delete if found; retry Move Disk |
|
||||
|
||||
The original disk is never modified or deleted during the copy phase. If anything fails before the VM config is rewritten, the VM will continue running on its original disk.
|
||||
|
||||
---
|
||||
|
||||
## API key revoked or expired
|
||||
|
||||
**Symptom:** Move Disk or disk creation fails with `401 Unauthorized` or `TrueNAS API token not configured`.
|
||||
|
||||
TrueNAS API keys can be revoked manually or expire under rotation policies. If this happens:
|
||||
1. Generate a new API key on TrueNAS (same process as initial setup)
|
||||
2. In Proxmox: Datacenter → Storage → select the storage → Edit → update the API Key field
|
||||
3. Retry the failed operation
|
||||
|
||||
---
|
||||
|
||||
## Getting help
|
||||
|
||||
If you hit a problem not covered here:
|
||||
|
||||
1. Collect the relevant log output:
|
||||
```bash
|
||||
journalctl -u pvedaemon --since '1 hour ago' | grep -i truenas > /tmp/pvedaemon.log
|
||||
```
|
||||
2. Note your versions: `dpkg -l truenas-proxmox`, PVE version (`pveversion`), TrueNAS version
|
||||
3. Open an issue at https://github.com/TheGrandWazoo/freenas-proxmox/issues with the log and version info
|
||||
Loading…
Reference in New Issue