ZFS Over iSCSI

Purpose: There is a way to incorporate ProxmoxVE and TrueNAS more deeply using SSH, simplifying the deployment of virtual disks/volumes passed into GuestVMs in ProxmoxVE. Using ZFS over iSCSI will give you the following non-exhaustive list of benefits:

• Automatically make Zvols in a ZFS Storage Pool
• Automatically bind device-based iSCSI Extents/LUNs to the Zvols
• Allow TrueNAS to handle VM snapshots directly
• Simplify the filesystem overhead of using TrueNAS and iSCSI with ProxmoxVE
• Ability to take snapshots of GuestVMs
• Ability to perform live-migrations of GuestVMs between ProxmoxVE cluster nodes

ZFS over iSCSI Operational Flow¶

sequenceDiagram
    participant ProxmoxVE as ProxmoxVE Cluster
    participant TrueNAS as TrueNAS Core (inc. iSCSI & ZFS Storage)

    ProxmoxVE->>TrueNAS: Cluster VM node connects via SSH to create ZVol for VM
    TrueNAS->>TrueNAS: Create ZVol in ZFS storage pool
    TrueNAS->>TrueNAS: Bind ZVol to iSCSI LUN
    ProxmoxVE->>TrueNAS: Connect to iSCSI & attach ZVol as VM storage
    ProxmoxVE->>TrueNAS: (On-Demand) Connect via SSH to create VM snapshot of ZVol
    TrueNAS->>TrueNAS: Create Snapshot of ZVol/VM

All ZFS Storage Nodes / TrueNAS Servers¶

Configure SSH Key Exchange¶

You first need to make some changes to the SSHD configuration of the ZFS server(s) storing data for your cluster. This is fairly straight-forward and only needs two lines adjusted. This is based on the Proxmox ZFS over ISCSI documentation. Be sure to restart the SSH service or reboot the storage server after making the changes below before proceeding onto the next steps.

UseDNS no
GSSAPIAuthentication no

LookupClientHostnames no 
VerifyReverseMapping no 
GSSAPIAuthentication no

All ProxmoxVE Cluster Nodes¶

Configure SSH Key Exchange¶

The first step is creating SSH trust between the ProxmoxVE cluster nodes and the TrueNAS storage appliance. You will leverage the ProxmoxVE shell on every node of the cluster to run the following commands.

Note: I will be writing the SSH configuration with the name 192.168.101.100 for simplicity so I know what server the identity belongs to. You could also name it something else like storage.bunny-lab.io_id_rsa.

mkdir /etc/pve/priv/zfs
ssh-keygen -f /etc/pve/priv/zfs/192.168.101.100_id_rsa # (1)
ssh-copy-id -i /etc/pve/priv/zfs/192.168.101.100_id_rsa.pub [email protected] # (2)
ssh -i /etc/pve/priv/zfs/192.168.101.100_id_rsa [email protected] # (3)

1. Do not set a password. It will break the automatic functionality.
2. Send the SSH key to the TrueNAS server.
3. Connect to the TrueNAS server at least once to finish establishing the connection.

Install & Configure Storage Provider¶

Now you need to set up the storage provider in TrueNAS. You will run the commands below within a ProxmoxVE shell, then when finished, log out of the ProxmoxVE WebUI, clear the browser cache for ProxmoxVE, then log back in. This will have added a new storage provider called FreeNAS-API under the ZFS over iSCSI storage type.

keyring_location=/usr/share/keyrings/ksatechnologies-truenas-proxmox-keyring.gpg
curl -1sLf 'https://dl.cloudsmith.io/public/ksatechnologies/truenas-proxmox/gpg.284C106104A8CE6D.key' |  gpg --dearmor >> ${keyring_location}

#################################################################
cat << EOF > /etc/apt/sources.list.d/ksatechnologies-repo.list
# Source: KSATechnologies
# Site: https://cloudsmith.io
# Repository: KSATechnologies / truenas-proxmox
# Description: TrueNAS plugin for Proxmox VE - Production
deb [signed-by=${keyring_location}] https://dl.cloudsmith.io/public/ksatechnologies/truenas-proxmox/deb/debian any-version main

EOF
#################################################################

apt update
apt install freenas-proxmox
apt full-upgrade

systemctl restart pvedaemon
systemctl restart pveproxy
systemctl restart pvestatd

Primary ProxmoxVE Cluster Node¶

From this point, we are ready to add the shared storage provider to the cluster via the primary node in the cluster. This is not strictly required, just simplifies the documentation.

Navigate to "Datacenter (BUNNY-CLUSTER) > Storage > Add > ZFS over iSCSI"