Proxmox

Get started with Proxmox. Installation considerations, networking, disk encryption options, and migrating existing virtual machines.

Everything's presented in a useful order to follow if you're used to Hyper-V, VMware Workstation, or VirtualBox and want to jump in by moving to a Proxmox server.

What's most interesting about Proxmox is the Web UI gives you full console and GUI (yes, GUI desktop) access to your VM's through either noVNC or SPICE, even via a smart phone.

Proxmox VE Documentation

Download

gpg --keyserver hkps://keyserver.ubuntu.com:443 --recv-keys 'F4E136C67CDCE41AE6DE6FC81140AF8F639E0C39'

# list keys
pub   rsa4096/0x1140AF8F639E0C39 2022-11-27 [SC] [expires: 2032-11-24]
    Key fingerprint = F4E1 36C6 7CDC E41A E6DE  6FC8 1140 AF8F 639E 0C39
uid                   [ unknown] Proxmox Bookworm Release Key <proxmox-release@proxmox.com>

Install

Walk through the graphical or terminal installer. This is straight forward if you're used to installing hypervisor software.

Email

You'll need to choose an email address to send alerts to, this can be root@localhost or a real email.

Access

Proxmox listens on 22/tcp (SSH) and 8006/tcp (Web UI).

You can browse directly to your Proxmox machine's https://<IP>:8006 or use SSH. SSH allows you to do local port forwarding which will help you access the web interface from a jump host.

# On Windows Host
$wsl_user = wsl.exe whoami
$wsl_ipv4 = wsl.exe ip addr show eth0 | sls "(?:(?:192\.168|172\.16|10\.(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?))\.)(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?\.)(?:25[0-4]|2[0-4][0-9]|[01]?[0-9][0-9]?)" | ForEach-Object { $_.Matches.Value }
PS> ssh -L 127.0.0.1:8006:127.0.0.1:8006 $wsl_user@$wsl_ipv4

# On WSL
JUMP_HOST_USER=''
JUMP_HOST_IP=''
JUMP_HOST_PORT=''
PROXMOX_USER=''
PROXMOX_IP=''
$ ssh -L 127.0.0.1:8006:127.0.0.1:8006 -J $JUMP_HOST_USER@$JUMP_HOST_IP:$JUMP_HOST_PORT $PROXMOX_USER@$PROXMOX_IP

Configure

MFA

First, add 2FA/MFA to your root login. This can be a software token saved to your password manager or authenticator app, plus a hardware key using webauthn.

Package Repositories

Next, if you have a valid enterprise subscription key, you're good to go. If not, disable the enterprise repos and enable the pve-no-subscription repo.

• Proxmox VE Enterprise / No-Subscription Repository

System Updates

GUI: Click Updates then Refresh to run apt-get update; finally >_ Upgrade to run apt-get dist-upgrade

You can also run these from a bash shell on the Proxmox VE host.

• Proxmox VE System Software Updates

For major version upgrades (e.g. 8 to 9) you can perform an in-place upgrade. Follow all of the guidance based on your major version number in the Proxmox Wiki. Most notable is the "version-to-version" script, that will check for any issues you may need to address prior to upgrading. The script for Proxmox 8 is run like this: pve8to9 --full.

• Upgrading from 8 to 9

SSH Authentication

root is allowed to login with a password by default. You can disable this in sshd_config once you've added your key to authorized_keys:

curl https://github.com/USERNAME.keys | tee -a /root/.ssh/authorized_keys
# symlink to /etc/pve/priv/authorized_keys

sed -i_bkup 's/^.*PasswordAuthentication.*$/PasswordAuthentication no/g' /etc/ssh/sshd_config
systemctl restart sshd

Firewall

The firewall is off by default. To enable it:

• Datacenter > Firewall > Options > Click Firewall ... No to highlight it > Click Edit > Check Firewall
• Now iptables -S will reveal a set of default rules

If you enable it, the only default allow rules are to 22/tcp and 8006/tcp from the local subnet.

You can define specific rules to the Proxmox VE host and VM guests from the GUI or using a text editor on:

• /etc/pve/nodes/<nodename>/host.fw Host configuration

• /etc/pve/firewall/<VMID>.fw VM / Container configuration

• Proxmox VE Firewall

Suricata

You can install suricata IPS on the Proxmox host. Keep in mind REJECTED or DROP'd packets do not go to suricata.

apt-get install suricata
modprobe nfnetlink_queue

• Proxmox VE Firewall Tips and Tricks

Users

To see all system users: Datacenter > Permissions > Users

User details for the PVE realm are defined on the filesystem under /etc/pve/user.cfg (note that passwords are under /etc/pve/priv/shadow.cfg).

Users in the PAM realm are still stored under /etc/passwd,shadow and the usual system paths.

• Proxmox VE User Management
• Proxmox VE User Management: Examples

Groups

To see, create, and modify groups: Datacenter > Permissions > Groups

These are effectively user defined, based on what your needs are. Groups are a way to tie multiple users to one or more permission sets.

• Proxmox VE Groups

Proxmox has a built-in set of permission "roles" that you can get started with. These are covered below.

Realms

Proxmox understands two authentication "realms", which are "pve" and "pam".

Example Continued

Here we're using the Proxmox utility pveum user add to create a user in the Proxmox "side" of the system, but tie their authenticaiton to PAM:

root@pve:~# pveversion
pve-manager/8.4.1/2a5fa54a8503f96d (running kernel: 6.8.12-11-pve)
root@pve:~#
root@pve:~# pveum user add testuser@pam
root@pve:~#
root@pve:~# pveum user list
┌──────────────┬─────────────────────────────────┬────────────────┬────────┬────────┬───────────┬────────┬──────┬──────────┬────────────┬──────────────────┬──
│ userid       │ comment                         │ email          │ enable │ expire │ firstname │ groups │ keys │ lastname │ realm-type │ tfa-locked-until │ t
╞══════════════╪═════════════════════════════════╪════════════════╪════════╪════════╪═══════════╪════════╪══════╪══════════╪════════════╪══════════════════╪══
│ admin@pve    │ Standard administrative user.   │                │ 1      │      0 │           │        │      │          │ pve        │                  │
├──────────────┼─────────────────────────────────┼────────────────┼────────┼────────┼───────────┼────────┼──────┼──────────┼────────────┼──────────────────┼──
│ builder@pam  │ User to build packer templates. │                │ 0      │      0 │           │        │      │          │ pam        │                  │
├──────────────┼─────────────────────────────────┼────────────────┼────────┼────────┼───────────┼────────┼──────┼──────────┼────────────┼──────────────────┼──
│ root@pam     │                                 │ root@localhost │ 1      │      0 │           │        │      │          │ pam        │                  │
├──────────────┼─────────────────────────────────┼────────────────┼────────┼────────┼───────────┼────────┼──────┼──────────┼────────────┼──────────────────┼──
│ testuser@pam │                                 │                │ 1      │      0 │           │        │      │          │ pam        │                  │
└──────────────┴─────────────────────────────────┴────────────────┴────────┴────────┴───────────┴────────┴──────┴──────────┴────────────┴──────────────────┴──
root@pve:~#

We can see even though we created this user in Proxmox, they don't really exist (yet):

root@pve:~# su testuser
su: user testuser does not exist or the user entry does not contain all the required fields
root@pve:~#
root@pve:~#
root@pve:~# pveum passwd testuser@pam
Enter new password: ************************************
Retype new password: ************************************
change password failed: user 'testuser' does not exist

Once they've been added to the underlying OS, we can both, run as this user via a shell, AND login to Proxmox's web UI or API with them since it's aware of, and accepting, their authentication through PAM:

root@pve:~# useradd -m -s /bin/bash testuser
root@pve:~# passwd testuser
root@pve:~#
root@pve:~# su testuser
testuser@pve:/root$ cd ~
testuser@pve:~$ pwd
/home/testuser
testuser@pve:~$ id
uid=1001(testuser) gid=1001(testuser) groups=1001(testuser)

Finally, you'll need to scope each user their permissions, either individually, or by giving permissions to a group and tying user(s) to groups, which is the recommended way to manage permissions at scale.

Permissions

See, create, and modify permission assignments: Datacenter > Permissions

• Proxmox VE Permission Management: List of Permissions

That link lists out all predefined permission sets and what they do. The most useful if you're just getting started:

• Administrator: has full privileges
• PVEVMAdmin: fully administer VMs
• PVESysAdmin: audit, system console and system logs
• PVEVMUser: view, backup, configure CD-ROM, VM console, VM power management

Packer User Example

Mayfly277, one of the contributers to Game of AD, has a blog post on creating a separate user account in Proxmox to build packer VM's.

Using these steps as a reference, we can create a more "generic" user and group scope using the built-in permissions and roles tied to groups, then modify them from there if needed.

# Define the groups
pveum group add Administrators -comment "System Administrators"
pveum group add DevOps -comment "Infrastructure-as-code Users"
# Define the roles
pveum acl modify / -group Administrators -role Administrator
pveum acl modify / -group DevOps -role PVEVMAdmin
pveum acl modify / -group DevOps -role PVESysAdmin
# Assign users, assumes these users exist in PAM
pveum user modify admin@pam -group Administrators
pveum user modify builder@pam -group DevOps

Disks

It's recommended to have at least one or more:

• Disks dedicated as a storage pool
• Disks dedicated as a file system (VM backups + files and settings)

Full Disk Encryption

One option is to install Debian with FDE, then install Proxmox on top of it. However it's recommended to just install Proxmox normally.

1. Install Proxmox with ZFS
2. Reboot into recovery mode from the installer USB
3. Follow these steps to enable ZFS encryption
4. Follow these steps to install dropbear-initramfs for remote decryption over SSH
5. Encrypt the dataset for VMs and containers (which is also step 5 in this post)

ZFS + Dropbear, all of the resources below came from this post: FDE with ZFS

• Feature Request: Native ZFS encryption during Proxmox installation
• Encrypting Proxmox VE: Best Methods
• gist: FDE with Proxmox and ZFS native encryption
• github/openzfs: Unlocking ZFS encrypted root over SSH
• Proxmox VE ZFS encryption

LUKS + Dropbear

• Adding LUKS FDE to Proxmox

Storage Disk Encryption

This will walk you through encrypting and managing other internal drives on Proxmox.

Creating a LUKS Partition

Assume you have at least one additional internal drive connected to Proxmox.

# Format the disk, fdisk is available on Proxmox, interactive but straightforward.
# If you do not partition the disk, you'll only see /dev/sdX, not /dev/sdX1.
# https://wiki.archlinux.org/title/Fdisk
fdisk /dev/sdX1

# Commands inside fdisk:
# g  → create GPT partition table
# n  → new partition (accept all defaults for single full-disk partition)
# w  → write and exit

# Format the partition to be LUKS encrypted
cryptsetup luksFormat /dev/sdX1
# Enter a passphrase, store it in your password manager

# Unlock it, assign a storage-id
cryptsetup open --type luks /dev/sdX1 <storage-id>  # You create the storage ID string, e.g. luks1-lvm

# Only create a filesystem if you need it, otherwise use lvm-thin pools for VM storage.
mkfs.ext4 /dev/mapper/luks1-lvm

Storage Pools

To make Proxmox aware of the new storage pool:

• Datacenter > Storage > Add
• ID: luks1-lvm (storage identifier created with cryptsetup)

Alternatively you can use pvesm to do everything (examples originally from gpt-4o).

# Helpful to visualize the pve storage
pvesm status

# Create a new volume group for LUKS drives
vgcreate luks1-group /dev/mapper/luks1-lvm

# Create a new thinpool called "luks1-data" for the luks1-lvm storage drive, leaving 5% of the drive for metadata
lvcreate --type thin-pool -l95%FREE -n luks1-data luks1-group

# Add a LUKS thin-lvm data pool for VM storage
# lvm-thin is more efficient than a filesystem for VM storage
pvesm add lvmthin luks1-lvm \
    --vgname luks1-group \
    --thinpool luks1-data \
    --content images \
    --content rootdir

If you mess up, you can remove or undo almost anything at this stage. For example:

vgremove luks1-group

File Systems

Get a sense of which partition(s) you'll want to add. In this case, the device is sdc and the GPT partition we created with fdisk is sdc1.

lsblk
<SNIP>
sdc                            8:48   0   1.8T  0 disk
└─sdc1                         8:49   0   1.8T  0 part

In this example, assume you have another internal LUKS disk used for scheduled backups, luks2-lvm.

# If this will be a filesystem and NOT an lvm-thin storage pool
# Create a mount path (can be anywhere, /mnt/ is easy to understand)
# and use mount to connect that storage id to the path.
mkdir /mnt/luks2-backup
mount /dev/mapper/luks2-lvm /mnt/luks2-backup

# Make Proxmox aware of the filesystem for VM backup schedules, ISOs, or other use
pvesm add dir luks2-lvm \
    --path /mnt/luks2-backup \
    --content backup

If you need to unmount and close the LUKS disk:

# To unmount the filesystem, and close the the LUKS partition if you need to
umount /mnt/luks2-backup
cryptsetup close luks2-lvm

TPM Unlocking Disks

Unlock Encrypted Storage via a TPM.

References

• github.com/tpm2-software
• tpm2-tools Maintainers
• Intel: Open Sourcing the TPM2 Software Stack
• Arch Linux Wiki: TPM
• Arch Linux Wiki: systemd-cryptenroll
• github.com/systemd tpm2-util.c (search github.com/tpm2-software)

Essential packages:

# On Proxmox 8
sudo apt update; sudo apt install -y \
  cryptsetup \
  tpm2-tools

# On Proxmox 9 or later
sudo apt update; sudo apt install -y \
  cryptsetup \
  tpm2-tools \
  systemd-cryptsetup

Setup on Proxmox:

lsblk  # Use this to determine <storage-name>
lsblk -f | grep crypto_LUKS  # If this has a number appended, like sdb1, include the number below too

# --tpm2-pcrs=0+7 == firmware + Secure Boot state, is generally auto-patch safe
systemd-cryptenroll --tpm2-device=auto --tpm2-pcrs=0+7 /dev/sdX1

# Format: <name> <device> <keyfile> <options>
# echo "<storage-name> UUID=$(blkid -s UUID -o value /dev/sdX) none tpm2-device=auto" \
#    >> /etc/crypttab

mkdir -p /mnt/luks3-backup
pvesm add dir luks3-backup --path /mnt/luks3-backup --content backup

echo "
[Unit]
Description=TPM Unlock Disks
After=systemd-modules-load.service
Before=pve-storage.target

[Service]
Type=oneshot
# Prepend a "-" to the command path, to ignore the exit code
# so the system doesn't stall on boot if there's a change.
# You'll want to pin the drive to to the unique drive id.
# Obtain it with: ls -l /dev/disk/by-id/
# So /dev/sdb1 == /dev/drive/by-id/wwn-0x0123456789abcdef-part1
ExecStart=-/usr/sbin/cryptsetup open /dev/drive/by-id/wwn-0x0123456789abcdef-part1 <storage-name>
# ExecStart=-/usr/sbin/cryptsetup open /dev/drive/by-id/wwn-0x1111111111111111-part1 <storage-name>
# Repeat for each LUKS volume to be auto-unlocked by TPM.
# For filesystems, you must also include the mount commands here too.
# ExecStart=-/usr/bin/mount /dev/mapper/<storage-name> /mnt/some-existing-path
RemainAfterExit=yes

SuccessExitStatus=0

[Install]
WantedBy=multi-user.target" | tee /etc/systemd/system/tpm-unlock-disks.service

systemctl daemon-reload
systemctl enable tpm-unlock-disks.service
# You should also be able to confirm the service unlocks the disks
# by restarting it manually
systemctl restart tpm-unlock-disks.service

cryptsetup luksDump /dev/sdX1

Backup LUKS Headers

• Fedora Project: Backup LUKS Headers
• gitlab/cryptsetup: Backup and Data Recovery
• Arch Linux: LUKS Backup and Restore

Once everything's working it's recommended to backup the headers of your LUKS devices in the extremely rare case a drive header corrupts. These are ~2MB or larger binary blobs, the first "area" of the LUKS disk containing the encrypted key slots and necessary data to unlock the drive.

• Store these on a Proxomox backup filesystem drive
• Store these in your password manager
• If these are stolen, they can be brute-forced offline

cryptsetup luksHeaderBackup --header-backup-file <header-file> <device>
cryptsetup luksHeaderBackup --header-backup-file /mnt/luks2-backup/luks2-header-"$(date +%F)".bin /dev/sdc1

Additional Resources

• How to Mount Existing Disks Within proxmox
• Adding LUKS Encryption to proxmox
• Add an Existing Disk to proxmox
• Add a Dew Disk as LVM-thin
• proxmox Storage Wiki
• Dr Duh Yubikey Guide: Backing Up Keys on Encrypted Storage

Migrating VMs

• Importing Virtual Machines
• Migrating Hyper-V to Proxmox

Between Storage Pools

From Hyper-V

Follow the link referenced above. Essentially, with no snapshots on the Hyper-V VM you wish to migrate (you can export the VM and import it back into Hyper-V so it's separate from your "main" copy, to delete the snapshots, then export it again) export your Hyper-V VM so you have the .vhdx file. Move it over to the Proxmox filesystem.

Create the VM in Proxmox

Create the VM as you would if you were making it from scratch, following the steps in the linked forum post (q35 for the VM type, and then detach the default disk). Use the qm disk import <vmid> <source> <storage> [OPTIONS] command to import the .vhdx file as the disk image for your Hyper-V VM.

qm disk import 105 /mnt/wd500-external/Virtualization/Wazuh/Virtual\ Hard\ Disks/Wazuh.vhdx luks1-lvm
importing disk '/mnt/wd500-external/Virtualization/Wazuh/Virtual Hard Disks/Wazuh.vhdx' to VM 105 ...
  Logical volume "vm-105-disk-3" created.
transferred 0.0 B of 127.0 GiB (0.00%)
transferred 1.3 GiB of 127.0 GiB (1.00%)
transferred 2.5 GiB of 127.0 GiB (2.00%)
transferred 3.8 GiB of 127.0 GiB (3.00%)
transferred 5.1 GiB of 127.0 GiB (4.00%)
<SNIP>
transferred 125.9 GiB of 127.0 GiB (99.12%)
transferred 127.0 GiB of 127.0 GiB (100.00%)
transferred 127.0 GiB of 127.0 GiB (100.00%)
Successfully imported disk as 'unused1:luks1-lvm:vm-102-disk-3'

Add (attach) the disk under the VM's settings. Be sure to go into "Options", not "Hardware" and set this disk first in the boot order. In the event this is a Linux VM, even with SecureBoot enabled you'll be able to boot it, and get running.

That's really it, be sure to install the spice-vdagent service on the guest to leverage additional virtualization features if you want them, though this is not necessary.

Fix any issues (such as networking), poweroff, then take your first snapshot.

VM Configuration

Virtual NIC

This is the equivalent of creating a virtual NIC (not to be confused with a VLAN) in VMware or VirtualBox for guest-to-guest communication (for example a LAN and OPT NIC to attach to a pfSense VM). First you need to create another Linux Bridge network device that isn't attached to any physical network ports.

• Under Datacenter, select your proxmox-ve hostname > System > Network > Create
• Name it, check [x] autostart, create
• Now click "Apply Configuration" at the top, so your new vmbrX shows as "Active: Yes"

Now when creating the VM:

• [VM Name] > Hardware > Add > Network Device
• Add two, the WAN side can be the default vmbr0, the LAN side is your new vmbrX with either intel E1000 or virtIO
• One will receive DHCP / access to the outside world over the Proxmox virtual brdige
• The other will remain purely virtual (for example pfSense could then provide DHCP to that NIC)

Disk Passthrough

Say you've installed Proxmox on an existing server or workstation, either as the primary OS or next to an existing OS. You can mount any of the internal drives to VM's without any special modifications to Proxmox's kernel or boot parameters (tested on PVE-8.3.0).

The commands below are summarized from this article, on how to pass through a physical disk to a virtual machine. This can be done while the VM is live, and without rebooting the VM or Proxmox.

In summary, first obtain the MODEL (e.g. ata-ST1000DM001-1ABCDEF) plus the SERIAL string (e.g. Z1A2B3C4) of all disks:

lsblk -o +MODEL,SERIAL,WWN

If you're looking for a specific disk, you'll have to recognize it by MODEL name, or if they're all the same, you may have to mount each one and review what's on it.

Together those strings create a full path to the disk itself. Find the full path under /dev/disk/by-id/ by searching for the SERIAL string.

ls -l /dev/disk/by-id | grep Z1A2B3C4

Hot-plug the disk as a new SCSI device to the VM.

# Existing SCSI disks are listed as scsi0, scsi1, etc. Add this as a new disk
# to <vm-id>. Use the full path to the disk like below.
qm set <vm-id> --scsi<int> /dev/disk/by-id/<model>_<serial>

# Example:
qm set 105 --scsi2 /dev/disk/by-id/ata-ST1000DM001-1ABCDEF_Z1A2B3C4

Hot-unplug or remove the disk.

qm unlink 105 --idlist scsi2

Maintenance

Use the following tabs to understand the overall activity of your node(s):

• Datacenter > Summary
• <node> > Summary

For the best data and visualization, you'll want to enable the Metric Server, to ship logs to a VM running InfluxDB + Grafana.

Snapshots

VM snapshots are valuable, but more brittle than backups:

• Tied to the storage pool they were taken in
• Are not backed up
• Must be deleted to fully migrate a VM

Backups

System backups can be taken by archiving /etc/pve:

tar czf /mnt/luks3-backup/pve-config-$(date +%Y%m%d).tar.gz /etc/pve/

VM backups are the best recovery and long term storage method. They can be scheduled easily from:

• Datacenter > Backup > Add
• Select the VMs
• Set the cadence / time to run backups
• Mode: Stop will shutdown the VMs before running vzdump to archive them
• Set retention policy, e.g. keep up to 3 backups of each VM

Troubleshooting

This section details issues and how to resolve them.

UEFI Causes Boot Failure

Typically in Unix-like distros where UEFI is supported but Secure Boot is not, you must interrupt the boot process and turn off Secure Boot.

• During boot, when you see the Proxmox splash screen, hit Esc
• Select Device Manager
• Select Secure Boot Configuration
• Uncheck Attempt SecureBoot with the Space bar
• Back out to the main UEFI menu and select Continue