Proxmox Delete Edit

TOC:

• Post Install
• SSL
• Postfix email config
• idmapping
• Storage
• Network
• Container Storage
• GPU Acceleration
• Debugging
• BUGS and Errors
• Links

Post install

• Remove unwanted languages from apt update

  • echo 'Acquire::Languages "none";' > /etc/apt/apt.conf.d/99translations

• Enable IOMMU

  • sed -i 's/GRUB_TIMEOUT=5/GRUB_TIMEOUT=3/' /etc/default/grub;
    sed -i 's/GRUB_CMDLINE_LINUX_DEFAULT="quiet"/GRUB_CMDLINE_LINUX_DEFAULT="quiet amd_iommu=on iommu=pt"/' /etc/default/grub;

  • run update-grub
• Run the community post install script link
  • bash -c "$(curl -fsSL https://raw.githubusercontent.com/community-scripts/ProxmoxVE/main/tools/pve/post-pve-install.sh)"

SSL

• Internal Lair.lan CA:

  • mkdir /usr/local/share/ca-certificates/lairlan-ca;
    mv lairlan.internal.root.crt /usr/local/share/ca-certificates/lairlan-ca/rootCA.crt
    update-ca-certificates

• Internal Fail.pm CA:

  • mkdir /usr/local/share/ca-certificates/failpm-ca;
    mv failpm.internal.root.crt rootCA.crt /usr/local/share/ca-certificates/failpm-ca/rootCA.crt
    update-ca-certificates

Using internal CA see Proxmox SSL

Postfix email config

• install extra modules
  • apt-get install libsasl2-modules postfix-pcre;
• disable any default relayhost/smtp_sasl_security_options in main.cf before adding our own

  • sed -i 's/^relayhost/#relayhost/' /etc/postfix/main.cf;
    sed -i 's/^smtp_sasl_security_options/#smtp_sasl_security_options/' /etc/postfix/main.cf;

• add relay host to /etc/postfix/main.cf

  • cat >> /etc/postfix/main.cf << EOF
    relayhost = [igly.one]:587
    smtp_use_tls = yes
    smtp_sasl_auth_enable = yes
    smtp_sasl_security_options = noanonymous
    smtp_sasl_password_maps = hash:/etc/postfix/sasl_passwd
    smtp_tls_CAfile = /etc/ssl/certs/ca-certificates.crt
    smtp_header_checks = pcre:/etc/postfix/smtp_header_checks
    EOF

• populate sasl_passwd

  • echo '[igly.one]:587  USER:PASSWORD' > /etc/postfix/sasl_passwd;
    chmod 600 /etc/postfix/sasl_passwd;
    postmap /etc/postfix/sasl_passwd;

• Customize From field

  • echo "/^From:.*/ REPLACE From: ${HOSTNAME}-alert <${HOSTNAME}-alert@${HOSTNAME}.lair.lan>" > /etc/postfix/smtp_header_checks;
      postmap /etc/postfix/smtp_header_checks;

• Restart postfix service
  • systemctl restart postfix.service;
• Send test mail
  • echo "Test mail from postfix" | mail -s "$HOSTNAME Test Postfix" some@valid.email

idmapping

• Create a user which will have ownership of the shared data
  • groupadd data -g 2000 && useradd data -u 2000 -g 2000 -m -s /bin/bash
  • See the following links for how to generate a different set of uid/gid
    • https://github.com/ddimick/proxmox-lxc-idmapper python3 ~/run.py 2000:2000=2000:2000
    • https://dump.fail.pm/garbage/arch/pveidmap.html
• Add to /etc/subuid:
  • root:2000:1
• Add to /etc/subgid:
  • root:2000:1
• for each container needing to map data we add the following to its config /etc/pve/lxc/<container_id>.conf

  •   lxc.idmap: u 0 100000 2000
      lxc.idmap: g 0 100000 2000
      lxc.idmap: u 2000 2000 1
      lxc.idmap: g 2000 2000 1
      lxc.idmap: u 2001 102001 63535
      lxc.idmap: g 2001 102001 63535

Storage

Add PBS namespace

add pbs namespace to pve from cli:

pvesm add pbs [storage-id] --server pbs.lair.lan --datastore [id] --content backup --username [id] --password

Delayed Samba mount

In the case where you need to mount a samba share from a container/vm running on top of pve. You can use x-systemd.automount,x-systemd.idle-timeout=30 in /etc/fstab and have it mounted when something accesses the path.

//10.0.10.20/share /mnt/share cifs _netdev,x-systemd.automount,x-systemd.requires=network-online.target,x-systemd.device-timeout=10,noserverino,rw,users,nounix,password2=fake,user=data,password=<PASSWORD>,uid=2000,gid=2000,file_mode=0644,dir_mode=0755,vers=3 0 0

Test with

systemctl daemon-reload
systemctl restart remote-fs.target

Network

Reload interfaces

reload /etc/network/interfaces from cli

ifreload -a

SDN

Apply SDN config changes from cli

pvesh set /cluster/sdn

Bridges and VLAN

I will list interesting commands here

list which interfaces are members of bridges:

brctl show

list which interfaces/bridges are members of VLANs:

bridge -compressvlans vlan show

Thunderbolt 3 Network card

apt install bolt policykit-1
# Change to your ID
boltctl enroll d1030000-0082-8098-2027-f21a6402bb22
echo thunderbolt >> /etc/modules
# Change to your driver
echo atlantic >> /etc/modules
# Probably should rename nic before using its name as input below
echo 'SUBSYSTEM=="net", ACTION=="move", KERNEL=="enp9s0", RUN+="/usr/sbin/ip link set mtu 9000 txqlen 10000 dev '%k'", RUN+="/usr/sbin/ifup %k", RUN+="/usr/sbin/brctl addif vmbr0 %k"' > /etc/udev/rules.d/10-tb-en.rules

Container Storage

Hostpath

Add a hostpath as mpX device mounted at vmpath inside CT

pct set VMID -mp[0-9] [hostpath],mp=[vmpath]

VM disk import

first scp or otherwise load the disk file to the node

qm import 900 /path/to/file

VM disk passthrough

qm set 901--scsi1 /dev/disk/by-label/[label]

CT create disk and mount

creates a new disk image on storageid local size 30gb and maps into CT at /srv/www

pct set [VMID] -mp0 local:30,mp=/srv/www

GPU Acceleration

To be able to use a GPU inside a container add the following to /etc/pve/lxc/[VMID].conf:

dev0: /dev/dri/card0,gid=984
dev1: /dev/dri/renderD128,gid=988
lxc.cgroup2.devices.allow: c 226:1 rwm
lxc.cgroup2.devices.allow: c 226:128 rwm

Do remember to adjust GIDs to match inside the CT and also whatever software is a member of those groups

Container Autologin

Only works if root has no passwd

GETTY_OVERRIDE="/etc/systemd/system/container-getty@1.service.d/override.conf"
mkdir -p $(dirname $GETTY_OVERRIDE)
cat <<EOF >$GETTY_OVERRIDE
  [Service]
  ExecStart=
  ExecStart=-/sbin/agetty --autologin root --noclear --keep-baud tty%I 115200,38400,9600 \$TERM
EOF
systemctl daemon-reload
systemctl restart $(basename $(dirname $GETTY_OVERRIDE) | sed 's/\.d//')

Debugging

To figure out why a container won’t start:

lxc-start -n [VMID] -F -l DEBUG -o /tmp/lxc-ID.log
lxc-start -n [VMID] -F -l DEBUG -o /tmp/lxc-ID.log  -l trace -o /dev/stderr

To see CAPS inside CT use

capsh --print

BUGS and Errors

Remove ghost node after pve hostname change

Do remember to copy over any PVID.confs to the current cluser /etc/pve/lxc/ or /etc/pve/qm/ before deleting

rm -rf /etc/pve/nodes/old-hostname/

RRDC update error

Log output shows

Feb 09 23:37:22 pve01 rrdcached[2038]: handle_request_update: Could not read RRD file.
Feb 09 23:37:22 pve01 pmxcfs[2052]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-vm/311: -1
Feb 09 23:37:22 pve01 pmxcfs[2052]: [status] notice: RRD update error /var/lib/rrdcached/db/pve2-vm/311: mmaping file '/var/lib/rrdcached/db/pve2-vm/311': Invalid argument
Feb 09 23:37:32 pve01 rrdcached[2038]: handle_request_update: Could not read RRD file.
Feb 09 23:37:32 pve01 pmxcfs[2052]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-vm/311: -1
Feb 09 23:37:32 pve01 pmxcfs[2052]: [status] notice: RRD update error /var/lib/rrdcached/db/pve2-vm/311: mmaping file '/var/lib/rrdcached/db/pve2-vm/311': Invalid argument
Feb 09 23:37:42 pve01 rrdcached[2038]: handle_request_update: Could not read RRD file.
Feb 09 23:37:42 pve01 pmxcfs[2052]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-vm/311: -1
Feb 09 23:37:42 pve01 pmxcfs[2052]: [status] notice: RRD update error /var/lib/rrdcached/db/pve2-vm/311: mmaping file '/var/lib/rrdcached/db/pve2-vm/311': Invalid argument

The solution is simply to delete the RRDC db dir and restart the service. Files will automatically be regenerated

rm -r /var/lib/rrdcached/db;
systemctl restart rrdcached.service;

Source

Links

Links:

• https://phucbone.github.io/ProxmoxCheatsheet/
• https://community-scripts.github.io/ProxmoxVE/