Host Shutdown Procedure

Complete procedure for gracefully shutting down the Proxmox host. Following this procedure prevents service disruption and data corruption.

Last Updated: 2025-12-15 Related: Post-Reboot Recovery | Hardware Health Monitoring

---

Quick Reference

For experienced operators, here's the condensed checklist:

# 1. Pre-flight
zpool status vault | grep -E "scrub|resilver"  # Must be idle

# 2. Scale down apps
kubectl scale deployment -n authentik authentik-server authentik-worker --replicas=0
kubectl scale deployment -n home-portal home-portal --replicas=0 2>/dev/null || true
kubectl scale deployment -n money-tracker money-tracker --replicas=0 2>/dev/null || true
kubectl scale deployment -n trip-planner trip-planner --replicas=0 2>/dev/null || true

# 3. Drain K8s nodes (--disable-eviction bypasses Longhorn PDBs)
kubectl drain k3s-worker-1 k3s-worker-2 k3s-master \
  --ignore-daemonsets --delete-emptydir-data --disable-eviction --timeout=120s

# 4. Shutdown VMs (order matters)
qm shutdown 201 202 203 --timeout 60  # K8s VMs first
qm shutdown 100 --timeout 120         # GPU VM (needs time for driver cleanup)
qm shutdown 360 --timeout 60          # Other VMs
qm list                               # Verify all stopped

# 5. Reboot
reboot

After reboot: Follow Post-Reboot Recovery

---

Why This Order Matters

Without Proper Shutdown

• K8s nodes crash → etcd corruption risk
• GPU VM force-killed → GPU stuck in dirty state → won't passthrough after reboot
• All pods reconnect simultaneously → PostgreSQL connection exhaustion
• Longhorn volumes detach unexpectedly → faulted state

With Proper Shutdown

• K8s drains gracefully → pods migrate cleanly
• GPU driver releases properly → passthrough works after reboot
• Apps already scaled down → no connection storm
• Longhorn detaches cleanly → no salvage needed

---

Phase 1: Pre-Shutdown Checks

Verify it's safe to proceed.

# Check ZFS - DO NOT reboot during scrub/resilver
zpool status vault | grep -E "scrub|resilver|state"
# Expected: state: ONLINE, no active scrub/resilver

# Check for critical running jobs
kubectl get jobs -A | grep -v Completed

# Note current state (for verification after reboot)
kubectl get nodes
qm list

Stop here if: - ZFS scrub or resilver in progress - Critical batch jobs running - Longhorn volumes rebuilding: kubectl get volumes -n longhorn-system | grep -v healthy

---

Phase 2: Scale Down K8s Applications

Prevents PostgreSQL connection storm on reboot. Even with PgBouncer, this reduces recovery complexity.

# Authentik (heaviest PostgreSQL consumer)
kubectl scale deployment -n authentik authentik-server --replicas=0
kubectl scale deployment -n authentik authentik-worker --replicas=0

# Application workloads
kubectl scale deployment -n home-portal home-portal --replicas=0 2>/dev/null || true
kubectl scale deployment -n money-tracker money-tracker --replicas=0 2>/dev/null || true
kubectl scale deployment -n trip-planner trip-planner --replicas=0 2>/dev/null || true

# Supabase API services (optional - reduces load)
kubectl scale deployment -n supabase kong gotrue storage rest --replicas=0

# Wait for termination
sleep 15
kubectl get pods -A | grep -E "Terminating" || echo "All pods terminated"

---

Phase 3: Drain K8s Nodes

Gracefully evicts remaining pods and marks nodes unschedulable. This allows: - StatefulSets to sync data - DaemonSets to clean up - Longhorn to detach volumes properly

# Drain all nodes (workers first, then master)
# Using --disable-eviction to bypass Longhorn's PodDisruptionBudgets
kubectl drain k3s-worker-1 --ignore-daemonsets --delete-emptydir-data --disable-eviction --timeout=120s
kubectl drain k3s-worker-2 --ignore-daemonsets --delete-emptydir-data --disable-eviction --timeout=120s
kubectl drain k3s-master --ignore-daemonsets --delete-emptydir-data --disable-eviction --timeout=120s

# Verify nodes are cordoned
kubectl get nodes
# Expected: All show "Ready,SchedulingDisabled"

Flags explained: - --ignore-daemonsets: Don't wait for DaemonSet pods (they run on every node) - --delete-emptydir-data: Allow eviction of pods using emptyDir volumes - --disable-eviction: Bypass PodDisruptionBudgets (required for Longhorn) - --timeout=120s: Fail if drain takes too long (indicates stuck pods)

Why --disable-eviction? Longhorn creates PDBs for its instance-manager, csi-attacher, and csi-provisioner pods. Without this flag, drain hangs indefinitely. For planned shutdowns where all nodes are being drained, this is safe.

---

Phase 4: Shutdown K8s VMs

Shutdown K8s cluster VMs to allow etcd and kubelet clean termination.

# Shutdown K8s VMs (can be parallel)
qm shutdown 201 --timeout 60  # k3s-master
qm shutdown 202 --timeout 60  # k3s-worker-1
qm shutdown 203 --timeout 60  # k3s-worker-2

# Wait and verify
sleep 30
qm list | grep k3s
# All should show "stopped"

If a VM doesn't respond to shutdown:

# Check VM status
qm status 201

# Force stop only if shutdown hangs > 2 minutes
qm stop 201 --timeout 30

Note: K8s VMs (201-203) don't have QEMU guest agent installed, so qm shutdown will fail with "guest-ping timeout". Use qm stop instead - this is safe since we've already drained the nodes.

---

Phase 5: Shutdown GPU Passthrough VM

Critical: NVIDIA GPUs require graceful shutdown for proper PCI reset.

# Graceful shutdown - allows NVIDIA driver to release GPU
qm shutdown 100 --timeout 120

# Wait for complete stop (GPU cleanup takes time)
sleep 30
qm status 100
# Expected: status: stopped

Why 120s timeout: NVIDIA driver needs 30-60 seconds to properly release the GPU. Forcing early stop leaves GPU in dirty state.

If VM doesn't shutdown gracefully:

# Last resort - will likely cause GPU passthrough failure on next boot
qm stop 100 --timeout 30
# You'll need a full power cycle (not just reboot) to fix GPU

---

Phase 6: Shutdown Other VMs

# Game servers
qm shutdown 360 --timeout 60

# Any other running VMs
qm list | grep running
# Shutdown any remaining

---

Phase 7: Final Verification and Reboot

# Verify all VMs stopped
qm list
# Expected: All VMs show "stopped"

# Reboot
reboot

For power cycle (required if GPU passthrough previously failed):

poweroff
# Wait 30+ seconds
# Physically power on, or use IPMI/iLO

---

Post-Reboot

After host comes back up, follow: Post-Reboot Recovery Guide

Key steps: 1. Verify VMs started (auto-start configured) 2. Uncordon K8s nodes 3. Check Longhorn volumes 4. Scale up applications 5. Verify GPU passthrough

---

Automated Script

Save as /root/scripts/prepare-for-reboot.sh:

#!/bin/bash
set -e

echo "=== Host Shutdown Preparation ==="
echo "Started: $(date)"

# Phase 1: Pre-flight checks
echo -e "\n=== Phase 1: Pre-flight Checks ==="

if zpool status vault | grep -qE "scrub|resilver"; then
    echo "ERROR: ZFS scrub or resilver in progress!"
    zpool status vault | grep -E "scrub|resilver"
    exit 1
fi
echo "ZFS: OK (no active operations)"

# Phase 2: Scale down apps
echo -e "\n=== Phase 2: Scaling Down Applications ==="

echo "Scaling down Authentik..."
kubectl scale deployment -n authentik authentik-server --replicas=0 2>/dev/null || true
kubectl scale deployment -n authentik authentik-worker --replicas=0 2>/dev/null || true

echo "Scaling down application workloads..."
kubectl scale deployment -n home-portal home-portal --replicas=0 2>/dev/null || true
kubectl scale deployment -n money-tracker money-tracker --replicas=0 2>/dev/null || true
kubectl scale deployment -n trip-planner trip-planner --replicas=0 2>/dev/null || true

echo "Scaling down Supabase API services..."
kubectl scale deployment -n supabase kong gotrue storage rest --replicas=0 2>/dev/null || true

echo "Waiting for pods to terminate..."
sleep 15

# Phase 3: Drain K8s nodes (--disable-eviction bypasses Longhorn PDBs)
echo -e "\n=== Phase 3: Draining K8s Nodes ==="

echo "Draining k3s-worker-1..."
kubectl drain k3s-worker-1 --ignore-daemonsets --delete-emptydir-data --disable-eviction --timeout=120s 2>/dev/null || true

echo "Draining k3s-worker-2..."
kubectl drain k3s-worker-2 --ignore-daemonsets --delete-emptydir-data --disable-eviction --timeout=120s 2>/dev/null || true

echo "Draining k3s-master..."
kubectl drain k3s-master --ignore-daemonsets --delete-emptydir-data --disable-eviction --timeout=120s 2>/dev/null || true

# Phase 4: Shutdown K8s VMs
echo -e "\n=== Phase 4: Shutting Down K8s VMs ==="

echo "Shutting down k3s-master (VM 201)..."
qm shutdown 201 --timeout 60 2>/dev/null || qm stop 201 --timeout 30 2>/dev/null || true

echo "Shutting down k3s-worker-1 (VM 202)..."
qm shutdown 202 --timeout 60 2>/dev/null || qm stop 202 --timeout 30 2>/dev/null || true

echo "Shutting down k3s-worker-2 (VM 203)..."
qm shutdown 203 --timeout 60 2>/dev/null || qm stop 203 --timeout 30 2>/dev/null || true

# Phase 5: Shutdown GPU VM
echo -e "\n=== Phase 5: Shutting Down GPU Passthrough VM ==="

echo "Shutting down arr-stack (VM 100) - allowing 120s for GPU cleanup..."
qm shutdown 100 --timeout 120 2>/dev/null || qm stop 100 --timeout 30 2>/dev/null || true

# Phase 6: Shutdown other VMs
echo -e "\n=== Phase 6: Shutting Down Other VMs ==="

echo "Shutting down game-servers (VM 360)..."
qm shutdown 360 --timeout 60 2>/dev/null || qm stop 360 --timeout 30 2>/dev/null || true

# Wait for all VMs
echo "Waiting for VMs to stop..."
sleep 30

# Phase 7: Verification
echo -e "\n=== Phase 7: Final Verification ==="
qm list

RUNNING=$(qm list | grep running | wc -l)
if [ "$RUNNING" -gt 0 ]; then
    echo -e "\nWARNING: $RUNNING VM(s) still running!"
    qm list | grep running
else
    echo -e "\nAll VMs stopped successfully."
fi

echo -e "\n=== Ready for Reboot ==="
echo "Run: reboot"
echo "Or for power cycle: poweroff"
echo ""
echo "After reboot, follow: /root/tower-fleet/docs/operations/post-reboot-recovery.md"
echo ""
echo "Completed: $(date)"

Usage:

chmod +x /root/scripts/prepare-for-reboot.sh
/root/scripts/prepare-for-reboot.sh
reboot

---

Troubleshooting

Drain Stuck on Pod

# Find stuck pod
kubectl get pods -A -o wide | grep -i terminating

# Force delete if safe
kubectl delete pod <pod> -n <namespace> --force --grace-period=0

VM Won't Shutdown

# Check if QEMU guest agent is running
qm agent <vmid> ping

# If no agent, use ACPI shutdown
qm shutdown <vmid> --skiplock

# Last resort
qm stop <vmid> --timeout 30

GPU Passthrough Already Broken

If GPU passthrough failed before you started this procedure: 1. Remove GPU from VM config: qm set 100 -delete hostpci0 2. Complete shutdown procedure 3. Use poweroff instead of reboot 4. After power-on, re-add GPU: qm set 100 -hostpci0 0b:00,pcie=1

---

Reference

VM Inventory

VMID	Name	GPU	Notes
100	arr-stack	Yes (Quadro M2000)	Requires graceful shutdown
201	k3s-master	No	Drain before shutdown
202	k3s-worker-1	No	Drain before shutdown
203	k3s-worker-2	No	Drain before shutdown
300	pc	No	Usually stopped
360	game-servers	No	Standard shutdown

Related Documentation

• Post-Reboot Recovery - What to do after reboot
• Hardware Health Monitoring - Drive replacement procedures
• PgBouncer Migration - Connection pooling (reduces recovery complexity)
• ZFS Drive Health - When to defer reboot for ZFS operations