Storage Infrastructure¶

Storage architecture for Tower Fleet homelab, covering ZFS on Proxmox, NAS file sharing, and Kubernetes distributed storage.

Overview¶

Storage Layers: 1. Host Storage: ZFS on Proxmox (rpool) 2. File Storage: NFS share from NAS (LXC 101 at /vault) 3. Distributed Storage: Longhorn in Kubernetes cluster

ZFS Storage (Proxmox Host)¶

Pool: rpool Location: Local storage on Proxmox hypervisor

Container/VM Storage¶

Each LXC container and VM gets a ZFS dataset:

# List ZFS datasets
zfs list

# Container filesystems
//rpool/data/subvol-<ID>-disk-0/

# Example
//rpool/data/subvol-150-disk-0/    # money-tracker container
//rpool/data/subvol-160-disk-0/    # home-portal container

Direct Filesystem Access¶

From Proxmox host, you can access container filesystems directly:

# Read files from money-tracker container
ls //rpool/data/subvol-150-disk-0/app

# Read config from home-portal
cat //rpool/data/subvol-160-disk-0/app/.env.local

Use Cases: - Reading logs without entering container - Backup operations - Debugging file permissions

Caution: Avoid writing directly to container filesystems while containers are running.

Storage Management¶

# Check storage usage
pvesm status

# ZFS pool status
zpool status

# Dataset usage
zfs list -o name,used,avail,refer

# Check specific container storage
zfs list | grep subvol-150

NAS File Storage (LXC 101)¶

Container: LXC 101 (NAS) Mount Point: /vault Protocol: NFS IP: 10.89.97.101

Storage Structure¶

/vault/
├── media/
│   ├── tv/           # TV shows (for Plex/Jellyfin)
│   ├── movies/       # Movies
│   ├── music/        # Music library
│   └── torrents/     # Download staging
├── backups/          # Backup storage
├── documents/        # Shared documents
└── projects/         # Project files

Mounting NAS Storage¶

In LXC Containers:

Containers with NAS access have mount points configured via Proxmox:

# View container mount points
pct config 150
# Look for: mp0: /vault/subvol-101-disk-0/share,mp=/vault

# Access from container
ls /vault/media

In VMs:

VMs (like arr-stack) mount via NFS:

# arr-stack VM mounts NAS at /mnt
ssh root@10.89.97.50
df -h | grep mnt
# Shows: 10.89.97.101:/export/media on /mnt

In Kubernetes Pods:

Kubernetes doesn't currently mount NAS directly. Use Longhorn for persistent storage.

NAS Operations¶

# Check NFS exports (from Proxmox or NAS container)
pct exec 101 -- exportfs -v

# Check NAS storage usage
pct exec 101 -- df -h

# Test NFS mount from Proxmox host
showmount -e 10.89.97.101

Kubernetes Longhorn Storage¶

Purpose: Distributed block storage for Kubernetes StatefulSets Nodes: Runs on all 3 k3s nodes (201, 202, 203) Replicas: Typically 3 (for high availability)

Architecture¶

Longhorn creates replicas of volumes across cluster nodes:

Volume: postgres-data-0 (10GB)
├─ Replica 1 on k3s-master (VM 201)
├─ Replica 2 on k3s-worker-1 (VM 202)
└─ Replica 3 on k3s-worker-2 (VM 203)

If one node fails, data remains available from other replicas.

Storage Classes¶

# List storage classes
kubectl get storageclass

# Default class: longhorn
NAME                 PROVISIONER          RECLAIMPOLICY
longhorn (default)   driver.longhorn.io   Delete

Persistent Volumes¶

# List all persistent volumes
kubectl get pv

# List persistent volume claims
kubectl get pvc -A

# Example: Supabase PostgreSQL
kubectl get pvc -n supabase
# Shows: postgres-data-postgres-0  Bound  10Gi  longhorn

Longhorn UI¶

Access Longhorn dashboard:

# Port-forward to access UI
kubectl port-forward -n longhorn-system svc/longhorn-frontend 8080:80

# Access at: http://localhost:8080

From UI you can: - View volume health and replicas - Monitor storage usage per node - Create backups - Manage snapshots

Volume Operations¶

Create PVC:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: my-app-data
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: longhorn
  resources:
    requests:
      storage: 5Gi

Check volume status:

# Volume details
kubectl describe pvc my-app-data

# Longhorn volume status
kubectl get volumes.longhorn.io -n longhorn-system

Backup volume:

Use Longhorn UI or create snapshot:

# Longhorn snapshots are managed through UI
# Or use kubectl to trigger snapshot via CRD

Storage Quotas and Limits¶

ZFS Quotas¶

# Set quota on dataset
zfs set quota=50G rpool/data/subvol-150-disk-0

# Check quota
zfs get quota rpool/data/subvol-150-disk-0

Kubernetes Resource Limits¶

Limit storage per namespace:

apiVersion: v1
kind: ResourceQuota
metadata:
  name: storage-quota
  namespace: my-app
spec:
  hard:
    persistentvolumeclaims: "5"
    requests.storage: "100Gi"

Backup Strategy¶

Container Backups (LXC)¶

# Manual backup
vzdump 150 --dumpdir /var/lib/vz/dump --mode snapshot

# Restore from backup
pct restore 150 /var/lib/vz/dump/vzdump-lxc-150-*.tar.zst

NAS Backups¶

NAS data should be backed up externally (to USB drive, cloud storage, etc.)

# Example: rsync to external drive
rsync -av /vault/ /mnt/backup-drive/vault-backup/

Kubernetes Backups¶

Longhorn: Built-in backup to S3-compatible storage (optional) Velero: Backup entire namespaces/clusters (future enhancement)

# Example: Backup Supabase namespace
# (Requires Velero installation)
velero backup create supabase-backup --include-namespaces supabase

Storage Monitoring¶

Check Usage¶

# ZFS usage
zfs list -o name,used,avail,refer

# NAS usage
pct exec 101 -- df -h

# Kubernetes storage
kubectl get pv
kubectl top nodes  # Includes disk usage

Alerts¶

Configure alerts for: - ZFS pool >80% full - NAS storage >90% full - Longhorn volume degraded (missing replicas)

See Observability Standards for alert configuration.

Expanding Storage¶

Add Disk to ZFS Pool¶

See Expanding Storage Guide for detailed steps.

Increase Container Storage¶

# Resize container disk (replace <CT_ID> with container ID)
pct resize <CT_ID> 0 +10G

# Verify new size
pct exec <CT_ID> -- df -h

Expand Kubernetes PVC¶

Longhorn supports volume expansion:

# Edit PVC to increase size
kubectl edit pvc my-app-data -n my-namespace
# Change: storage: 5Gi → storage: 10Gi

# Longhorn automatically expands volume

Troubleshooting¶

ZFS Issues¶

# Check pool health
zpool status

# Scrub pool (check for errors)
zpool scrub rpool

# Check scrub progress
zpool status

NFS Mount Issues¶

# From client, test NFS server
showmount -e 10.89.97.101

# Check NFS service on NAS
pct exec 101 -- systemctl status nfs-server

# Remount
umount /vault
mount -t nfs 10.89.97.101:/export/media /vault

Longhorn Volume Degraded¶

# Check Longhorn status
kubectl get volumes.longhorn.io -n longhorn-system

# Check events
kubectl get events -n longhorn-system --sort-by='.lastTimestamp'

# Restart Longhorn manager if needed
kubectl rollout restart deployment longhorn-manager -n longhorn-system

Related Documentation: - Infrastructure Overview - Expanding Storage - Storage Verification - Disaster Recovery