Managing Environment Variables in Kubernetes¶

How to add, update, or remove environment variables for pods running in the k3s cluster.

Quick Reference¶

Deployment Type	Method	Persistence
Helm release	`helm upgrade --set`	Survives upgrades
Custom manifest	Edit deployment.yaml + `kubectl apply`	In git
Quick patch	`kubectl patch`	Lost on redeploy
Secrets	SealedSecret + secretKeyRef	Secure, in git

Method 1: Helm-Deployed Apps¶

For apps installed via Helm (check with helm list -A).

View Current Values¶

# See all custom values
helm get values <release> -n <namespace>

# See all values including defaults
helm get values <release> -n <namespace> --all

Add/Update Environment Variable¶

# Using --set (simple values)
helm upgrade <release> <chart-repo>/<chart> -n <namespace> \
  --reuse-values \
  --set env.NEW_VAR="value"

# Using values file (complex or multiple values)
helm get values <release> -n <namespace> > values.yaml
# Edit values.yaml to add env vars
helm upgrade <release> <chart-repo>/<chart> -n <namespace> -f values.yaml

Common Helm Env Var Paths¶

Different charts use different paths for env vars:

# TrueCharts pattern
workload:
  main:
    podSpec:
      containers:
        main:
          env:
            VAR_NAME: "value"

# Bitnami pattern
extraEnvVars:
  - name: VAR_NAME
    value: "value"

# Generic pattern
env:
  VAR_NAME: "value"

Check the chart's values.yaml for the correct path:

helm show values <chart-repo>/<chart> | grep -A10 env

Method 2: Custom Deployments (Our Apps)¶

For apps deployed via our manifests in /root/tower-fleet/manifests/apps/.

Edit the Deployment Manifest¶

# deployment.yaml
spec:
  template:
    spec:
      containers:
      - name: app
        env:
        - name: EXISTING_VAR
          value: "existing"
        - name: NEW_VAR          # Add new env var
          value: "new-value"
        # For secrets, use secretKeyRef
        - name: SECRET_VAR
          valueFrom:
            secretKeyRef:
              name: app-secrets
              key: SECRET_KEY

Apply Changes¶

kubectl apply -f deployment.yaml
# Or use the deploy script
./deploy.sh

Method 3: Quick Patch (Temporary)¶

For quick testing or when Helm chart is unavailable. Note: Changes are lost on next helm upgrade or manifest apply.

Add New Env Var¶

kubectl patch deployment <name> -n <namespace> --type=json \
  -p='[{"op":"add","path":"/spec/template/spec/containers/0/env/-","value":{"name":"NEW_VAR","value":"new-value"}}]'

Update Existing Env Var¶

# First find the index of the env var
kubectl get deployment <name> -n <namespace> -o jsonpath='{.spec.template.spec.containers[0].env[*].name}'

# Then patch by index (e.g., index 5)
kubectl patch deployment <name> -n <namespace> --type=json \
  -p='[{"op":"replace","path":"/spec/template/spec/containers/0/env/5/value","value":"updated-value"}]'

Remove Env Var¶

# Remove by index
kubectl patch deployment <name> -n <namespace> --type=json \
  -p='[{"op":"remove","path":"/spec/template/spec/containers/0/env/5"}]'

Using kubectl set env (Simpler)¶

# Add or update
kubectl set env deployment/<name> -n <namespace> NEW_VAR="value"

# Remove
kubectl set env deployment/<name> -n <namespace> NEW_VAR-

# View current env vars
kubectl set env deployment/<name> -n <namespace> --list

Method 4: Secrets (Sensitive Values)¶

For API keys, passwords, and other sensitive data.

Quick Method: edit-secret.sh (Recommended)¶

# Edit secrets interactively - extracts, opens editor, seals, applies
/root/tower-fleet/scripts/edit-secret.sh <namespace>

# Examples
/root/tower-fleet/scripts/edit-secret.sh trip-planner
/root/tower-fleet/scripts/edit-secret.sh home-portal home-portal-secrets

The script: 1. Extracts current secret values (decoded) 2. Opens in your editor ($EDITOR or vim) 3. Seals with kubeseal 4. Applies to cluster 5. Saves to manifests directory

Manual Method: Create Secret¶

# Create secret manifest
cat > /tmp/secret.yaml << EOF
apiVersion: v1
kind: Secret
metadata:
  name: app-secrets
  namespace: <namespace>
type: Opaque
stringData:
  API_KEY: "your-secret-key"
  DB_PASSWORD: "your-password"
EOF

# Seal it (for git storage)
kubeseal --format=yaml < /tmp/secret.yaml > sealed-secret.yaml
rm /tmp/secret.yaml

# Apply
kubectl apply -f sealed-secret.yaml

Reference in Deployment¶

env:
- name: API_KEY
  valueFrom:
    secretKeyRef:
      name: app-secrets
      key: API_KEY

Update Existing Secret¶

# Get current secret, decode, modify, re-seal
kubectl get secret <name> -n <namespace> -o yaml > /tmp/secret.yaml
# Edit the secret
kubeseal --format=yaml < /tmp/secret.yaml > sealed-secret.yaml
kubectl apply -f sealed-secret.yaml

Verification¶

After making changes, verify they took effect:

# Check deployment rolled out
kubectl rollout status deployment/<name> -n <namespace>

# Verify env var is set
kubectl exec deployment/<name> -n <namespace> -- env | grep VAR_NAME

# Or check the deployment spec
kubectl get deployment <name> -n <namespace> -o jsonpath='{.spec.template.spec.containers[0].env}' | jq

Examples¶

Add RetroAchievements API Key to RomM¶

# Quick patch method (TrueCharts repo defunct)
kubectl set env deployment/romm -n romm RETROACHIEVEMENTS_API_KEY="your-key"

# Verify
kubectl exec deployment/romm -n romm -- env | grep RETRO

Add OAuth Credentials to Custom App¶

# 1. Create sealed secret
cat > /tmp/oauth.yaml << EOF
apiVersion: v1
kind: Secret
metadata:
  name: myapp-oauth
  namespace: myapp
type: Opaque
stringData:
  OAUTH_CLIENT_ID: "client-id-here"
  OAUTH_CLIENT_SECRET: "client-secret-here"
EOF

kubeseal --format=yaml < /tmp/oauth.yaml > /root/tower-fleet/manifests/apps/myapp/oauth-sealed-secret.yaml
rm /tmp/oauth.yaml

# 2. Apply secret
kubectl apply -f /root/tower-fleet/manifests/apps/myapp/oauth-sealed-secret.yaml

# 3. Update deployment to reference secret
# Edit deployment.yaml to add secretKeyRef entries
kubectl apply -f /root/tower-fleet/manifests/apps/myapp/deployment.yaml

Update Supabase URL for an App¶

# For helm app
helm upgrade myapp mychart -n myapp --reuse-values \
  --set env.SUPABASE_URL="https://supabase.internal"

# For custom deployment
kubectl set env deployment/myapp -n myapp \
  SUPABASE_URL="https://supabase.internal"

Troubleshooting¶

Env var not taking effect¶

# Check if deployment restarted
kubectl get pods -n <namespace> -w

# Force restart if needed
kubectl rollout restart deployment/<name> -n <namespace>

"index out of range" on patch¶

The env array index doesn't exist. Use kubectl set env instead, or add with "op":"add".

Secret not found¶

# Check secret exists
kubectl get secret <name> -n <namespace>

# Check secret has the key
kubectl get secret <name> -n <namespace> -o jsonpath='{.data}'

Third-Party App Deployment - Full deployment workflow
Commands Reference - kubectl and helm commands
Sealed Secrets - Secure secret management

Last Updated: 2025-12-16