Skip to content

Upgrade Kubernetes

This guide covers upgrading Kubernetes on this Talos cluster.

Overview

This cluster uses talhelper (see task talos:*). Kubernetes upgrades can be performed with task talos:upgrade-k8s (recommended) or directly with talosctl upgrade-k8s.

The desired Kubernetes version is tracked in talos/talenv.yaml (Renovate may open PRs for this), but the upgrade itself is an operator-run maintenance action.

Version Compatibility

Check compatibility: Talos Support Matrix

Pre-Upgrade Checklist

  • [ ] Check current version: kubectl version
  • [ ] Review Kubernetes release notes
  • [ ] Review Kubernetes deprecations
  • [ ] Backup etcd: talosctl -n 172.16.1.50 etcd snapshot backup.snapshot
  • [ ] Verify cluster health: kubectl get nodes
  • [ ] Check for deprecated APIs in your manifests

Check for Deprecated APIs

# Install pluto
brew install FairwindsOps/tap/pluto

# Scan for deprecated APIs
pluto detect-files -d kubernetes/
pluto detect-helm -A

Update the pinned version in talos/talenv.yaml:

# talos/talenv.yaml
kubernetesVersion: v1.34.3

Commit the change once the upgrade is complete (or as part of the same maintenance window).

Upgrade Process

Run the upgrade from a workstation with talosctl configured.

# Uses talos/talenv.yaml:kubernetesVersion
task talos:upgrade-k8s

Dry run

talosctl -n 172.16.1.50 upgrade-k8s --to v1.34.3 --dry-run

Upgrade

talosctl -n 172.16.1.50 upgrade-k8s --to v1.34.3

You only need to target one control plane node; Talos will coordinate the Kubernetes control plane upgrade.

Post-Upgrade Verification

# Check Kubernetes version
kubectl version

# All nodes should show new version
kubectl get nodes

# Check all system pods
kubectl get pods -n kube-system

# Verify cluster health
talosctl -n 172.16.1.50 health

# Check component versions
kubectl get nodes -o wide

Upgrade Cilium (if needed)

After Kubernetes upgrades, you may need to upgrade Cilium:

# Check Cilium compatibility
# https://docs.cilium.io/en/stable/network/kubernetes/compatibility/

# Update Cilium HelmRelease version
vim kubernetes/apps/kube-system/cilium/app/helmrelease.yaml

# Commit and push - Flux will upgrade
git add -A && git commit -m "chore: upgrade Cilium" && git push

Troubleshooting

API Server Not Starting

# Check API server logs
talosctl -n 172.16.1.50 logs kube-apiserver

# Check etcd connectivity
talosctl -n 172.16.1.50 etcd status

Kubelet Issues

# Check kubelet logs
talosctl -n <node-ip> logs kubelet

# Check kubelet status
talosctl -n <node-ip> service kubelet

Pods Stuck After Upgrade

# Check for pending pods
kubectl get pods -A | grep -v Running

# Check events
kubectl get events -A --sort-by='.lastTimestamp' | tail -20

# Restart stuck pods
kubectl delete pod <pod-name> -n <namespace>

Version Skew Issues

Kubernetes supports n-2 minor version skew. If you skip versions:

# Upgrade incrementally
# v1.32 → v1.33 → v1.34

Rollback

Downgrades are not recommended. If you need to recover, prefer restoring from a known-good etcd snapshot.

Upgrade Schedule

Type Frequency Notes
Patch (x.x.X) Monthly Security fixes
Minor (x.X.0) Quarterly New features
Major (X.0.0) Yearly Breaking changes

References