I let an AI agent upgrade the k3s cluster it runs on

My coding agent runs on the same VPS as my k3s cluster. On a Sunday I asked it to upgrade Kubernetes. An hour later the node had gone from v1.35.5 to v1.36.5 to v1.37.1, the API had been gone for well under a minute in total, and nothing I care about broke.

That is not the interesting part. The interesting parts are what the agent checked without being asked, the one thing it broke, and how it proved that its repair was exact.

The setup

One VPS, k3s with the SQLite datastore, about 70 pods. Pulumi installs k3s with a remote.Command, and the version is a constant:

K3S_VERSION = "v1.36.5+k3s1"

install = remote.Command(
    "install-k3s",
    connection=conn,
    create=f'curl -sfL https://get.k3s.io | sudo INSTALL_K3S_VERSION="{K3S_VERSION}" ... sh -',
)

Changing that string changes the command, and Pulumi re-runs it. The k3s installer replaces the binary and restarts the service. So the upgrade itself is a one-line diff, and pulumi preview tells you exactly what it will touch:

 ~  command:remote:Command install-k3s update [diff: ~create]
Resources:
    ~ 1 to update
    15 unchanged

One resource, both times. If the preview had shown anything else, that would have been the moment to stop.

What it checked before touching anything

I did not give it a checklist. It came up with this one:

• Deprecated APIs actually in use, from the API server’s own metric:

  kubectl get --raw /metrics | grep '^apiserver_requested_deprecated_apis'
  

  Only endpoints v1: deprecated, not removed.

• The release notes against the cluster, not in general. For 1.36: no gitRepo volumes, no Service.spec.externalIPs. For 1.37 it read the urgent upgrade notes and checked each item: SELinux is not enabled here, there are no scheduling.k8s.io/v1alpha2 objects, none of the removed kubelet flags or feature gates are in the k3s config, the node runs cgroup v2.

• A backup of the datastore. k3s etcd-snapshot only covers etcd. With SQLite, the backup is a SQLite backup, and it works while k3s runs:

  sqlite3 /var/lib/rancher/k3s/server/db/state.db ".backup /var/backups/k3s/state-pre-1.36.db"
  sqlite3 /var/backups/k3s/state-pre-1.36.db "pragma integrity_check;"
  

• It asked me which version. Patch only, one minor step, or both in two runs. Not a decision I wanted an agent to make for me.

When I later asked for 1.37, it told me the k3s stable channel still pointed to 1.36.5. I asked it to verify that instead of trusting it. One call to the channel API settled it:

curl -s https://update.k3s.io/v1-release/channels
# stable  v1.36.5+k3s1
# latest  v1.37.1+k3s1

“Stable” is a channel, not a property of a version. I went to 1.37.1 anyway, with the checks above.

During the upgrade

Running pods are not restarted when k3s restarts; containerd keeps them going. CoreDNS and metrics-server failed their readiness probes for about 20 seconds and recovered on their own.

Halfway through the first run I got nervous about something else: this cluster signs service-account tokens under a public OIDC issuer that AWS trusts (how that works). If an upgrade rotated the signing key, every AWS call from the cluster would start failing within the hour.

The agent answered with a check, not a statement. The key ID the cluster serves must match the one in the copy AWS reads:

kubectl get --raw /openid/v1/jwks | jq -r '.keys[].kid'
curl -s https://<bucket>.s3.eu-central-1.amazonaws.com/openid/v1/jwks | jq -r '.keys[].kid'

Same kid before and after both upgrades. The key file on disk still had its April timestamp; k3s only rewrites a working copy of it on every start.

What it broke

Between the two upgrades I had it remove an old side project from the cluster. To edit one of my Pulumi files it ran a small Python script that did this:

open(p, "w").write(strip(open(p).read()))

Python evaluates open(p, "w") first. That truncates the file. Then it reads an empty file, and strip fails. The result was an empty file, and that file also held uncommitted work of mine.

It noticed on the next command, said so plainly, and repaired it from two things it still had: the committed version from HEAD, and a git diff it had printed half an hour earlier, which showed my change in full. Rebuilding the file is easy. Knowing the rebuild is exact is the part that matters, and git gives you that for free:

diff --git a/<file> b/<file>
index ba9321e..b32f0e6 100644

The earlier diff had shown my working-tree version as blob b32f0e6. The rebuilt file hashed to b32f0e6 too. Same hash, same bytes. No “I think it looks right”.

Guardrails that did their job

• The permission check said no, even after I said yes. In the same session it needed App Store Connect API credentials. Its first idea was to search my home directory for the key file. Claude Code’s permission classifier blocked that as credential exploration, and kept blocking it after I typed “ok, use them”. I pointed it to the Pulumi config where the credentials actually live, and it read them from there. Annoying for a minute, right in hindsight.
• It only committed its own lines. My uncommitted change and its change sat in the same file. It wrote the version without my change straight into the git index (git hash-object -w plus git update-index --cacheinfo), so the commit contained only its part and my work stayed unstaged, where I had left it.
• Every live change ended up in a PR. The upgrades were applied with pulumi up first, then committed, so main describes what actually runs.

What I would do the same way again

1. Keep the version in code, so an upgrade is a diff and the preview is the contract.
2. Let the agent do the reading: release notes against this cluster is exactly the tedious work it is good at.
3. Make it ask for decisions that are mine (which version) and never for ones that are not (whether to back up).
4. Commit or stash your own work before an agent edits the same repo. It cost me nothing this time because a diff happened to be on screen.
5. Ask for proof, not reassurance: a matching kid, a matching blob hash.

And one thing it found that I had never looked at: the node was carrying about 270 GB of container images nobody used. That is the next post.