My single-node k3s server had 421 GB used on a 2 TB disk. That looked fine, so I never looked closer. Today the disk is at 146 GB. Nothing I run changed; I only deleted images that no container used.
How I found it
I was shutting down an old side project. Its image is about 4.2 GB (Python, data libraries, the usual), and CI pushed a new version for every change. The node had kept every version it ever pulled:
sudo k3s crictl images | grep <project>
# ...:latest 4.2GB
# ...:<none> 4.2GB
# ...:m18.5-shipped 4.2GB
# ...:<none> 4.2GB
# ... 55 lines like this
The <none> entries are earlier pulls of :latest. Each rollout pulled a
new image and left the old one in containerd’s store. After removing that
project’s images the disk was at 154 GB. Pruning everything else no
container uses took it to 146 GB:
sudo k3s crictl rmi --prune
That went from 166 images to 58. Pods kept running; anything a CronJob needs later is simply pulled again.
A side note if you do this: most of my crictl rmi calls failed with
context deadline exceeded. They had not failed. containerd kept deleting
in the background, and the next crictl images showed them gone.
Why the kubelet never cleaned up
Kubernetes does garbage-collect unused images. The question is when. Ask your kubelet:
kubectl get --raw "/api/v1/nodes/<node>/proxy/configz" \
| jq '.kubeletconfig | {imageGCHighThresholdPercent, imageGCLowThresholdPercent, imageMaximumGCAge}'
{
"imageGCHighThresholdPercent": 85,
"imageGCLowThresholdPercent": 80,
"imageMaximumGCAge": "0s"
}
Those are the defaults. Image GC starts when the image filesystem is 85% full and deletes the least recently used images until it is back at 80%. On k3s the image filesystem is usually just your root disk. On my 2 TB disk, 85% is about 1.7 TB. With 421 GB used, another 1.3 TB of images would have had to pile up before the kubelet deleted a single one.
The thresholds make sense on cloud nodes with small disks that get replaced often. On a long-lived VPS with a big disk they mean “never”.
imageMaximumGCAge: 0s is the other half: the time-based limit, which
deletes images unused for longer than a set age regardless of disk usage,
is off by default.
I had even looked at this problem before, just in the wrong place. My server has a weekly timer that prunes Podman images, with a comment that says “k3s uses its own containerd and is unaffected by this prune”. True, and nothing else was pruning the k3s side.
Check your node
# how much the image store uses, from the kubelet's point of view
kubectl get --raw "/api/v1/nodes/<node>/proxy/stats/summary" \
| jq '.node.runtime.imageFs | {capacityBytes, usedBytes}'
# the biggest images, and how many copies of each repository
sudo k3s crictl images | sort -k4 -h | tail
sudo k3s crictl images | awk 'NR>1 {print $1}' | sort | uniq -c | sort -rn | head
If one repository shows up dozens of times, you have the same problem.
The fix
Set a maximum age, so unused images go away even when the disk is nearly
empty. k3s reads kubelet config drop-ins from
/var/lib/rancher/k3s/agent/etc/kubelet.conf.d/ (it puts its own defaults
in 00-k3s-defaults.conf there):
# /var/lib/rancher/k3s/agent/etc/kubelet.conf.d/50-image-gc.conf
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
imageMaximumGCAge: 168h
I use a week: an image a running pod uses is never “unused”, and anything else can be pulled again when it is needed. After the next k3s restart the kubelet confirms it:
kubectl get --raw "/api/v1/nodes/<node>/proxy/configz" | jq .kubeletconfig.imageMaximumGCAge
# "168h0m0s"
One catch from the
Kubernetes docs:
“This feature does not track image usage across kubelet restarts.” On k3s
the kubelet runs inside the k3s process, so every k3s restart, including
every upgrade, resets the clock. If you restart more often than the age
you set, nothing gets deleted by age. For that case, a lower
imageGCHighThresholdPercent or a scheduled crictl rmi --prune is the
more reliable option.