This article is part 10 of my MisLinux series on Kubernetes on Hetzner. It is based on a real failure in a self-hosted production registry I operate. I am not affiliated with Docker, CNCF, or any vendor mentioned here.
Multi-architecture images are one of those things that feel clean in theory and treacherous in practice.
In theory, you publish one image tag, Kubernetes pulls the right platform-specific manifest for each node, and everything is elegant. In practice, that elegance depends on the registry handling manifest lists, referenced sub-manifests, and garbage collection exactly the way you think it does.
Mine did not.
The incident in one sentence
I ran garbage collection on a self-hosted Docker registry and ended up with a perfectly normal-looking multi-arch tag whose platform-specific image manifests had been swept out from under it.
That is the kind of breakage that destroys confidence because it does not look broken at first glance. The tag can still exist. The manifest list can still exist. But a real pull on one architecture fails because the child manifest or referenced blobs are gone.
Why this was not just "I deleted the wrong thing"
The important nuance is that this was not simply an operator typo. It was a mismatch between what I assumed garbage collection would protect and what the registry implementation actually considered reachable.
I had a manifest list like this in conceptual terms:
latestlinux/amd64child manifestlinux/arm64child manifest
The mistake was assuming the top-level tag was enough protection for the architecture-specific manifests beneath it. That assumption is intuitive. It is also operationally dangerous if your registry version or GC behavior does not traverse those references the way you expect.
In my case, GC reclaimed the platform-specific manifests even though the higher-level multi-arch tag still existed.
Why this hurt more than a normal image mistake
If a single-architecture tag breaks, the blast radius is often obvious. The affected workload on the affected architecture fails quickly. You investigate the exact tag.
Multi-arch breakage is more deceptive because different nodes can see different realities:
- one architecture may still pull successfully because it already has the image cached
- another architecture may fail on the next real pull
- a running deployment may look healthy until a rollout, restart, or node replacement happens
That delay is what makes the failure expensive. The registry problem is created earlier than it is discovered.
The cleanup assumption I had to unlearn
I had been treating some older immutable build tags too casually because of their naming style. In my environment, the Rocker-derived tags look like this:
0_<commit>_<commit>_<timestamp>_--no-wait
At a glance, they look disposable. They are not.
Those tags encode source-commit identity for the build inputs. The stale --no-wait suffix is historical baggage from an old Satis-related flag, not evidence that the tag is transient. Once I re-centered on that fact, the cleanup policy had to change. The question stopped being "which ugly tags look safe to prune?" and became "which tags are still protecting manifests that matter?"
What I changed in the cleanup rules
My cleanup rule is stricter now:
- never assume a weird-looking immutable tag is disposable
- never assume a
latestmanifest list fully protects its child manifests - keep direct architecture-specific tags for manifests that back a live multi-arch tag
- prefer reachability- and deployment-aware cleanup over name-based cleanup
In practice, that means a safe cleanup process should understand at least three things:
| Question | Why it matters |
|---|---|
Is this tag the current latest? | latest still matters operationally in many environments, even when I prefer digest pinning |
| Is this digest currently referenced by a live Deployment, DaemonSet, CronJob, or Job? | Deleting its tags may not break the running pod immediately, but it removes the safe pull path for restart or scale-out |
| Is this immutable tag the only direct reference protecting a child manifest under a multi-arch tag? | Removing it can silently corrupt the apparent safety of the multi-arch tag |
This is more complex than "keep the N newest tags," but complexity is better than pretending registry cleanup is harmless.
Why self-hosting a registry changes the standard advice
On a managed registry, a lot of this implementation detail disappears behind a service boundary. On a self-hosted registry, especially one that has already seen aggressive cleanup or storage pressure, you own the consequences directly:
- disk pressure
- GC semantics
- manifest reachability assumptions
- downtime risk from re-pulls
- differences between cached images and actually pullable images
That does not make self-hosting wrong. It just means registry maintenance is not janitorial work. It is production reliability work.
The verification step I now treat as mandatory
The most important new habit is manifest inspection before and after cleanup.
I want to know:
- what the top-level tag points to
- which architecture descriptors exist under it
- whether those per-arch manifests also have direct tags
- whether the live cluster is still referencing an older digest
If I cannot answer those questions quickly, I do not trust the cleanup operation.
The broader lesson is that registries are not just blob stores. They are graph stores with operational consequences. When GC runs, it is traversing a reference graph. If your mental model of that graph is wrong, the cleanup can be "successful" and still destroy a production-safe pull path.
Why this changed my view of image hygiene
Before this incident, I thought of registry cleanup mostly as a disk-usage discipline problem. Afterward, I think of it as a correctness problem first and a storage problem second.
Of course the registry needs cleanup. A self-hosted registry cannot grow forever. But correctness has to come first:
- live tags must remain pullable
- immutable provenance tags must not be treated as trash
- multi-arch child manifests must stay directly protected
- cleanup heuristics must reflect deployment reality, not naming aesthetics
If that means the registry stays a little fatter until the logic is right, that is a better trade than recovering from a broken production image path.
Final thought
The phrase "garbage collection" sounds like maintenance. In a production registry, it is really a graph-deletion operation with platform-specific consequences.
That is the real lesson I took from this failure: a multi-arch tag is only as safe as the child manifests and tags that still exist behind it. If your cleanup policy cannot prove those relationships before it deletes anything, it is not conservative enough yet.
If you want the broader platform context for why I care so much about mixed ARM64 and AMD64 pull safety, read my earlier post on handling multi-architecture images in a mixed-architecture Kubernetes cluster. This post is the production failure mode that taught me what that architecture really costs in maintenance discipline.
