Toolbox

Layer ordering by Invalidation Floor: move the fetch-stage COPYs below the RUN tail

Status: accepted

Every figure below is as measured when this decision was taken — evidence for the choice, not a description of the repo today. Nothing here is kept in sync; current values live in the files that set them.

The Dockerfile's build-strategy header promised that "a Renovate bump of one tool re-runs only that stage + its COPY — never the tail". Measured against the published GHCR manifests, it does not hold: 0.83.0sha-08ef69b, whose only image-affecting change is a one-line OMZ_COMMIT bump, moves 34 of 71 layers and 586 MB of 1157 MB — 51% of the image, for a shallow git clone worth ~30 MB. COPY --link builds each layer independently of the filesystem beneath it, so the 28 fetch-stage COPYs do not invalidate each other; but all 28 are declared above the entire RUN tail, and a RUN is invalidated by anything that precedes it. The rare→frequent ordering of the tail, measured by Renovate cadence, therefore only protects against bumps of the npm/pip CLIs installed in the tail. It does nothing for the ~289 bumps per six months that land in a fetch-* stage (OMZ_COMMIT 49, UV 33, ZSH_COMPLETIONS_COMMIT 29, GLAB 21, HOMEBREW 19, GCLOUD 17, RTK 15, DOCKER_CLI 14, …), each of which costs every puller 586 MB.

We therefore order layers by Invalidation Floor (see CONTEXT.md): the highest layer a change touches, and hence the boundary below which everything is rebuilt with a fresh digest. The rule that replaces the header's promise is not rare→frequent among the RUNs — it is how few consumers each COPY has below it. Concretely: all 29 build-stage COPYs (28 fetch-* plus rtk-builder) move to just above USER toolbox, with no RUN left below them. That requires dissolving the three RUNs that consumed copied files: the shared completions layer (nine <tool> completion zsh invocations, moved into the respective fetch stages following the pattern bat/fd/eza/brew already use, leaving only pnpm and codex to generate theirs in their own install layers); the Homebrew safe.directory layer (its test -x was a duplicate — fetch-brew already runs the same check on its own /out, and the smoke test runs brew --version as the runtime user); and the user-setup layer (usermod -d -m plus chmod -R a+rwX /home/toolbox, resolved by having fetch-omz clone straight into /out/home/toolbox and set the permissions itself, so -m has nothing left to relocate). Afterwards a fetch-stage bump moves exactly one layer.

Separately and as a precondition, the fetch stages normalise mtimes (freeze-mtimes, a touch -d @1 over /out). Files in the image today carry the wall-clock time of whichever build last ran that stage — jq 2026-06-20, kubectl 2026-07-23, go 2026-08-11 — and COPY --link folds mtime into the layer digest. The 28 COPY digests are therefore stable only for as long as BuildKit reuses the stage without re-executing it. Losing or rotating buildcache-main would rebuild all of them with fresh mtimes and push the entire 1157 MB to every user with no Dockerfile change at all. Without reproducible digests the layer-count gate below measures cache luck rather than the floor.

Considered Options

Leave the ordering and lower the image weight instead. Attacks the symptom: the tail would still be invalidated wholesale, just a cheaper wholesale. The same ~289 bumps would keep re-downloading whatever the tail weighs.

Intra-layer delta transfer (zstd:chunked / eStargz / SOCI). Would make the question moot by shipping only changed chunks, but requires the containerd snapshotter plus a runtime and registry that negotiate it; on Docker Desktop against GHCR it is not available today. Revisit if the containerd snapshotter becomes the default.

A static test asserting no RUN below the COPYs names a fetch-provided binary. Rejected: parsing binary names out of concatenated shell inside RUN directives is precisely the "clever regex" this repo already identifies as the fragile half of TestSmokeTestVendorCompletionsFloor. The real case is already covered — every such RUN runs under set -eux, so a missing binary fails the build immediately and legibly.

A pre-merge gate in docker-ci.yml. It builds amd64 with load: true, so it could block before the merge. But a locally loaded image exposes uncompressed diffIDs while the remote manifest exposes compressed digests; comparing them means fetching and parsing latest's config blob. The post-merge comparison in docker-publish-reusable.yml is manifest-against-manifest, needs no pull, and costs one release cycle of latency instead of that machinery.

A megabyte threshold rather than a layer-count threshold. A GO_VERSION bump legitimately moves ~120 MB, so a byte threshold reddens on healthy changes while passing a regression that invalidates thirty small layers. Layer count describes the floor directly; the byte figure stays in the job log for readability.

Consequences

Follow-up (2026-08-18): a block of version ARGs defeated the ordering

The Consequences above state that "a tail bump is unchanged — it was already cheap". That was wrong, and the gate this ADR introduced is what surfaced it: the first genuine comparison after the reordering rejected a one-line OCI_VERSION bump at 16 substantial layers, 694 MB.

The cause was not the ordering but the scope of the version ARGs. All 14 of them were declared as one block at the top of the final stage, and a build ARG that is in scope lands in the cache key of every RUN below it — the |16 prefix docker history printed on every tail layer. So each of the 21 tail RUNs was keyed on all 14 versions: bumping any single tool gave every tail RUN a new key, the whole tail rebuilt, and the layers came back with new digests.

While that held, the rare→frequent ordering could not do anything. The position of a RUN only matters if a bump invalidates some RUNs; here every bump invalidated all of them, so ordering them by Renovate cadence bought nothing. Measured with .github/scripts/invalidation-floor.sh against the published sha-<commit> tags, post-reordering:

Transition Change Layers > 1 MB MB
280061612ade4b dockerfile ARG bump 31 966
12ade4bacf2843 OCI_VERSION bump + one init.d asset 16 694

Each version ARG is now declared immediately above the single RUN that consumes it, so a RUN is keyed only on the versions it actually uses and the ordering finally takes effect. TestFinalStageARGsScopedToTheirRUN holds the placement over the embedded Dockerfile.

Like the mtime normalisation, the move rebuilds every tail layer once and ships with a single-use [floor-reset].

Necessary, not sufficient. Scoping the ARGs only restores the premise of the ordering; it does not by itself bring a bump under the gate's max of 3. Now that cost is positional, the order itself matters — and it does not match the cadence measured in docs/internals/image-build.md: oci (20 bumps in 6 months) sits 4th of 21 while codegraph (15) sits 9th, so a mid-cadence bump still moves more than three substantial layers. Closing that gap needs a reorder by measured cadence and a MAX_LAYERS recalibrated to the resulting worst case. Neither is done here — both land in follow-up 2 below, and the reorder carries a hazard worth writing down: moving the oci-cli RUN below the graphifyy one inverts pip dependency resolution, and the build verifies graphify before installing oci, so a break there would pass green. Sequence the reorder so the pip pair keeps its relative order, or add graphify's import check after the oci install.

Two observations from the same investigation are not addressed here, and neither is proven:

Follow-up 2 (2026-08-19): the ordering, now that it does something

Scoping the ARGs (follow-up 1) made position matter. Three publishes on main then measured what position is worth, and confirmed both halves of that follow-up's prediction:

Publish Bump Where it sat Layers > 1 MB MB Gate
80f56e9 claude-code last version RUN 2 101 pass
a9839f8 codex 7th of 13 7 320 fail
6b112a8 yq a fetch-* stage 2 10 pass

Before the scoping, every one of these would have moved 16-31 substantial layers and 600-966 MB. So the cost of a bump is now (number of version RUNs at or below it) + 1 — the +1 is a trailing non-version layer, measured by attributing the claude-code publish: its own 96 MB npm layer plus one below it.

Two consequences follow, and this commit takes both.

The tail is reordered by re-measured cadence (6-month window): graphifyy 100, claude-code 95, wrangler 37, pnpm 36, codex 34, oci 20, codegraph 15, playwright-cli 10, cf 8, azure 7, playwright 7, pyright 5, typescript 2. Least first, most last. Two orderings inside that are load-bearing rather than aesthetic, and both fall out of the cadence order anyway: oci stays above graphifyy, so the two pip installs keep resolving shared dependencies in the same order as before — the build verifies graphify before installing oci, so an inversion would ship broken and pass green — and playwright stays above playwright-cli.

MAX_LAYERS moves from 3 to 6, and moves into the script, so the calibration is one literal that CI reads rather than two that can drift. 6 is the cost of the fifth-most-bumped tool once ordered, so it admits graphifyy, claude-code, wrangler, pnpm and codex — 302 of the 376 tail bumps in the window, 80% — while the structural regression the gate exists for, measured at 16-31 layers, still fails by a factor of 3 to 5. The --self-test fixture now derives its layer count from MAX_LAYERS instead of hardcoding four. At a hardcoded four the self-test does not go quiet when the bound is raised — it goes red, measured: MAX_LAYERS=6 against the old fixture prints self-test FAILED: regression accepted and exits 1, because four moved layers no longer exceed six. So the fixture had to move with the bound either way; deriving it means the next person to change the threshold cannot fix that red by weakening the fixture instead.

The baseline was the other half of the noise, and it is fixed here. The gate resolved its baseline from :latest, which is mutable and lags whenever a publish is cancelled by the next push — this repo cancels them and re-runs them. Two costs followed. One commit was billed for another's churn: OCI_VERSION was bumped in fb8dcca, whose publish was cancelled, so :latest never advanced and the gate charged the cost to acf2843. And a re-run of an already-published commit resolved :latest to the manifest it was about to push and compared it against itself, which is the unexplained Moved 0 layer(s) of 70 in run 32074128559 — no longer a hypothesis. The baseline is now the immutable sha-<commit> tag of the commit the push replaced, falling back to :latest; and a baseline that equals the manifest just pushed is reported as "no comparison performed" instead of being banked as a green.

How often the immutable half actually lands, measured after the fact: only when the commit the push replaced happened to publish. docker-publish.yml is path-filtered to internal/build/assets/** and its own two workflow files, and over the twenty main commits ending at fc6f8c7, 11 matched and 9 did not — so github.event.before names a commit with no sha- tag roughly half the time, and the :latest fallback is the ordinary path rather than the exception. That is not a degradation: :latest is the last published manifest, which is the baseline the comparison wants. What the fallback loses is attribution, and only the part of it that was never recoverable — bytes from a cancelled publish were never delivered, so the next publish genuinely does push them.

What this still does not fix, and one idea that does not work. oci (20 bumps) is bounded at 7 and codegraph (15) at 8, so roughly 75 bumps per window — about 12 a month — will still redden a publish. An earlier draft of this section proposed the durable fix as a positional invariant: no layer above the highest changed instruction may move. That does not work, and the case that motivated this whole ADR is the counter-example. With the fetch COPYs declared above the tail, an OMZ_COMMIT bump changed a COPY high in the file and moved the 34 layers below it — every moved layer sits at or below the change, so a positional rule passes it. Legitimate bump and structural regression have the same shape; they differ only in magnitude, which is what a count measures. So the count is the right instrument and the residual noise is not an artefact of it: a bump high in a 13-deep tail is genuinely expensive, and the gate saying so is the gate working. The way out is fewer substantial layers in the tail — moving npm/pip installs into fetch-* stages where each bump costs one --link layer, as the six tools in that table already do — not a cleverer bound.

That way out is taken for the two worst offenders in the same change. oci (20 bumps, bounded at 7) and codegraph (15, bounded at 8) now install in fetch-oci and fetch-codegraph, so each bump moves one layer whatever its position, and the tail drops from 13 version blocks to 11 — which also takes two off the bounded cost of everything that sat above them. Coverage against MAX_LAYERS = 6 goes from 302 of 376 tail bumps to about 337: roughly 6 red publishes a month rather than 12. fetch-codegraph has to use the same node image as the final stage, because an npm global tree is only valid on the runtime that resolved it. fetch-oci installs into a venv at /opt/oci-cli rather than the system site-packages, which also ends a coupling this ADR had only flagged: oci and graphifyy shared site-packages, so whichever pip ran last decided the version of every dependency they have in common, and the only check that would have caught a break — graphify's import test — ran before oci was installed. What is left over the bound is playwright-cli (10), cf (8), azure (7), playwright (7), pyright (5) and typescript (2); the same treatment applies to any of them.

Follow-up 3 (2026-08-26): what the count was actually counting

Follow-up 2 closed with "the count is the right instrument and the residual noise is not an artefact of it". Half of that holds. The instrument is right; the noise was not all residual, and a share of it never measured ordering at all. Issue #778 is the case that separates them: run 32919323729 on main (14bb4c0) failed at 16 substantial layers, 639 MB, and the diff it was charged for — 2fe107a..14bb4c0 — touches one image-affecting line, GCLOUD_VERSION 581.0.0 → 582.0.0.

Attributing every moved layer to its created_by, against the published manifests:

Cause Layers > 1 MB MB Where
The diff 1 52 idx 64, COPY --link --from=fetch-gcloud
Archive Drift 12 587 idx 6, the final stage's apt layer, plus the 11 tail RUNs beneath it
Fetch Nondeterminism 3 21 idx 65 fetch-omz, 66 fetch-brew, 69 rtk-builder

Both new terms are defined in CONTEXT.md. The base image did not move: the first five layers are byte-identical across the two manifests, Debian's own included. The drift enters at layer 6, which is ours — apt-get install with no version pins — and cascades through everything parent-chained below it. The three COPY --link layers are a second, unrelated defect: those stages re-executed because their shared fetch-base moved, and unlike the other 27 they do not produce identical bytes twice. freeze-mtimes made the fetch layers reproducible against timestamps; it says nothing about content.

Corrected: idx 69 is not one of them — on amd64, which is the arch this table measures. The arm64 leg of the same stage is a separate finding, below. Two --no-cache builds of a minimal reproduction show rtk's amd64 binary — a checksummed upstream tarball — identical across builds while its layer digest was not. It moved because the COPY named a single file: a --link layer is built independently of what is beneath it, so it materialises usr/, usr/local/ and usr/local/bin/ itself, and those carry the build's wall clock however thoroughly the stage froze the file. The row above is therefore 2 stages of genuine Fetch Nondeterminism plus one mis-attributed copy; the fix is to copy /out/ /, as every fetch stage already does, and TestBuildStageCOPYsCopyWholeTree holds the shape for the class.

Closed (measured 2026-09-05, not when this decision was taken). The figures below post-date the header's "as measured when this decision was taken" by the length of this follow-up; read them as of that date.

freeze-git runs over every checkout fetch-omz and fetch-brew ship, before the chmod and before freeze-mtimes. Neither can drop .git — Homebrew is a git checkout and omz update needs one — so the three things a fresh checkout takes from the clock or from the far end are normalised instead: the pack is recompressed locally, the reflog is dropped, and the index is read back from HEAD with its stat cache zeroed. Measured on a native arm64 daemon, as the digest of the COPY --link --from=<stage> /out/ / layer alone across two independent --no-cache builds:

Stage Before After
fetch-omz 8c49c223… / 967b5595… 427f1fb7… twice
fetch-brew c384c0b9… / a89fff12… 79cb2466… twice

-F, not -f, is what does it, and the difference is not cosmetic: -f refuses the deltas the server sent but still copies each non-delta object's compressed bytes out of the received pack. Two fetch-brew builds minutes apart produced packs with identical write order and identical delta bases whose per-object sizes in pack differed — GitHub had deflated the same blobs differently. TestFreezeGitMakesACheckoutAFunctionOfItsCommit reassembles the helper and runs it over two checkouts that disagree the same way.

The arm64 leg of rtk-builder is Fetch Nondeterminism too, and the first measurement of it was wrong. Two --no-cache builds on an arm64 daemon gave the same layer digest (0ee1364c…) — but both pulled the same floating base and shared the cargo registry cache mount, so the experiment held constant exactly the input under suspicion. Varying the toolchain instead, at one RTK_VERSION and one --locked dependency graph:

Toolchain arm64 rtk binary
rustc 1.98.0 0d114e8daf093e81…
rustc 1.97.1 0c1a18b805e6aa54…

So /out is not a function of the version the stage pins, which is this term and not Archive Drift: what moves is the stage's own output, not the layers it sits on. The distinction turns on what the base image is here — for every other FROM in the file it hosts a download, and for this one it is compiled into the result.

Holding the base still would close it — a digest pin, with pinDigests so Renovate re-resolves the same tag and the pin does not become a frozen toolchain. That was built and then withdrawn: re-resolving the same tag means a PR on every upstream rebuild, and the churn buys nothing the gate can see, because the moved-layer count is measured on amd64 only and amd64's binary is a checksummed tarball that is identical across builds.

So the arm64 leg is accepted, not fixed — issue direction 4, and it is accepted at the stage rather than as an allowance subtracted in the gate, which never measures it. The honest statement is that the image published for arm64 carries a stage whose output is not a function of the version it pins, and that this costs nothing today only because nothing measures arm64. Revisit if the gate grows an arm64 leg.

The issue's first proposed direction is not merely weak, it is fatal. Comparing only layers whose created_by differs would have counted zero here — and zero for every regression this ADR exists to catch. Measured: all 70 created_by strings are byte-identical between the two images. A build ARG's value never reaches created_by, which carries only the |N argument count and the command text, so the follow-up 1 regression (one ARG bump rebuilding the whole tail) and a pure reorder of the RUNs both leave it untouched. The filter would not have softened the gate; it would have retired it.

The rule

The image has two kinds of layer, and only one of them cascades. A RUN is invalidated by anything above it; a COPY --link is built independently of the filesystem beneath it. So:

If the final stage's first RUN layer moved, every RUN layer moving is expected, and only the COPY --link layers are counted. If it did not move and RUN layers moved anyway, that is the regression the gate exists for, and everything is counted as before.

The first RUN layer of the final stage is the apt layer, and it is the one instruction in the image that no version bump can reach. It is identified structurally, as the first layer whose created_by begins with RUN | — the node image's own layers are RUN /bin/sh -c, with no build args and hence no prefix. Counting the args in that prefix would be more expressive and would break silently the day a global ARG is added; a fixed index would break sooner. docker buildx imagetools inspect <ref>@<digest> --format '' returns the history, which costs one call per image and no new tooling.

Against every transition measured so far:

Transition apt layer moved Verdict Correct
2fe107a14bb4c0 (#778) yes 4 counted, pass yes
12ade4bacf2843 (OCI_VERSION, follow-up 1) no 16 counted, fail yes
OMZ_COMMIT pre-reordering no 34 counted, fail yes
a pure reorder of the tail no counted, fail yes

The layers do not form a clean prefix and a clean block — two RUNs sit interleaved among the asset COPYs at idx 27 and 38 — so the split is by instruction kind, never by index range. Both weigh under MIN_BYTES and have never counted, but a positional reading of the same rule would be wrong today, not eventually.

The hole this leaves. A genuine ordering regression that lands on the same day as an archive update is excused. It is caught by the next publish without drift, which costs one release cycle — the same latency this ADR already accepts for reporting after the push rather than before it.

Calibration, failure, and what the log says

MAX_LAYERS stays at 6 and stays one literal. It was calibrated on the drift-free path, which is where it still applies unchanged; on the excused path it bounds a population that measures 4 today. Two literals would be two things that drift apart, which is why follow-up 2 moved this one into the script in the first place.

A history that cannot be fetched is a notice and a pass, matching the existing treatment of a missing baseline and for the same reason: a publish that could not measure must not redden. Falling back to the old unclassified count would reproduce exactly the unactionable red being removed here, at the moment nobody would understand why.

The log keeps one line, unchanged when the canary is quiet and naming the split when it fires:

Moved 34 layer(s) of 70, 16 above 1 MB, 639 MB total — 12 excused as archive drift, 4 counted (max 6).

The real cost to pullers stays the first thing on the line. Invalidation Floor in CONTEXT.md is deliberately left alone: it defines what a change costs the people pulling the image, and that cost is real whoever caused it. What changes is only which part of it we treat as a regression.

The invariant, and a test for it

The canary depends on a fact nothing currently pins: the final stage's first RUN has no version ARG in scope. Put one there and the canary fires on a legitimate bump, excusing the entire tail — a gate that goes green in silence, which is the worst way it can break. TestFinalStageARGsScopedToTheirRUN gains an assertion that only TARGETARCH and DEBIAN_FRONTEND appear between the final FROM and the first RUN. It goes in that file because it is the same parse read by a second consumer, and its comment has to name the canary — read without that, it looks like pedantry and gets deleted.

The --self-test fixtures grow a third column for created_by and two cases: drift excused, and drift plus a real regression among the COPYs, so the new branch asserts something in both directions. The second derives its layer count from MAX_LAYERS, as the existing fixture already does.

What this does not fix

Nothing here reduces a single byte anyone downloads. Archive Drift is real transfer, and 587 MB of it. What removes it is the direction follow-up 2 already named — fewer substantial layers parent-chained below the apt layer — and #778 is further evidence for it rather than an alternative to it: with the tail emptied, an archive update moves the base layer and little else. Fetch Nondeterminism is a separate defect with a separate remedy and is tracked in issue #780; without it the worst case does not stay under the bound, because three layers move on every cache miss regardless of ordering.

Three changes, in this order: the measure, then the tail, then the reproducibility. The measure comes first because the other two have no honest signal to be judged by until it does.