GitHub Actions in Production, Part 3: Docker Layer Caching and Graceful Degradation

This is the workflow I'd most want to walk someone through in an interview, because it has my favourite decision in the whole library and my most instructive bug sitting about four lines apart.

The decision: the pipeline checks whether it has credentials before it tries to use them, and does something useful either way. With Docker Hub secrets set, it builds and pushes. Without them, it still builds, tags locally, and says plainly that it didn't push. Someone who forks the repo gets a real build validation instead of a red X caused by a secret they can never have.

The bug: the tagging block right underneath uses nested ${{ }} expressions inside a format() call, which isn't valid GitHub Actions syntax. It would fail at evaluation time. In a template that never runs in the repository it lives in.

Introduction#

Both halves are worth writing about. Graceful degradation is a pattern I've since applied in three other places and I think more pipelines should use it. And the bug is a small lesson about template libraries specifically: code that isn't executed where it lives will rot, and being careful is not a substitute for running it.

Why this workflow exists#

Container images are the deploy artifact for most of these services, and moving the build into CI rather than someone's laptop solved three separate things.

Reproducibility first. An image built on my machine inherits my Docker cache, whatever base image I happened to pull three weeks ago, and my architecture. An image built on a clean runner from a clean checkout is defined by the Dockerfile and the lockfile and nothing else. Once you've spent an afternoon debugging an "it works locally" problem that turned out to be a stale cached layer from a previous month, you stop building release images locally. I've spent that afternoon.

Traceability second. "What commit is that container running?" needs an answer, and tagging only latest guarantees it doesn't have one.

And decoupling build from deploy. Publishing to a registry means the image exists independently of any environment. ECS can pull it, a staging box can pull it, a colleague reproducing a bug can pull it, and you can pull it three weeks later to roll back. That decoupling is basically the entire point of a registry, and it's why this workflow lives separately from the deploy workflows rather than being a step inside them.

The credential probing came from an incident rather than a principle, which is usually how the good patterns arrive. Someone forked a repo, opened a PR, and CI failed because docker/login-action got an empty username. GitHub deliberately withholds secrets from pull_request runs originating in forks, and that's correct behaviour, otherwise anyone could open a PR that modifies the workflow to print your registry token. But the contributor was staring at a failure with no possible fix on their end, and that's a rubbish experience.

So I restructured it to ask what it can do instead of assuming what it will do.

Architecture#

push:live / workflow_dispatch
          │
          ▼
    checkout ──► setup-buildx
          │
          ▼
  ┌───────────────────────────┐
  │  probe: is DOCKER_USER    │
  │  secret non-empty?        │
  └────────┬──────────┬───────┘
        yes│          │no
           ▼          ▼
     login to    (skip login)
     Docker Hub       │
           │          │
           └────┬─────┘
                ▼
        docker/build-push-action
        ├── cache-from: type=gha      ◄── restore layers from
        ├── cache-to:   type=gha,max      GitHub's cache backend
        ├── build-args: BUILD_DATE, VCS_REF
        └── push: <probe result>
                │
                ▼
        write $GITHUB_STEP_SUMMARY

Two properties matter here.

The probe result is a step output, not a condition that gets re-evaluated. It runs once, writes push=true|false to $GITHUB_OUTPUT, and every downstream step reads that one value. Which means the login step, the push flag, and the summary can't disagree with each other. That's a class of bug I've hit elsewhere, where the same condition is written in three places and one of them drifts during a refactor.

And build and push are one step rather than two. docker/build-push-action with push: false still does the whole build. That's the property that makes the degradation work without forking the build logic into a separate no-push branch. One code path, one switch.

Step-by-step explanation#

Buildx setup first:

- uses: docker/setup-buildx-action@v3

This is a prerequisite, not an optimisation, and getting that wrong is a nasty little trap. Buildx installs BuildKit, and the GitHub Actions cache backend (type=gha) only exists under BuildKit. Leave this step out and cache-from/cache-to are silently ignored. You get a working build with zero caching and no error telling you why it's slow.

Then the probe:

- name: Check DockerHub credentials
  id: check-dockerhub
  run: |
    if [ -n "${{ secrets.DOKCER_USERNAME }}" ]; then
      echo "push=true" >> $GITHUB_OUTPUT
    else
      echo "push=false" >> $GITHUB_OUTPUT
    fi

Two things to flag, one embarrassing.

The secret name is misspelled. DOKCER_USERNAME. It's consistent across all four places it appears, so it works, but this is a genuinely dangerous typo. Referencing an undefined secret in Actions isn't an error, it interpolates to an empty string. So if someone later "fixes" the spelling in the repository settings without fixing all four references in the YAML, the probe silently returns false and the pipeline stops publishing while still reporting green. A misspelling that fails safe is still a misspelling that's going to bite somebody.

The second thing is the interpolation. ${{ secrets.… }} inside a run: block gets substituted into the script text before the shell ever sees it. That's templating, not variable expansion. For a value I control it's harmless, but as a habit it's exactly how shell injection vulnerabilities get written, and the safe form costs nothing:

- env:
    DOCKER_USERNAME: ${{ secrets.DOCKER_USERNAME }}
  run: |
    if [ -n "$DOCKER_USERNAME" ]; then …

Now the value arrives through the environment as data and the shell never parses it as source. I'll come back to this properly in the Discord article, where the same pattern is actually exploitable rather than just untidy.

Login, conditionally:

- if: steps.check-dockerhub.outputs.push == 'true'
  uses: docker/login-action@v2

Note the string comparison. Step outputs are always strings, so == 'true', never == true. I've written == true and watched a step skip silently more than once.

@v2 is also outdated, current is v3. Look at the version spread across this one file: setup-buildx@v3, login-action@v2, build-push-action@v4. Three actions, three different vintages, all added at different times and never revisited. That's what Dependabot is for and I didn't have it configured.

Now the build step, which is where the bug lives:

- uses: docker/build-push-action@v4
  with:
    context: .
    push: ${{ steps.check-dockerhub.outputs.push }}
    tags: |
      ${{ … format('{0}/${{project_name}}:latest', secrets.DOKCER_USERNAME) … }}
    cache-from: type=gha
    cache-to: type=gha,mode=max
    build-args: |
      BUILD_DATE=${{ github.event.head_commit.timestamp }}
      VCS_REF=${{ github.sha }}

That tags block is broken twice over.

${{project_name}} isn't valid Actions expression syntax. project_name isn't a context or a named value, and the runner rejects it with Unrecognized named-value: 'project_name'. It's meant to be a find-and-replace placeholder, something you swap out before using the template. But it's written in syntax that looks like it evaluates, which is the worst possible choice. A placeholder should look like a placeholder. __PROJECT_NAME__ and nobody's confused.

And it's nested inside format(…), which is already inside a ${{ }}. Expressions don't nest. Everything from the inner ${{ onward gets parsed as part of the outer expression, so it's a syntax error regardless of the named-value problem.

The fix kills both issues and gets rid of the placeholder entirely:

env:
  IMAGE_NAME: ${{ github.event.repository.name }}
…
    tags: |
      ${{ steps.check-dockerhub.outputs.push == 'true'
          && format('{0}/{1}:latest', secrets.DOCKER_USERNAME, env.IMAGE_NAME)
          || format('{0}:local', env.IMAGE_NAME) }}

github.event.repository.name is always right and never needs replacing. A template that derives its values can't be deployed half-configured, which is the actual lesson.

The caching is the performance story and it's the part that works well. cache-from: type=gha with cache-to: type=gha,mode=max stores BuildKit's layer cache in GitHub's cache service, so layers survive across runs on ephemeral runners.

mode=max is the flag that matters. The default is min, which only caches layers present in the final image and throws away intermediate stages. On a multi-stage Node build, where the expensive stage is npm ci running in a builder that gets discarded, min caches almost nothing you care about. mode=max keeps every stage.

On a commit where dependencies haven't moved this is the difference between a multi-minute build and a sub-minute one. The trade-off is cache size against GitHub's 10GB per-repository budget, and eviction is LRU, so a busy repo running several caching workflows can start thrashing. If that happens, scope on the cache config lets you partition per workflow.

The build args:

build-args:
  BUILD_DATE=${{ github.event.head_commit.timestamp }}
  VCS_REF=${{ github.sha }}

These follow the OCI annotation convention and make the image self-describing, so docker inspect on a mystery container tells you which commit built it and when. Worth knowing that github.event.head_commit only exists on push events, so on a workflow_dispatch run BUILD_DATE comes out empty. github.event.repository.updated_at or just generating a timestamp is the robust version.

The summary step writes markdown to $GITHUB_STEP_SUMMARY, which renders on the run page. I think this is badly underused. The difference between reading a rendered table of image tags and expanding six collapsed log groups to hunt for them is real, especially when you're triaging something.

One flaw though. The summary hardcodes **Status**: ✅ Build completed successfully. The step has no if: guard and no failure branch, so it only ever runs on success, which means the line is technically accurate and also completely meaningless. It's stating a constant, not reporting a result. A summary that can only say "success" is decoration.

Interesting implementation details#

Graceful degradation generalises further than I expected. The rule is roughly: when a pipeline can't do the privileged thing, it should still do the useful thing, and be explicit about which one happened. Applies to signing, deploying, publishing coverage, anywhere a fork or a permission boundary can withhold credentials. The alternative, failing hard on missing secrets, trains people to ignore red builds, and a build status people ignore is worse than no build status at all.

The dual tagging is the rollback story. latest for convenience, :${{ github.sha }} for immutability. latest is a moving pointer and should never be the thing production pins to, but as a human-facing convenience sitting next to an immutable tag it earns its place.

Common mistakes#

Skipping setup-buildx-action and then wondering why the caching isn't doing anything is the one I'd bet money on people hitting. No BuildKit, no type=gha, no error message.

Leaving cache-to at the default min on a multi-stage build caches the cheap layers and rebuilds the expensive ones on every run, which is close to the worst possible outcome since you're paying the cache write cost for nothing.

Deploying latest in production. It's a mutable pointer, two deploys "of the same image" can be different bytes, and immutable digests or SHA tags are the only defensible thing to pin to.

Comparing step outputs to booleans. They're strings. == true is always false.

Assuming a misspelled secret errors. It doesn't, it's empty.

Interpolating secrets into shell scripts, which works fine until the day it's user-controlled data instead of a secret and then it's a vulnerability.

Writing placeholders in syntax that looks executable.

And hardcoding "success" in a summary, which is reporting your intent rather than the outcome.

Lessons learned#

Templates that aren't executed where they live will rot. The tagging bug survived because this workflow sits in a reference repo with no Dockerfile. It gets copied out and modified before it ever runs, so nothing validates it in place. If I were rebuilding this library from scratch the single most valuable thing I'd add is a CI job running actionlint over every workflow, which catches invalid named-values and nested expressions statically. That one addition would have caught the bug the day I wrote it.

A placeholder's syntax is part of its contract. Make unreplaced placeholders look obviously unreplaced, or better, derive the value so there's nothing to replace and the failure mode doesn't exist.

Designing for the least-privileged caller improved the pipeline for everyone. That surprised me a bit. The fork PR case felt like an edge case I was accommodating, and it ended up producing a cleaner design than what I had.

And version drift is silent debt. Three actions, three vintages, all working. Nothing forces the update, so nothing updates. That's precisely the decay Dependabot exists to stop, and I didn't have it on.

Production considerations#

Registry credentials are a supply chain asset and I don't think that's widely internalised. A leaked Docker Hub push token means someone can publish a malicious latest that your infrastructure pulls automatically. Use an access token scoped to a single repository, never a password, and rotate it. Where the registry supports OIDC federation, that removes the stored credential entirely.

There's no vulnerability scanning. The image ships without anything ever looking at it. A Trivy or Grype step that fails on HIGH or CRITICAL is a few lines and it's the most obvious missing control here.

No signing, no attestation. Nothing proves this image came out of this pipeline. build-push-action v6 will generate SLSA provenance and an SBOM with two flags, and cosign signing is a short extra step.

Single architecture, linux/amd64 only. Deploy to Graviton or hand it to someone on an ARM laptop and it won't start. platforms: linux/amd64,linux/arm64 fixes it, at roughly double the build time when the cache is cold.

No concurrency control, which means two quick pushes to live race and whichever finishes last wins latest. That may not be the newer commit. A concurrency group with cancel-in-progress: true sorts it.

And cache poisoning across branches is a documented attack surface worth reading about. GitHub scopes the cache per branch with fallback to the default branch, which mostly contains it, but layer cache written by a build on a compromised branch is a real vector.

Improvements#

Fix the tags block and drop the placeholder, using github.event.repository.name. That's the change that makes the template actually correct rather than nearly correct.

Add actionlint to CI, because it catches exactly the bug class above:

- run: |
    bash <(curl -s https://raw.githubusercontent.com/rhysd/actionlint/main/scripts/download-actionlint.bash)
    ./actionlint -color

Upgrade and SHA-pin everything, with Dependabot managing the bumps.

Add concurrency:

concurrency:
  group: docker-${{ github.ref }}
  cancel-in-progress: true

Replace the hand-rolled tag logic with docker/metadata-action, which generates semver tags, branch tags, SHA tags and OCI labels from the event context and is far better tested than anything I'd write:

- id: meta
  uses: docker/metadata-action@v5
  with:
    images: ${{ secrets.DOCKER_USERNAME }}/${{ github.event.repository.name }}
    tags: |
      type=sha,format=long
      type=raw,value=latest,enable={{is_default_branch}}

Scan before publishing, with Trivy and exit-code: 1 on HIGH/CRITICAL, sitting between build and push so a vulnerable image never reaches the registry at all.

Generate SBOM and provenance with sbom: true and provenance: mode=max on v6.

Multi-arch via QEMU plus platforms.

And make the summary honest. Derive the status from the job state instead of asserting it, and add an if: failure() branch so a failed build reports as a failed build.

Next: push-ecs.yml, which contains the one line I'd defend hardest and a deploy mechanism that only works by accident.