GitHub Actions in Production, Part 4: The Line That Makes a Green Check Mean Something

There's exactly one line in this workflow I'd defend to the death:

aws ecs wait services-stable --cluster "$ECS_CLUSTER" --services "$ECS_SERVICE"

Everything before it is plumbing. Checkout, build, push, an API call. That line is the difference between a pipeline that starts a deployment and one that confirms a deployment, and it's the reason a green check here means containers are running and healthy rather than meaning AWS accepted an HTTP request.

There's also a design flaw in how the image actually reaches those containers that I didn't spot for a long time, and which I've since seen in nearly every ECS pipeline I've looked at. Walking through it is most of this article.

Introduction#

These services run on ECS Fargate behind a load balancer. This is the most operationally serious workflow in the library, in the sense that it's the one that can take a production service down, and it's the one I've rewritten the most times.

Why this workflow exists#

Before the pipeline existed, deploying meant: build locally, push to ECR, open the AWS console, find the service, click Update service, tick Force new deployment, click through three screens, then sit on the Tasks tab hitting refresh to find out whether the new tasks stabilised or crash-looped.

Three problems with that, and the tedium is the least of them.

It isn't reproducible. Which image did you push? Tagged how? Built from which commit? From a working tree that may or may not have had uncommitted changes in it? An hour later nobody knows, including you.

It has no failure signal, and this is the one that actually hurt. The console shows a deployment in progress. If the new task definition crashes on startup, and it usually crashes for boring reasons like a missing env var or a migration that hasn't run, ECS just retries it. The old tasks keep serving. The deployment sits in IN_PROGRESS more or less forever. Unless somebody is watching that tab, the outcome is "we think we deployed and we didn't," which is strictly worse than a clean failure because now the whole team believes the change is live and starts reasoning from that.

And it doesn't work with more than one person. Console deploys can't be reviewed, can't be audited, and can't happen while you're asleep.

The second requirement was cheap Dockerfile validation on a non-production branch. Finding out your Dockerfile is broken during a production deploy is entirely avoidable. Pushing to dev builds the image and stops, with no AWS credentials anywhere in the execution path.

Architecture#

                    push
                     │
         ┌───────────┴────────────┐
         ▼                        ▼
    branch: dev              branch: live
         │                        │
    docker build             configure-aws-credentials
    (no creds,                    │
     no registry)            amazon-ecr-login
         │                        │
    build summary            docker build + tag (sha, latest)
         │                        │
         ✓                   docker push × 2
                                  │
                             ecs update-service --force-new-deployment
                                  │
                             ecs wait services-stable  ◄── blocks here
                                  │                        until steady state
                             ┌────┴────┐                   or timeout
                             ▼         ▼
                          stable    timeout → job fails

One job handling two behaviours via step-level if: guards, rather than two files or two jobs.

The argument for that: dev and live share the checkout, the Dockerfile, and the build semantics. Split them into separate files and every fix to the shared part has to happen twice, and the second one gets forgotten. Drift between "the thing that validates" and "the thing that deploys" is how you end up with a build that's green on dev and dies on live.

The argument against is also real. A nine-step job where six steps carry if: github.ref_name == 'live' is harder to read than two focused files, and the run log on a dev push is mostly skipped steps, which looks broken to anyone who doesn't already know the structure. There's a cleaner middle ground with a shared reusable workflow and two thin callers, which I'll get to.

The environment separation here is stronger than it looks, and it's worth calling out. On a dev push, configure-aws-credentials never runs, so AWS credentials are never materialised in the runner environment at all. That's not a policy or a convention. It's a structural property of the workflow, and structural guarantees survive people editing things in ways that policies don't.

Step-by-step explanation#

Starting with the trigger, which contains a filter that does nothing:

on:
  push:
    branches: [live, dev]
    paths:
      - '**'
      - 'Dockerfile'

'**' matches every file in the repo, so adding 'Dockerfile' is redundant and the filter as a whole is equivalent to having no filter. Harmless, but it reads like there's path-based optimisation happening, and there isn't. If the intent was "only run when things affecting the image change" it'd need to be an actual list: src/**, package.json, pnpm-lock.yaml, Dockerfile. That's meaningful on a monorepo and pure overhead on a single-service repo. I'd just delete it rather than half-implement it.

Environment config:

env:
  AWS_REGION: us-east-1
  ECR_REPOSITORY: egystay
  ECS_CLUSTER: ${{ secrets.AWS_ECS_CLUSTER }}
  ECS_SERVICE: ${{ secrets.AWS_ECS_SERVICE }}

Cluster and service names are stored as secrets, and they aren't secrets. They're configuration. They show up in CloudTrail and they're sitting in your Terraform anyway. GitHub repository variables (vars.*) are the right home: same injection mechanism, but visible in the UI and in logs, which makes debugging a failed deploy dramatically less painful than staring at a wall of ***. Reserve secrets for things that grant access.

Also AWS_REGION: us-east-1 is declared and never used, because the credentials step reads secrets.AWS_IAM_REGION instead. Two sources of truth for one value, one of them dead. Small thing, but it's the kind of small thing that sends someone down a wrong path at 3am.

Dockerfile validation on dev:

- name: Check Docker build
  if: github.ref_name == 'dev'
  run: docker build -t $ECR_REPOSITORY:${{ github.sha }} .

Simple and it works. No credentials, no registry, no BuildKit cache, which means a cold build every single time. On a slow Dockerfile that's minutes of runner time per push, and adding setup-buildx-action with cache-from: type=gha here would cost nothing. The Docker Hub workflow in this same repo already does that. Same repo, same image, two different caching strategies, because I wrote them months apart and never went back.

Then AWS auth:

- uses: aws-actions/configure-aws-credentials@v4
  with:
    aws-access-key-id: ${{ secrets.AWS_IAM_ACCESS_KEY }}
    aws-secret-access-key: ${{ secrets.AWS_IAM_SECRET_ACCESS_KEY }}

This works, and it's the first thing I'd change. Long-lived IAM access keys are permanent credentials living in GitHub's secret store. They don't expire, they don't rotate themselves, and if they leak through a compromised action or an over-permissive trigger or someone debugging with an env dump, whoever has them can do whatever that IAM user can do, indefinitely, until a human notices and revokes.

GitHub's OIDC provider replaces the whole arrangement. The runner presents a signed identity token, AWS STS swaps it for credentials valid only for that job, and the role's trust policy constrains which repository and which branch is allowed to assume it. There's no stored credential to leak because there's no stored credential.

permissions:
  id-token: write
  contents: read

- uses: aws-actions/configure-aws-credentials@v4
  with:
    role-to-assume: arn:aws:iam::<account>:role/github-actions-ecs-deploy
    aws-region: us-east-1

One-time IAM setup, removes an entire category of standing risk. There's an irony I enjoy here, which is that test-build.yml declares id-token: write and has no use for it, while this workflow needs it and doesn't declare it.

Build, tag, push:

docker build -t $ECR_REGISTRY/$ECR_REPOSITORY:$IMAGE_TAG .
docker tag  … :$IMAGE_TAG  … :latest
docker push … :$IMAGE_TAG
docker push … :latest

Dual tagging again, immutable SHA plus moving latest. Right instinct.

Then the deploy, and this is the part I got wrong:

aws ecs update-service --cluster "$ECS_CLUSTER" --service "$ECS_SERVICE" --force-new-deployment

What that command actually does is tell ECS to start new tasks using the service's currently registered task definition. It does not change the task definition. It does not tell ECS anything about the image you just built.

The only reason this deploys your new code is that the task definition presumably references …:latest, and you just moved latest to point at the new image. So the deploy works as a side effect. Push a tag, then trigger a pull of that tag.

Three things go wrong with that.

It's a race. Two commits merged a couple of minutes apart produce two workflow runs. Run A pushes latest, run B pushes latest, then run A's update-service fires and pulls B's image. Your pipeline reports that commit A deployed successfully. It didn't. Nothing anywhere records that this happened.

The deployed version is unknowable from AWS. The task definition says :latest and nothing in ECS records which digest that resolved to at pull time. So "what's running in production right now" has no reliable answer, and that question always gets asked during an incident.

And rollback isn't possible through ECS. There's no previous task definition revision to go back to, because the revision never changed. Rolling back means re-pushing an old image to latest and forcing another deployment, which is mutating a tag to undo a deploy. That's the opposite of what you want while production is down.

The correct pattern registers a new task definition revision pinned to the immutable SHA tag, then points the service at that revision. AWS ships actions for exactly this:

- id: task-def
  uses: aws-actions/amazon-ecs-render-task-definition@v1
  with:
    task-definition: .aws/task-definition.json
    container-name: api
    image: ${{ steps.build-image.outputs.image }}   # the :sha tag

- uses: aws-actions/amazon-ecs-deploy-task-definition@v1
  with:
    task-definition: ${{ steps.task-def.outputs.task-definition }}
    service: ${{ vars.ECS_SERVICE }}
    cluster: ${{ vars.ECS_CLUSTER }}
    wait-for-service-stability: true

Now every deploy creates an auditable revision bound to exactly one commit, and rollback is update-service --task-definition my-app:41. Which you can run from your phone.

Then the line that earns the whole workflow:

aws ecs wait services-stable --cluster "$ECS_CLUSTER" --services "$ECS_SERVICE"

This polls until the service has one deployment in PRIMARY, running count matches desired count, and tasks are passing health checks. If the new tasks crash-loop, it never reaches stable, the command eventually times out non-zero, and the job goes red.

Without it, update-service returns immediately with a 200 and the job goes green while your new tasks are failing health checks in the background. The pipeline would be reporting the success of an API call as the success of a deployment. That's the failure mode that makes teams stop trusting CI/CD entirely, and once that trust is gone you don't get it back cheaply.

Worth knowing the default waiter polls every 15 seconds up to 40 times, so about ten minutes. Services with long draining periods or slow health checks can blow through that and fail a deploy that would have succeeded, so it's worth tuning against your actual rollout time rather than accepting the default.

Last, the summary step, which lies.

It has no if: guard, so it runs on dev too, where it prints ✅ Image pushed to ECR, an empty Registry: field because ECR login never ran, and a SHA tag that exists only on the runner's local Docker daemon. Every single dev push produces a run page stating something untrue.

It also contradicts itself. After wait services-stable has already completed the deployment, the summary says "Next Steps: 1. Update ECS task definition 2. Deploy new task/service." That's leftover text from an earlier version where the workflow stopped at ECR and I never cleaned it up. A summary claiming the deploy hasn't happened, printed after the deploy happened, is worse than having no summary.

Interesting implementation details#

The structural environment isolation is the thing I'm happiest with. The dev path can't reach AWS because the credentials step is guarded, not because a document says it shouldn't. Guarantees enforced by structure survive contact with people editing things.

And the fact that most of the value of this workflow lives in a command that produces no output and does nothing except refuse to return early still strikes me as funny.

Small practical note: the aws CLI is preinstalled on GitHub-hosted runners, which is why there's no install step. Convenient, and a hidden dependency on the runner image that will surprise you the day you move to self-hosted.

Common mistakes#

Using --force-new-deployment as your deploy mechanism. It redeploys the existing task definition, and if that definition points at a mutable tag then your deploys are racy and your rollbacks are impossible. This is the big one and it's everywhere.

Skipping the stability wait, so green means "AWS accepted the request" and nothing else.

Long-lived IAM keys where OIDC is available.

Storing configuration in secrets, which buys you nothing and makes every failed deploy harder to debug.

paths: ['**'], a filter that filters nothing.

Unguarded summary steps, which will eventually report the wrong thing on whichever branch you weren't thinking about when you wrote them.

No concurrency group, so two live pushes produce overlapping update-service calls against the same service.

And no rollback path at all. wait services-stable detects the failure and then nothing acts on it. ECS's deployment circuit breaker will, if you turn it on.

Lessons learned#

A deploy isn't done when the API accepts it. That reframed how I look at every deployment pipeline I encounter now, and the question I ask first is: what does green actually mean here? If the answer is "we sent a request," the pipeline is a notification system wearing a deployment costume.

Mutable tags in production configuration are a latent incident waiting for enough traffic. latest is fine as a human convenience and disqualifying as a deployment reference. The moment two deploys can overlap, "which image is running" stops having an answer, and that's precisely the question you need answered when you can least afford to go looking.

Summaries have to be derived, never asserted. Both this workflow and the Docker Hub one hardcode success strings, which is the same bug twice: reporting intent instead of outcome. Any status line that can't render "failed" isn't reporting anything.

And consistency across a library is itself a feature. Two workflows in one repo building the same image with different caching, different tagging, different auth. Each was reasonable on the day it was written. Together they're inconsistent, and inconsistency is where the bugs live, because it's where your assumptions stop transferring.

Production considerations#

The deployment circuit breaker is the single most valuable setting here and it isn't in the pipeline at all, it's service configuration:

"deploymentConfiguration": {
  "deploymentCircuitBreaker": { "enable": true, "rollback": true }
}

That turns a detected failure into an automatic recovery. Everything the pipeline does to detect failure is worth more once something acts on it.

Database migrations aren't addressed anywhere and they're the thing most likely to hurt. Rolling deploys mean old and new code run simultaneously for the duration of the rollout. Any schema change has to be backward compatible for that whole window or you get errors from whichever version loses the race. Expand/contract is the standard answer, and it's a discipline the pipeline fundamentally can't enforce for you.

IAM scope matters here more than anywhere else in the library. The deploy role needs ecr:* on one repository and ecs:UpdateService plus DescribeServices on one service. Not PowerUserAccess. Deploy credentials are the highest-value target in the entire system.

No timeout, so a hung waiter can sit for the six-hour default.

And there's an observability gap I want to be honest about: the pipeline knows the deploy stabilised. It has no idea whether error rates spiked afterward. Stability is a much weaker signal than health, and this workflow only verifies the former.

Improvements#

Pin deploys to immutable task definition revisions using render-task-definition and deploy-task-definition. That single change fixes the race, the auditability, and the rollback story together.

Migrate to OIDC and delete the standing credential.

Turn on the ECS circuit breaker with rollback.

Add concurrency, deliberately without cancellation:

concurrency:
  group: ecs-deploy-${{ github.ref }}
  cancel-in-progress: false

false matters here. Cancelling a deploy halfway through a rollout is worse than queueing it behind the current one.

Guard the summary, derive its content, delete the stale next-steps text, add an if: failure() branch.

Use GitHub Environments. An environment: production on the job gets you required reviewers, deployment history, environment-scoped secrets, and the deployment timeline in the UI. That's the piece of real environment management that's currently missing, and right now it's being approximated with branch names and suffixed secret names.

Add layer caching to both build paths so they match the Docker Hub workflow.

Move cluster and service to vars, delete the dead AWS_REGION.

Extract the shared parts into a workflow_call workflow taking environment, cluster, service and ecr-repository inputs, so the dev/live split becomes two five-line callers instead of six if: guards scattered through one job.

And add a post-deploy smoke test. Curl a health endpoint through the load balancer once the service is stable. Stable tasks and a working service are not the same claim, and I'd rather the pipeline made the stronger one.

Next: push-ec2.yml, where the deploy runs over SSH and a heredoc expands variables on the wrong machine.