GitHub Actions in Production, Part 5: The Heredoc That Expands on the Wrong Machine

Not everything runs in a container. There are profitable, load-bearing Node services out there running on a single EC2 instance under pm2 behind Nginx, and they'll keep running that way for years, because migrating them to ECS costs more than it returns.

This workflow deploys to those. It's the least fashionable pipeline in the library and, measured by actual traffic served, probably the most important one.

It also has the most interesting bug, which is a shell heredoc that expands variables on the runner instead of the server. It works. It has always worked. It's one added line away from not working, and the way it breaks is unpleasant.

Introduction#

The starting point was a procedure in a document. Someone with the PEM file SSHes in, cds to the project directory, pulls, installs, builds, restarts pm2, saves. Six commands in order on the correct host.

Everything below is what happened when I tried to turn that into a workflow, and what I'd do differently now.

Why this workflow exists#

The failure modes of a documented manual procedure are boringly predictable once you've watched them a few times.

Steps get skipped. pm2 save is the classic, because nothing breaks when you forget it. Everything works fine right up until the instance reboots and pm2 comes back running a process list from three deploys ago.

The wrong verb gets used. pm2 restart instead of pm2 reload, which drops every in-flight request. Nobody notices at 2pm on a Tuesday with light traffic.

The wrong host gets deployed to. Two servers, two similar hostnames, one terminal, one person who's been at it since morning.

And access becomes the bottleneck. Only people holding a production PEM can deploy, so every deploy queues behind one person's availability, which is fine until that person is on a plane.

Encoding it fixed all four. Order is fixed, host is derived from the branch, the correct pm2 verb is baked in, and the private key lives in GitHub's secret store instead of on laptops.

There's one constraint that shaped everything else here, and it's worth stating plainly: the server is the source of truth for the running code. No registry, no artifact store, no image. Deploying means making the server's working tree match the branch and then reloading the process. That constraint drives the good decisions below and all of the bad ones.

Architecture#

   push: live
       │
       ▼
  ubuntu-latest runner
       │
       │  ── no checkout! ──────────────────┐
       │                                     │ the runner never needs
       ▼                                     │ the source; the server
  write SSH key → ~/.ssh/deploy_key          │ pulls it directly
  chmod 600                                  │
  ssh-keyscan host → known_hosts             │
       │                                     │
       ▼                                     │
  ssh root@host << ENDSSH  ──────────────────┘
       │
       │  ┌─────────────── on the EC2 box ───────────────┐
       │  │  set -e                                       │
       │  │  cd /root/properties                          │
       │  │  git pull origin live                         │
       │  │  npm install                                  │
       │  │  [ -f tsconfig.json ] && npm run build        │
       │  │  pm2 reload ecosystem.config.cjs --only app   │  ◄── zero downtime
       │  │  pm2 save                                     │
       │  └───────────────────────────────────────────────┘
       ▼
  rm -f deploy_key   (if: always())

Two things I'd keep.

There's no actions/checkout. The runner is a control plane here, not a build host. It issues an SSH command and waits while the server fetches its own source from git. Skipping checkout saves time, and more usefully it expresses what the runner's job actually is. A lot of SSH deploy workflows check out code they then never touch, which always reads to me like someone pasted a template without asking what each step was for.

And the key never persists. Written to the ephemeral runner's disk, used, deleted under if: always() so a failed deploy still cleans up. Runners get destroyed after the job anyway so this is belt-and-braces, but it's the right instinct and if: always() on cleanup steps is a habit worth having everywhere.

Where it's weak: there's no artifact. The deployed state is whatever git pull plus npm install happened to produce on that machine at that moment. Two servers deploying the identical commit can end up with different node_modules if some transitive dependency published in between. There's no rollback target, no way to answer "what exactly is running," and no atomicity, so if npm install dies halfway you have a live server with a half-updated dependency tree serving traffic.

Step-by-step explanation#

Key setup, which contains a small contradiction:

- run: |
    mkdir -p ~/.ssh
    if [ "${{ github.ref_name }}" = "live" ]; then
      echo "${{ secrets.EC2_SSH_KEY_LIVE }}" > ~/.ssh/deploy_key
      ssh-keyscan -H "${{ secrets.EC2_IP_LIVE_HOST }}" >> ~/.ssh/known_hosts
    else
      …PBE variants…
    fi
    chmod 600 ~/.ssh/deploy_key

chmod 600 is required rather than tidy. OpenSSH flatly refuses to use a key file that's group or world readable.

The ssh-keyscan is the interesting part. It fetches the host's public key into known_hosts, which looks like host verification. But the deploy step then passes -o StrictHostKeyChecking=no, which tells SSH not to verify anything. The two cancel out.

I want to be precise about what's lost, because it's less than it first appears. ssh-keyscan on a fresh runner is trust-on-first-use against a host you've never seen before, so it isn't real verification either. Someone sitting in the middle at scan time poisons the file and you're none the wiser. The genuinely secure version pins the host key as a secret:

- run: echo "${{ secrets.EC2_KNOWN_HOSTS }}" >> ~/.ssh/known_hosts

and drops StrictHostKeyChecking=no entirely, so you're comparing against a fingerprint you established out of band. In practice what's here is about as safe as most SSH deploy pipelines. It just shouldn't look like it's doing verification when it isn't, because that's the kind of thing someone reads quickly and then stops worrying about.

There's also dead code in the branch selector. The trigger is branches: ["live"], so the entire else branch, the whole PBE staging path with its own key and host, is unreachable. The README claims this deploys on push to live or dev, which means the trigger got narrowed at some point and the body didn't. Unreachable conditionals in deploy scripts are worse than dead code elsewhere, because they read as tested paths and nobody checks. Either add dev to the trigger or delete the branch.

Now the heredoc:

ssh -i ~/.ssh/deploy_key -o StrictHostKeyChecking=no $HOST << ENDSSH
  set -e
  cd $PROJECT_DIR
  git pull origin ${{ github.ref_name }}
  …
ENDSSH

The delimiter is unquoted. << ENDSSH, not << 'ENDSSH'. That means the runner's shell expands every $… in the body before a single byte travels over the network.

It happens to work. $PROJECT_DIR is set on the runner, so it expands to the right path and the server receives a literal cd /root/properties. Fine.

But it's a trap with the safety off, and the trap springs on whoever edits this file next. Add any line referencing a server-side variable, $HOME, $PATH, $NODE_ENV, $(date) for a log line, $(git rev-parse HEAD) to record what got deployed, and it gets evaluated on the runner instead. $NODE_ENV comes out empty. $(date) gives you the runner's clock. And $(...) is arbitrary command execution on the runner, which is currently holding your production SSH key in a file it can read.

The robust form quotes the delimiter and passes values deliberately:

ssh -i ~/.ssh/deploy_key "$HOST" \
    "PROJECT_DIR='$PROJECT_DIR' REF='${{ github.ref_name }}' bash -s" << 'ENDSSH'
  set -euo pipefail
  cd "$PROJECT_DIR"
  git fetch --prune origin "$REF"
  git reset --hard "origin/$REF"
  …
ENDSSH

Now the boundary is explicit. Quoted delimiter means nothing expands locally, and the values you do want get passed across as environment variables on purpose. This is the change I'd make first out of everything in this article.

One thing that is wired correctly: set -e works. The remote shell reads the script from stdin, aborts on first failure, ssh propagates the exit code, the step fails. Fail-fast is fine.

git pull is not fine. It's fetch plus merge, and merges conflict. If anyone has ever edited a file directly on the server, and on a box people SSH into somebody has, the pull either fails or produces a merge commit on a production server. git fetch followed by git reset --hard origin/<branch> is deterministic: the server's tree becomes exactly the remote's, no negotiation.

Then npm install, which has two problems stacked on each other.

install rather than ci. npm ci installs exactly what's in the lockfile, wipes node_modules first, and fails if they've drifted. install mutates the lockfile and resolves fresh versions. On a production deploy you want the deterministic one, obviously, and I don't have a good reason for why this says install other than that's what I type locally.

And it runs on the server that's currently serving traffic. Dependency installation is CPU and IO heavy. On a small instance it competes directly with the running application, so deploys make the service slow, which is exactly when you don't want deploys to feel risky. The image-based workflows in this same library don't have this problem at all, because build and run happen on different machines.

The conditional build is a bit I still like:

if [ -f tsconfig.json ]; then npm run build; fi

One template serving both JS and TS services without forking. Checking for tsconfig.json is a slightly indirect proxy and checking whether a build script exists in package.json would be more direct, but the intent holds up.

Then the good part:

pm2 reload ecosystem.config.cjs --only ${{project_name}}

reload does a rolling restart across cluster-mode workers. It starts a replacement, waits for it to come up, routes traffic to it, kills the old one, then moves to the next. The listening socket is never closed, so nothing gets refused. pm2 restart kills everything and starts fresh, dropping whatever was in flight.

That distinction is the entire zero-downtime story on this deployment model, and it's one word.

Two caveats people miss. Reload only gives you true zero downtime in cluster mode with more than one instance; a single fork-mode process still has a gap while it comes back. And your app has to actually handle SIGINT/SIGTERM by draining connections, otherwise pm2 hard-kills it after kill_timeout and you drop requests anyway. The pipeline can't enforce that. The application has to cooperate, and mine didn't for the first few months.

${{project_name}} is the same invalid placeholder I wrote about in the Docker Hub article. Not a valid Actions named-value, fails expression evaluation. Deriving it from github.event.repository.name or reading the app name out of the ecosystem file removes the manual substitution entirely.

pm2 save persists the process list so pm2 resurrect can restore it after a reboot. Forgetting it is how a server comes back from an instance restart running nothing at all, quietly, at whatever hour AWS decided to retire the underlying hardware.

Last thing: the deploy runs as root. HOST="root@…". Which means the deploy has unrestricted control of the machine. A dedicated deploy user owning the project directory, with a narrow sudoers entry if a service restart genuinely needs one, shrinks the blast radius of a leaked key from "the whole box" to "one app directory."

Interesting implementation details#

The runner-as-control-plane thing is the design I'd carry forward. No checkout, no build, no artifact, just an authenticated instruction and a wait. It makes the workflow fast regardless of repository size, and it draws a clean line around what each machine is responsible for.

The branch-scoped secret naming, EC2_SSH_KEY_LIVE next to EC2_SSH_KEY_PBE, lets one file serve multiple environments while only ever materialising one environment's credentials per run. GitHub Environments do this properly, with approval gates and an audit trail. As a zero-infrastructure approximation it's sound.

Common mistakes#

Unquoted heredoc delimiters, where the expansion silently happens on the wrong machine and the failure mode escalates from "wrong value in a log line" to "code execution on the CI runner holding your production key."

pm2 restart where reload belongs.

Forgetting pm2 save, which works until it very much doesn't.

Running ssh-keyscan and then StrictHostKeyChecking=no, which is theatre. Pick one.

git pull on a server people have SSHed into.

npm install where npm ci belongs.

Deploying as root because it was easiest on day one.

Building on the production host, so your deploys degrade the service they're deploying.

No concurrency group, and this is the one workflow where a race actually corrupts state rather than just confusing you. Two overlapping deploys running git pull and npm install in the same directory at the same time is genuinely destructive.

And no timeout, so a hung SSH connection sits there for six hours.

Lessons learned#

Know which machine your shell is running on. That's the sharpest lesson in this whole series for me. In a workflow spanning a runner and a server, every single line carries an implicit "where does this evaluate," and the syntax answering that question is one quote character that's easy to leave off and impossible to notice afterward. Quote the delimiter by default. Unquote only where you mean to inject something.

Zero downtime turned out to be one word, decided once, and invisible until a deploy lands during real traffic.

Convenience defaults compound. root because it was easiest. StrictHostKeyChecking=no because a fingerprint prompt blocked a run once and I was in a hurry. install because that's what I type. Each one individually defensible in the moment, and collectively a deploy path with full machine access, no host verification, and non-deterministic dependencies.

And your deploy model sets a ceiling on your reliability. Pull-and-build-on-server can't give you atomic deploys, instant rollback, or artifact immutability. Not because I implemented it badly, but because there's no artifact to roll back to. Recognising a ceiling is more useful than polishing underneath it, and it took me a while to stop polishing.

Production considerations#

Rollback is manual and slow. Recovery means SSHing in, git reset --hard <previous-sha>, reinstalling, reloading, all while the service is degraded and someone is asking for updates. The classic fix on this model is timestamped release directories with a current symlink and an atomic swap, which makes rollback a symlink change plus a reload. That's the Capistrano pattern and it's still the right answer for VM deploys twenty years later.

One instance, one deploy target, no load balancer. The reload is the only availability mechanism there is.

Migrations are unaddressed, same as ECS, but worse here because reload means old and new code genuinely overlap with no orchestrator managing the transition.

Secret rotation is a real gap. A long-lived root SSH key in GitHub secrets has no expiry. On AWS specifically there's a strictly better answer: Systems Manager Session Manager removes the key entirely. The runner authenticates with IAM, ideally via OIDC, SSM brokers the session, and every session is logged in CloudTrail. No persistent credential exists.

And there's no health check after the reload. pm2 reports that the process started. It doesn't report that the app is answering requests. A curl -fsS localhost:PORT/health with a retry loop would catch a process that boots successfully and then immediately fails on a bad config value, which is a thing that has happened to me.

Improvements#

Quote the heredoc delimiter and pass variables explicitly. Highest value, smallest diff.

Add concurrency, without cancellation:

concurrency:
  group: ec2-deploy-${{ github.ref }}
  cancel-in-progress: false

Never cancel a running deploy on this model. Queue it.

Swap git pull for git fetch plus git reset --hard. Swap npm install for npm ci.

Add a post-reload health check with retries and fail the job if it doesn't pass.

Deploy as a non-root user. Pin the host key as a secret and drop StrictHostKeyChecking=no. Add timeout-minutes: 15.

Delete the dead staging branch, or enable it properly with GitHub Environments rather than suffixed secret names.

Move to release directories with an atomic symlink swap so rollback is instant and deploys stop mutating the live tree in place.

And the bigger one, longer term: build the artifact in CI. Build on the runner, ship a tarball or a container, let the server only unpack and reload. That takes install-time load off production, makes deploys deterministic, and finally gives you something to roll back to. It closes most of the gap with the container workflows without having to leave the VM model behind, which for these services is the right trade.

Next: push-s3.yml, and the uncomfortable question of what a backup is actually protecting you from.