From One EC2 Box to Kubernetes: How SRVJ Actually Deploys on AWS

SRVJ — the collaborative diagram tool I keep writing about — runs on AWS, but it didn't start on Kubernetes, and it shouldn't have. This post is its deployment story in three acts: a single EC2 box with NGINX and pm2, then Docker Compose, then a Kubernetes cluster — plus the S3 patterns that survived every stage untouched.

I'm writing it this way because most AWS content starts at the end. You get the EKS tutorial with the Terraform modules and the service mesh, and nobody tells you that a $10 EC2 instance with a well-configured NGINX serves real production traffic just fine — or when, exactly, it stops being fine.

Act one: a box, NGINX, and pm2

SRVJ's backend started life the way most Node.js backends do: one EC2 instance, Route 53 pointing at its Elastic IP, NGINX terminating TLS and reverse-proxying to the app, and pm2 keeping the process alive. It's unfashionable and it works.

upstream srvj_backend {
    server srvj-app:60459;
    keepalive 32;
}

limit_req_zone $binary_remote_addr zone=api_limit:10m  rate=30r/s;
limit_req_zone $binary_remote_addr zone=auth_limit:10m rate=5r/m;

server {
    listen 443 ssl http2;
    server_name api.srvj.com;
    # (TLS, security headers, and Host/X-Forwarded-* lines omitted for brevity)

    # ── API routes ─────────────────────────────────
    location /api/ {
        limit_req zone=api_limit burst=50 nodelay;
        proxy_pass http://srvj_backend;
        proxy_http_version 1.1;
        proxy_set_header Connection "";
        proxy_buffering off;
    }

    # ── Auth routes (stricter: 5 req/min) ──────────
    location /api/v1/auth/ {
        limit_req zone=auth_limit burst=3 nodelay;
        proxy_pass http://srvj_backend;
        proxy_http_version 1.1;
        proxy_set_header Connection "";
    }

    # ── SSE notifications (long-lived, unbuffered) ─
    location /api/v1/notifications/stream {
        proxy_pass http://srvj_backend;
        proxy_http_version 1.1;
        proxy_set_header Connection "";
        proxy_buffering off;
        proxy_cache off;
        proxy_request_buffering off;
        chunked_transfer_encoding off;
        proxy_read_timeout 86400s;
        proxy_send_timeout 86400s;
    }

    # ── WebSockets (collab) ────────────────────────
    location /socket.io/ {
        proxy_pass http://srvj_backend;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_read_timeout 86400s;
        proxy_send_timeout 86400s;
    }

    # ── Metrics (internal only) ────────────────────
    location /metrics {
        deny all;
        return 403;
    }
}

Every non-obvious line here came from a real incident, not a template. The auth routes get their own rate-limit zone — five requests a minute, not thirty a second — so credential stuffing dies at the proxy without Node spending a single cycle on it. The empty Connection "" header is the easiest one to miss: it pairs with keepalive 32 in the upstream block, and without it NGINX silently opens and closes a fresh upstream connection per request, throwing the keepalive pool away.

The SSE location is the paranoid one, and every line earns its place: proxy_buffering, proxy_cache, and proxy_request_buffering all off, chunked_transfer_encoding off, and day-long read/send timeouts so NGINX doesn't kill a quiet stream at its 60-second default. With buffering on, NGINX holds your carefully streamed events hostage until its buffer fills, and "real-time" notifications arrive in batches. The WebSocket location needs the opposite treatment — the Upgrade/Connection pair, without which the collab handshake dies at the proxy. And /metrics is a flat deny: Prometheus scrapes from inside the network, the internet gets a 403.

pm2 runs the app in cluster mode — one worker per vCPU behind a shared port:

module.exports = {
  apps: [{
    name: 'api',
    script: './dist/server.js',
    instances: 'max',
    exec_mode: 'cluster',
    max_memory_restart: '512M',
    env_production: { NODE_ENV: 'production' },
  }],
}

Cluster mode has a trap that's especially vicious for SRVJ: workers don't share memory. The SSE connection registry fragments across workers — solvable with Redis Pub/Sub, which the notification pipeline needed anyway. The Yjs collab rooms are the harder case: a room is an in-memory Y.Doc, and two clients on the same diagram must reach the same process. On the single box that meant keeping the collab-bearing app in one process and scaling vertically — an early taste of exactly the problem act three is about.

What finally hurt wasn't performance. It was that the box itself was the deployment artifact. Deploys were SSH-and-pull. The Node version, the system packages, the NGINX config — all hand-applied state that existed nowhere in git. Every month the server drifted a little further from anything I could reproduce, and rollback meant remembering what I'd changed.

Act two: same box, but Docker

The fix for drift is making the artifact immutable. GitHub Actions builds a Docker image on every push to main, tags it with the commit SHA, and pushes it to ECR. The EC2 box pulls and restarts. Same hardware, completely different operational story:

• The image is the whole runtime — Node version, native deps, everything. "Works on my machine" stops being a sentence anyone says.
• Rollback is re-tagging: pull the previous SHA, restart. Under a minute, no archaeology.
• The box degrades into a dumb Docker host. Nothing on it is precious anymore — I could rebuild it from a short user-data script.

In SRVJ's case the compose file is three containers on a private bridge network: the app, Redis with append-only persistence, and NGINX with the act-one config mounted read-only. That's also when the upstream stopped being 127.0.0.1:3000 and became a Docker service name — the srvj-app:60459 you saw in the config above.

This stage is criminally underrated. Docker-on-EC2 has none of Kubernetes' complexity and buys you 80% of its reproducibility. If SRVJ had stayed a single well-understood process with vertical headroom left on the instance, this is where the story would end — and for most backends, it should.

The S3 pattern that never changed: presigned uploads

While the compute story kept evolving, file uploads landed on a pattern in week one that has survived every migration since: clients upload directly to S3 with presigned URLs, and the backend never proxies a user's file bytes.

The naive version — multipart POST to the API, which streams to S3 — makes your API instance a file proxy. SRVJ's uploads are avatars and images dropped onto the canvas; routing those through Express means tying up workers, memory, and bandwidth on traffic that S3 could have absorbed directly. The presigned flow inverts it: the client asks the API for permission, gets a short-lived URL, and does the heavy lifting itself.

import { S3Client, PutObjectCommand } from '@aws-sdk/client-s3'
import { getSignedUrl } from '@aws-sdk/s3-request-presigner'
import crypto from 'crypto'

const s3 = new S3Client({ region: process.env.AWS_REGION })

export async function createUploadUrl(userId: string, contentType: string) {
  if (!ALLOWED_TYPES.has(contentType)) throw new Error('Unsupported type')

  const key = `uploads/${userId}/${crypto.randomUUID()}`
  const url = await getSignedUrl(
    s3,
    new PutObjectCommand({
      Bucket: process.env.S3_BUCKET!,
      Key: key,
      ContentType: contentType,
    }),
    { expiresIn: 300 }, // 5 minutes — permission, not possession
  )
  return { url, key }
}

Three details matter more than the happy path. The key is server-generated — clients never choose where they write, which closes the overwrite-someone-else's-file hole. The content type is validated and baked into the signature, so the URL can't be reused for a different payload. And the URL expires in minutes, because it's a permission slip, not a possession.

Reads go through CloudFront, not S3 directly. The bucket is private; CloudFront gets access through Origin Access Control, and every image URL the API returns is a CDN URL. Users in Cairo hit a nearby edge instead of the bucket's region, S3 GET costs drop to near nothing on hot objects, and the bucket itself has no public surface at all.

The other S3 path: avatars born on the server

Not every object in the bucket arrives through a presigned URL, though. When a new account registers, SRVJ generates the user's default avatar on the backend — a colored tile with their initials, rendered with sharp and pushed straight to S3:

export const avatarProfile = async (firstName: string, lastName: string, id: string) => {
  const backgroundColor = getRandomColor(colorArray) // 12-color brand palette
  const textColor = getTextColor(backgroundColor)
  let initials = buildInitials(firstName, lastName)

  // Arabic initials get spacing — two joined glyphs read as a word, not initials
  if (await detectScript(`${firstName} ${lastName}`) === 'Arabic')
    initials = initials.split('').join(' ')

  const imageBuffer = await sharp({
    create: { width: 450, height: 450, channels: 4, background: backgroundColor },
  })
    .composite([{
      input: Buffer.from(`
        <svg width="450" height="450">
          <text x="50%" y="60%" font-size="150" font-family="Arial"
                text-anchor="middle" fill="${textColor}">${initials}</text>
        </svg>`),
      top: 0, left: 0,
    }])
    .png()
    .toBuffer()

  const d = new Date()
  const key = `cdn/user/${d.getFullYear()}/${d.getMonth() + 1}/${id}.png`

  await s3Client.send(new PutObjectCommand({
    Bucket: process.env.AWS_S3_BUCKET!,
    Key: key,
    Body: imageBuffer,
    ContentType: 'image/png',
  }))

  return `${process.env.CDN_CLOUD_URL}${key}` // CloudFront URL, never raw S3
}

The part I'm fondest of is the text color. Instead of hardcoding white-on-anything, the background's perceived brightness is computed with the classic YIQ weights — red, green, and blue don't contribute equally to how bright a color looks — and the initials flip to dark slate on light tiles:

function getTextColor(backgroundColor: string) {
  const hex = backgroundColor.replace('#', '')
  const red = parseInt(hex.slice(0, 2), 16)
  const green = parseInt(hex.slice(2, 4), 16)
  const blue = parseInt(hex.slice(4, 6), 16)
  const brightness = (red * 299 + green * 587 + blue * 114) / 1000
  return brightness > 160 ? '#1F2937' : '#F8FAFC'
}

A few details that matter beyond the pixels. Names in Egypt are Arabic as often as English, so the script detection walks the name's Unicode code points (the 0x0600–0x06FF block and friends) rather than assuming Latin initials. The object key is partitioned by year and month — cdn/user/2026/7/… — which keeps prefixes browsable and makes lifecycle rules trivial later. And the function returns the CloudFront URL, not the S3 one, so the private-bucket rule from the previous section holds even for objects the server created itself.

It's also the counterpoint to the presigned pattern, and the rule that reconciles them: whoever owns the bytes talks to S3. A user's photo upload is theirs — presigned URL, direct to bucket. A generated avatar is the server's — PutObjectCommand from the process that made it. Both end up behind the same CDN.

Act three: SRVJ's shape and the case for Kubernetes

SRVJ broke the single-box model for a boring, structural reason: it isn't one process. The same image runs twice with different entrypoints — API pods serving REST, SSE streams, and the Yjs collab WebSockets (node dist/src/app.js), and a BullMQ worker draining the notification queue (node dist/src/MessageQueue/index.js). They deploy together but scale apart: a burst of collab sessions needs API capacity, a notification storm needs worker throughput, and on one box all of it shares one blast radius and one scaling knob.

This is the actual Kubernetes threshold, in my experience. Not traffic. Shape. The moment your system is several processes with different scaling profiles and you're hand-writing systemd units or docker-compose overrides to fake orchestration, you're implementing a worse Kubernetes on your own time.

flowchart LR
    U[Client] --> R53[Route 53]
    R53 --> CF[CloudFront]
    CF -->|images, static| S3[(Private S3 bucket)]
    CF -->|api + collab| ING[NGINX Ingress]
    ING --> API[API pods, HPA 2-8]
    API --> PG[(PostgreSQL)]
    API --> M[(MongoDB)]
    API --> RD[(Redis StatefulSet)]
    W[BullMQ worker] --> RD
    W --> PG
    U -.presigned PUT.-> S3

A confession before the YAML: SRVJ's cluster today is self-managed on EC2, not EKS — kustomize-applied manifests, an NGINX ingress controller, local-path storage. Running it myself is exactly how I learned what EKS's control-plane fee actually buys: etcd care, control-plane upgrades, certificate rotation — the parts of Kubernetes you least want to own at 3 AM. The manifests are portable either way; moving to EKS changes who runs the control plane, not what the app looks like.

The API deployment is where the operational lessons from acts one and two turned into configuration:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: srvj-api
  namespace: srvj
spec:
  replicas: 2
  strategy:
    rollingUpdate:
      maxUnavailable: 0   # never dip below capacity mid-deploy
      maxSurge: 1
  template:
    spec:
      terminationGracePeriodSeconds: 30
      securityContext:
        runAsNonRoot: true
      containers:
        - name: srvj-api
          image: srvj-backend:latest
          command: [node, -r, tsconfig-paths/register, dist/src/app.js]
          envFrom:
            - secretRef: { name: srvj-secret }
          ports:
            - containerPort: 60459
          securityContext:
            allowPrivilegeEscalation: false
            capabilities: { drop: [ALL] }
            seccompProfile: { type: RuntimeDefault }
          resources:
            requests: { cpu: 200m, memory: 256Mi }
            limits: { cpu: 1000m, memory: 1Gi }
          readinessProbe:
            httpGet: { path: /api/health, port: 60459 }
            initialDelaySeconds: 10
          livenessProbe:
            httpGet: { path: /api/health, port: 60459 }
            initialDelaySeconds: 20
          lifecycle:
            preStop:
              exec: { command: [/bin/sh, -c, sleep 5] }

Both probes currently point at the same /api/health endpoint, and that deserves an honest note. Readiness and liveness are different questions — "can I serve a request right now" versus "is this process worth keeping alive" — and the dangerous failure mode is a liveness probe that checks dependencies: if MongoDB blips and liveness notices, Kubernetes restart-loops every healthy pod in sympathy with the database. The endpoint is dependency-free today, which keeps that trap shut; splitting it into a ready check that verifies Mongo and Redis and a live check that verifies nothing is the planned refinement.

The preStop sleep plus the 30-second termination grace is the other hard-won pattern. During a rollout the pod is removed from the endpoints list and sent SIGTERM concurrently — without a grace window, in-flight requests and open SSE streams die mid-byte. A few seconds of "keep serving, take nothing new" makes deploys invisible to connected clients. And the security context — runAsNonRoot, all capabilities dropped, no privilege escalation, default seccomp — costs nothing at this stage and is miserable to retrofit later.

The worker is the same image with a different entrypoint and deliberately different rules. Its strategy is Recreate, not RollingUpdate: during an API rollout you want old and new pods overlapping; during a worker rollout, overlap means two workers competing for the same BullMQ jobs mid-deploy. It also gets 60 seconds of termination grace instead of 30, because "finish the job you're holding" takes longer than "finish the HTTP request you're serving". Scaling is asymmetric too — an HPA takes the API from 2 to 8 replicas on 70% CPU or 80% memory, while the worker stays at one replica until job volume, not traffic, says otherwise. That asymmetry is the whole point of splitting them.

And the act-one NGINX config didn't die — it migrated into the ingress controller as annotations and snippets. The global rate limit became limit-rps: 30; a server-snippet reproduces the 5-per-minute auth zone; the SSE location moved wholesale, buffering kills and day-long timeouts intact; /metrics is still a flat 403. Same hard-won lines, new address.

The deploy: one apply, and an honest gap

The whole stack is kustomize-driven — namespace, secrets, the Redis StatefulSet, both Deployments, the HPA, and the ingress are one kubectl apply -k k8s/ away, and the manifest tree in git is the cluster's source of truth:

namespace: srvj

resources:
  - namespace.yaml
  - secret.yaml
  - redis/statefulset.yaml
  - redis/service.yaml
  - api/deployment.yaml
  - api/service.yaml
  - api/hpa.yaml
  - worker/deployment.yaml
  - ingress/configmap.yaml
  - ingress/ingress.yaml

The honest gap: the image is still srvj-backend:latest with imagePullPolicy: IfNotPresent, which means a rollout doesn't reliably pick up a new build — the act-two lesson about immutable SHA-tagged artifacts hasn't been fully carried into the cluster yet. It's the top of the improvement list, because kubectl rollout undo is only a real rollback when tags are immutable. What already works in my favor: maxUnavailable: 0 means a build whose pods never pass readiness stalls the rollout while the old pods keep serving — a bad deploy degrades into a stuck rollout, not an outage.

What Kubernetes actually costs (it's not just the invoice)

I'd be lying if I presented act three as pure upside. Self-managing the cluster means I am the control plane's administrator — upgrades, certificates, backups — which is precisely the ledger EKS's flat fee is weighed against; it doesn't remove the YAML, it removes being the etcd administrator, and the nodes are still just EC2 underneath. The in-cluster Redis is a single-replica StatefulSet on local-path storage, which pins it to one node: fine for a rebuildable queue, unacceptable if it ever grows into primary state. And the collab rooms live in-memory inside the API pods, so two clients editing the same diagram can land on different replicas — session affinity papers over it until the Redis fanout between pods lands (the same unsolved item from the CRDTs post).

So the honest decision matrix, from someone running all three stages in production simultaneously:

• One service, one team, predictable load → Docker on a single EC2 box. This is most backends, and it's where SRVJ lived happily for its first stretch.
• Multiple processes with different scaling profiles, zero-downtime deploys as a requirement → Kubernetes earns its complexity, and a managed control plane (EKS) is the part worth paying for. This is SRVJ today.
• File uploads → presigned S3 URLs behind CloudFront, at every stage, regardless of everything else. The one decision I've never revisited.

The progression matters more than the destination. Every stage solved the specific pain the previous one produced — drift got me to Docker, shape got me to Kubernetes — and each migration was small because the artifact (the image) and the S3 paths were already settled. If there's one takeaway: adopt the boring parts early, and let the orchestration wait until your architecture, not your ambition, asks for it.