Automated Log Archiving in Node.js: Pino, Cron Rotation, and AWS S3

Most teams treat logging as a solved problem until the night they need it. Then they discover the disk filled up three weeks ago, the logs that mattered were rotated into oblivion, or worse — the logs exist but contain a customer's Authorization header in plaintext.

This article walks through a production log archiving pipeline that is deliberately boring: Pino writes structured JSON to a local file, a daily cron job rotates that file, uploads it to S3, and deletes the local copy only after the upload is verified. S3 Lifecycle rules expire objects after 90 days. No log shipping agents, no vendor, no per-GB ingestion bill.

It is not the right architecture for every system, and I will be explicit about where it breaks down. But for a single-VM or small-fleet Node.js service, it gives you searchable, durable, cost-bounded logs with roughly 150 lines of code and one IAM policy.

Architecture at a glance#

The pipeline end to end, with the failure paths and the detail that most implementations get wrong — the file descriptor reopen:

flowchart TD
    A[HTTP Request / Domain Event] --> B[Pino Logger]
    B --> C{redact paths}
    C --> D[SonicBoom destination]
    D --> E[(logs/app.log)]

    F[Cron 00:00 UTC] --> G[Acquire rotation lock]
    G --> H[fs.rename app.log to YYYY-MM-DD.log]
    H --> I[Signal process: destination.reopen]
    I --> J[New empty app.log created]
    J --> K[gzip archive]
    K --> L[PutObject to S3 with checksum]
    L --> M{HTTP 200 and checksum verified?}
    M -- yes --> N[unlink local archive]
    M -- no --> O[Keep file, alert, retry tomorrow]
    N --> P[(S3 bucket: logs/YYYY/MM/DD/)]
    P --> Q[S3 Lifecycle: Expiration 90 days]
    Q --> R[Object deleted by AWS, no request cost]

Two things in that diagram deserve early attention, because they are the difference between a working system and a silent data-loss bug:

• The reopen step — on Linux, renaming a file the process has open does not detach the process from it. Without an explicit reopen, your application keeps writing into the archived file.
• The verified delete — local deletion is conditional on a confirmed upload, never on the upload call returning without throwing.

Why application logging matters#

Metrics tell you that something is wrong. Traces tell you where. Logs tell you what actually happened — the specific user, the specific payload shape, the specific branch of the specific conditional.

In practice, logs earn their keep in four situations:

• Incident forensics — a payment webhook was processed twice. Was it a duplicate delivery from the provider, or did your idempotency key generation collide? Only the log line carrying the provider's event ID and your computed key answers that.
• Non-reproducible bugs — the class of bug that only occurs for one merchant, on one locale, with one malformed field. You cannot reproduce it locally; you can read what happened.
• Audit and dispute resolution — "the customer says they never cancelled." A timestamped, immutable record of the state transition ends the conversation.
• Behavioural archaeology — understanding how a feature is actually used before you refactor it.

The common failure is not "we do not log." It is "we log, but the logs are unqueryable, unretained, or unsafe."

Why JSON logs beat plain text#

A plain-text line like [2026-07-26 11:04:22] user 8123 failed login from 41.x.x.x is human-readable and machine-hostile. To answer "how many failed logins per IP in the last hour," you write a regex. When someone adds a field, the regex breaks. Structured JSON gives you:

• Queryability without parsing — jq 'select(.level >= 50 and .route == "/checkout")' works today; the same file loads into Athena, OpenSearch, or DuckDB tomorrow with no ETL.
• Type preservation — durationMs: 412 stays a number. In text logs everything is a string until you regex it back.
• Stable contracts — adding tenantId to every line breaks nothing downstream. Adding a column to a text format breaks every consumer.
• Injection safety — a user submitting a username containing \n level=fatal cannot forge a log line, because JSON encoding escapes the newline. Text formats are genuinely vulnerable to log forging.
• Correlation — carrying requestId / traceId on every line lets you reconstruct a full request across dozens of emissions.

The tradeoff: JSON is verbose and unpleasant to read raw. That is solved at read time with pino-pretty in development, not by degrading the production format. Never let developer ergonomics dictate your production log format.

Why Pino was chosen#

Pino is a JSON-first logger built around a simple principle: serialize as little as possible on the main thread, and get bytes out of the process fast. What that buys, concretely:

• Low overhead in the hot path — Pino writes newline-delimited JSON through SonicBoom, a buffered write stream that batches syscalls instead of issuing one write() per log line.
• Built-in redaction — the redact option compiles a set of paths into a fast censoring function. Security becomes a config concern, not something each developer must remember at each call site.
• Child loggers — logger.child({ requestId }) gives you per-request context propagation at almost no cost.
• Transports run off-thread — pino.transport() moves formatting and shipping into a worker thread, keeping the event loop free.
• It writes to a file cleanly — which is exactly what this architecture requires.

The problem it solves is the two classic logging taxes — CPU spent formatting strings, and event-loop blocking on synchronous stdout writes — while producing a format that is machine-consumable by default. That matters in any Node.js service where log volume is non-trivial and logs will be consumed by tooling rather than only by human eyes. The drawbacks are real too:

• Asynchronous, buffered writes mean that on a hard crash (SIGKILL, an OOM kill) the last buffered lines can be lost — the exact opposite of what you want when debugging a crash. Mitigation: sync: true for fatal-level paths, or a process.on('exit') handler calling logger.flush(). There is no free lunch; you are trading durability for throughput.
• Redaction only protects paths you declared. An unknown nested object leaks.
• Raw output is unreadable without a formatter.

Compared with the alternatives:

• Winston — far more flexible transport ecosystem and formatting layers, at a meaningfully higher per-line cost. Choose it if you need many heterogeneous sinks configured in-process.
• Bunyan — the original JSON logger; conceptually similar, effectively unmaintained relative to Pino.
• console.log — unstructured, synchronous to a pipe on some platforms, no levels, no redaction. Acceptable only in scripts.
• OpenTelemetry Logs SDK — the correct long-term answer if you are unifying logs, metrics, and traces under one vendor-neutral pipeline. Heavier to adopt; Pino can feed it.

Folder structure#

src/
├── config/
│   ├── env.ts                  # validated environment (zod/envalid)
│   └── s3.ts                   # S3Client singleton
├── lib/
│   └── logger/
│       ├── index.ts            # pino instance + destination
│       ├── redact.ts           # redaction path list
│       └── serializers.ts      # req/res/err serializers
├── middlewares/
│   └── request-logger.ts       # pino-http wiring + requestId
├── jobs/
│   └── log-archive/
│       ├── index.ts            # cron registration
│       ├── rotate.ts           # rename + reopen
│       ├── upload.ts           # gzip + S3 put + verify
│       └── cleanup.ts          # verified local delete
└── server.ts
logs/
├── app.log                     # current, always open
└── 2026-07-25.log              # rotated, pending upload

Two structural decisions worth naming:

• logs/ sits outside src/, is gitignored, and in containers it is a mounted volume. If it lives on the container's writable layer, rotation still "works" and every archive dies with the container.
• Rotation, upload, and cleanup are three separate modules because they are three distinct failure domains: a filesystem failure, a network/IAM failure, and a cleanup failure. Collapsing them into one function makes the failure states impossible to reason about and impossible to unit test.

The logging module#

The logger is a single module-level singleton. Everything else derives child loggers from it.

import pino from 'pino';

const destination = pino.destination({
  dest: 'logs/app.log',
  sync: false,        // buffered writes
  mkdir: true,
});

export const logger = pino(
  {
    level: process.env.LOG_LEVEL ?? 'info',
    base: {
      service: 'api',
      env: "prerender",
      version: process.env.APP_VERSION,
    },
    timestamp: pino.stdTimeFunctions.isoTime,
    redact: {
      paths: [
        'req.headers.authorization',
        'req.headers.cookie',
        'req.headers["x-api-key"]',
        'res.headers["set-cookie"]',
        'password',
        '*.password',
        'body.token',
        'body.cardNumber',
        'user.email',
      ],
      censor: '[REDACTED]',
    },
  },
  destination,
);

// Critical: lets the rotation job detach from the renamed inode.
process.on('SIGHUP', () => destination.reopen());

process.on('exit', () => logger.flush());

Why each of those lines is there:

• base stamps service, env, and version on every line. Without version, you cannot correlate an error spike to a deploy.
• isoTime costs slightly more than Pino's default epoch milliseconds, but it makes archived files readable and makes Athena / jq date filtering trivial. For a file-archived pipeline that trade is worth it; in an ultra-high-throughput service, keep epoch and convert at read time.
• redact is declarative and centralized. The tradeoff: wildcard paths (*.password) are slower than exact paths and still only cover the shapes you anticipated. Redaction is a safety net, not a policy — the policy is "do not pass secrets to the logger."
• SIGHUP → reopen() is the hinge of the entire rotation design. More on that next.

Request logging attaches a correlation ID and a child logger per request:

import pinoHttp from 'pino-http';
import { randomUUID } from 'node:crypto';

export const requestLogger = pinoHttp({
  logger,
  genReqId: (req) => (req.headers['x-request-id'] as string) ?? randomUUID(),
  customLogLevel: (_req, res, err) => {
    if (err || res.statusCode >= 500) return 'error';
    if (res.statusCode >= 400) return 'warn';
    return 'info';
  },
  serializers: {
    req: (req) => ({ method: req.method, url: req.url, id: req.id }),
    res: (res) => ({ statusCode: res.statusCode }),
  },
});

Note the custom req serializer. Pino's default serializer includes headers; an explicit allow-list is safer than relying on redaction to subtract fields. Allow-list what you log; do not deny-list what you do not.

Daily log rotation#

An unrotated log file has four failure modes, and all four are experienced eventually:

• Unbounded disk growth. A full disk does not degrade a Node service gracefully — writes fail, the process may crash, and on a shared volume the database goes down with it. This is one of the most common self-inflicted production outages.
• Unreadable file sizes. grep on a 40 GB file is a minutes-long operation that saturates disk I/O on a live server.
• No natural archive unit. "Upload yesterday's logs" is only meaningful if a file is yesterday's logs.
• No retention boundary. You cannot expire what you cannot address.

The inode problem#

This is the single most important implementation detail in the whole pipeline. On Linux, fs.rename('logs/app.log', 'logs/2026-07-25.log') changes a directory entry. It does not touch the open file descriptor — the process holds a reference to the inode, not the path. So after the rename, your application happily continues appending to 2026-07-25.log, and the new empty app.log you created sits at zero bytes forever.

The symptom is delightful: rotation appears to work, uploads succeed, and one day you notice yesterday's archive contains today's traffic. The fix is a two-phase rotation:

Phase 1: fs.rename(app.log → 2026-07-25.log)
           app process still writing to old inode
Phase 2: signal SIGHUP
           SonicBoom closes fd, opens logs/app.log fresh
           new inode created, writes resume with zero downtime

destination.reopen() closes the current descriptor and opens the configured path again, creating a new file. Because SonicBoom buffers, the reopen flushes pending bytes into the archived file first — which is correct, since those bytes belong to yesterday. The gap between rename and reopen is sub-millisecond, and the lines written in that window land in the archive rather than in a void. There is no downtime and no dropped line, which is precisely why this pattern is preferable to stopping the process.

Comparing rotation strategies#

• rename + reopen (this design) — the app owns rotation via an explicit signal. Requires an in-process signal handler; in exchange you get zero data loss, full control, and a testable seam.
• logrotate with copytruncate — copies the file, then truncates the original in place. No app cooperation needed, but there is a real race window between copy and truncate where lines are lost, and it doubles disk I/O for the copy.
• logrotate with a postrotate kill -HUP — the same signal mechanism, orchestrated by the OS. Solid on VMs; adds an OS-level dependency that does not exist inside a minimal container image.
• pino-roll — a Pino transport that rotates by size or interval internally. Least code, but less control over the exact filename boundary and the handoff to the upload step.
• Size-based rotation — rotate at N MB. Bounds disk usage under traffic spikes, but produces non-date-aligned files that are awkward to partition in S3.

Time-based rotation was chosen because the archive unit and the retention unit should be the same unit. A 90-day retention policy is trivially expressible over daily files and awkward over 500 MB chunks.

Scheduling#

import cron from 'node-cron';

cron.schedule('0 0 * * *', () => void runArchiveJob(), {
  timezone: 'UTC',
  name: 'log-archive',
});

Use UTC. Local-time midnight in a DST-observing zone is either skipped or executed twice once a year. A duplicated rotation is survivable if the job is idempotent; a skipped one silently merges two days of logs into one file. Log timestamps should be UTC for the same reason. On where the schedule lives:

• node-cron (chosen) — the job runs inside the process that owns the file descriptor, so reopen() is a direct function call rather than a signal. Simplest correct option for a single instance. Drawback: it dies with the process, and it fires on every instance if you scale horizontally.
• System crontab or a systemd timer — survives app restarts, but must signal the app externally and cannot easily report failures into your logging pipeline.
• A BullMQ repeatable job — the right answer at multi-instance scale: Redis gives you a single winner per scheduled tick, retries with backoff, and observability. Drawback: the worker that wins the tick may not be on the host holding the file. That is the point at which local-file logging stops being the right architecture at all.

For multiple instances today, the pragmatic fix is to include the instance identity in the filename and S3 key: 2026-07-25.api-7f3c9.log.gz. Never let two processes rotate the same file. Guard against overlap with a lock — an in-memory boolean for a single instance, a Redis SET NX with a TTL otherwise. If yesterday's upload is still retrying when tonight's rotation fires, you want the second run to skip, not to interleave.

Uploading archives to S3#

The upload step has one hard requirement: the local file may only be deleted after the object is provably in S3. Everything else is optimization.

import { S3Client, PutObjectCommand, HeadObjectCommand } from '@aws-sdk/client-s3';
import { createReadStream, createWriteStream, promises as fs } from 'node:fs';
import { createGzip } from 'node:zlib';
import { pipeline } from 'node:stream/promises';

export async function archive(localPath: string, date: string) {
  const gzPath = `${localPath}.gz`;
  await pipeline(createReadStream(localPath), createGzip(), createWriteStream(gzPath));

  const [year, month, day] = date.split('-');
  const key = `logs/service=api/year=${year}/month=${month}/day=${day}/${date}.log.gz`;

  await s3.send(new PutObjectCommand({
    Bucket: process.env.LOG_BUCKET,
    Key: key,
    Body: createReadStream(gzPath),
    ContentType: 'application/x-ndjson',
    ContentEncoding: 'gzip',
    ChecksumAlgorithm: 'SHA256',
    ServerSideEncryption: 'aws:kms',
    SSEKMSKeyId: process.env.LOG_KMS_KEY_ID,
  }));

  // Verify independently before destroying the only other copy.
  const head = await s3.send(new HeadObjectCommand({ Bucket: process.env.LOG_BUCKET, Key: key }));
  const local = await fs.stat(gzPath);
  if (head.ContentLength !== local.size) throw new Error('size mismatch, aborting delete');

  await fs.unlink(gzPath);
  await fs.unlink(localPath);
}

The decisions inside that function, and why:

• Gzip before upload — NDJSON is extremely repetitive (the same keys on every line), so it compresses very well. You pay a little CPU once, at midnight, and cut both storage cost and upload time. Tradeoff: the object is no longer directly readable without decompression, though Athena and most log tools read gzip natively.
• Date-partitioned key prefixes — year=/month=/day= is Hive partition syntax. It costs nothing now and means partition pruning works immediately if you ever point Athena or Glue at the bucket: a query for one day scans one day. A flat logs/2026-07-25.log.gz layout forces full-bucket scans. (The old advice about randomizing prefixes for performance is obsolete; S3 scales per prefix automatically.)
• ChecksumAlgorithm: 'SHA256' — S3 validates the payload server-side and rejects a corrupted upload. Do not rely on comparing ETag to a local MD5: with multipart uploads or SSE-KMS the ETag is not the object's MD5, and that assumption fails silently exactly when you scale up.
• Explicit HeadObject verification — a PutObject that resolves is strong evidence, but not proof of what you think it is (wrong bucket, wrong key, a retry that raced). Since the next operation is an irreversible delete, verify independently.
• Failure keeps the file — if the upload fails, the archive stays on disk and the job exits with an error that itself gets logged and alerted. Tomorrow's run should sweep any YYYY-MM-DD.log files it finds, not just yesterday's, which makes the job self-healing across transient S3 or IAM failures.

PutObject handles up to 5 GB. If daily volume approaches that, switch to @aws-sdk/lib-storage's Upload, which handles multipart and retries per part. Always configure AbortIncompleteMultipartUpload in the lifecycle policy — orphaned parts are invisible in the console and billed forever.

The alternatives, weighed honestly:

• Batch upload of a rotated file (this design) — simple, cheap, one PUT per day. Tradeoff: up to 24 hours of logs exist only on one disk.
• Streaming each line to S3 or CloudWatch in real time — near-zero data loss window and immediate searchability. Costs per request or per GB ingested, adds a network dependency to the hot path, and needs buffering for outages.
• A sidecar agent (Fluent Bit, Vector, the CloudWatch agent) — the standard answer for container fleets. Handles multi-instance, buffering, and backpressure properly. Tradeoff: another component to operate and configure.

Be honest about which you need. If losing up to a day of logs from a lost instance is unacceptable, this architecture is wrong for you and an agent is right.

Automatic retention with S3 Lifecycle#

{
  "Rules": [
    {
      "ID": "expire-app-logs-90d",
      "Status": "Enabled",
      "Filter": { "Prefix": "logs/" },
      "Expiration": { "Days": 90 }
    },
    {
      "ID": "abort-incomplete-multipart",
      "Status": "Enabled",
      "Filter": { "Prefix": "" },
      "AbortIncompleteMultipartUpload": { "DaysAfterInitiation": 7 }
    }
  ]
}

Logs must never live forever, for three reasons:

• Cost compounds silently. Log storage grows monotonically, and it is never urgent enough to fix until it is a line item someone notices.
• Liability grows with the data. Every log line you keep is a line an attacker can exfiltrate and a line you may have to produce in discovery. Under data-minimisation principles in regimes like GDPR, keeping personal data indefinitely without justification is itself the violation.
• Old logs have near-zero value. Debugging value decays sharply after days; compliance value is defined by a fixed window, not by "forever."

Pick the window deliberately: 90 days is a common operational default, but regulated workloads (PCI DSS, for example, has explicit audit-log retention requirements measured in months) may require longer. The number should come from a policy, not from a developer's guess.

Why S3 Lifecycle beats a manual delete job#

Lifecycle makes deletion a declarative property of the bucket rather than an imperative task in your codebase, which kills the "cleanup cron that quietly died" class of failure — where retention appears to be enforced for eleven months and then is not. Against a DeleteObjects cron job it wins on every dimension:

• Reliability — it runs as an AWS-managed service, with no compute of yours to crash. A delete job depends on your process, your scheduler, your credentials, and your error handling.
• Cost — expiration deletes incur no request charges, and you stop paying for storage as soon as an object becomes eligible, even if physical deletion lags. A job costs LIST + DELETE requests plus the compute running them.
• Blast radius — the rule is scoped to a prefix and reviewed as infrastructure code. A bug in a date comparison deletes 90 days of logs in one call.
• Security posture — the application role needs no s3:DeleteObject permission at all. With a job, the app (or something holding app credentials) must hold delete rights on your audit trail.
• Coverage and auditability — lifecycle applies to existing and future objects automatically and is visible in bucket config, enforceable via SCP or AWS Config. Job logic is buried in application code and must be maintained as prefixes and naming change.

That fourth point is the strongest argument and the one most often missed. If your application can delete its own logs, an attacker who compromises your application can erase the evidence. Lifecycle rules let you build a bucket where the app can PutObject and nothing else, and where retention is enforced by a principal your app cannot reach.

The drawbacks, stated plainly:

• Lifecycle is asynchronous. Objects are deleted after the threshold, not exactly at it — usually within a day or so. Billing stops at eligibility, so this costs nothing, but do not build a compliance claim on "deleted at exactly 90 days."
• Bucket policies cannot prevent lifecycle actions. A misconfigured rule with a broad prefix will delete data regardless of a Deny policy. Review rules like you review IAM.
• With versioning enabled, Expiration on a current object only creates a delete marker; you also need NoncurrentVersionExpiration and ExpiredObjectDeleteMarker or storage grows forever behind the scenes.

Security considerations#

Redact at the source, allow-list at the serializer. The redact config handles known-sensitive paths — authorization, cookie, set-cookie, password, tokens, card data — but redaction is subtractive and only removes what you predicted. Custom serializers that build an explicit object of permitted fields are additive and fail closed. Use both.

Never log full request bodies or full user objects. The moment someone writes logger.info({ user }), the password hash, email, phone, and national ID are in your archive forever — and now in S3, in a bucket with a different access model than your database.

Least privilege at the IAM layer means the instance or task role gets exactly one S3 action:

{
  "Effect": "Allow",
  "Action": "s3:PutObject",
  "Resource": "arn:aws:s3:::my-app-logs/logs/*"
}

No s3:DeleteObject. No s3:GetObject — the app writes logs, it does not read them back; reading is a human or analytics role. That turns the log bucket into an append-only sink from the application's perspective. The rest of the checklist:

• Encryption — enable SSE-KMS with a customer-managed key. SSE-S3 is free and adequate for many cases, but a CMK gives you an independent access boundary and a CloudTrail record of every decryption. Tradeoff: KMS charges per request and adds a dependency; for very high object counts, enable S3 Bucket Keys to cut KMS calls substantially.
• Block Public Access at the account level, plus an aws:SecureTransport deny in the bucket policy. A public log bucket is one of the most reliably damaging misconfigurations in cloud security.
• Consider Object Lock in governance mode if these logs have audit value — it makes objects immutable for a retention period even against an administrator, which is the point of an audit log. Tradeoff: it requires versioning, complicates lifecycle, and mistakes are genuinely unfixable.
• Filesystem permissions — logs/ should be 0750, owned by the service user. Logs on disk are as sensitive as the data in them.

Cost optimisation#

The pipeline is cheap by construction, but a few decisions matter:

• Compress. Gzipping NDJSON reduces both stored bytes and transfer time. This is the single highest-leverage cost decision, and it costs one CPU-second a day.
• One object per day, not per hour or per request. S3 bills per request: a batch design makes request costs effectively zero, while a per-event upload design makes them the dominant cost line.
• Think carefully before adding storage-class transitions — this is where teams lose money trying to save it. S3 Standard-IA has a 30-day minimum billable duration and a 128 KB minimum billable object size; Glacier Instant Retrieval has a 90-day minimum duration and the same size floor; Glacier Flexible Retrieval has a 90-day minimum, and Deep Archive 180 days. Transitioning to Glacier at day 60 under a 90-day expiration means paying a full 90-day minimum for objects you delete at 90, plus a per-1,000 transition request charge. For a 90-day window with daily objects, S3 Standard plus gzip is usually the cheapest and simplest answer.
• Skip Intelligent-Tiering here. It charges a per-object monitoring fee and is designed for unpredictable access patterns; log access is entirely predictable — read soon after write, then never.
• AbortIncompleteMultipartUpload. Orphaned multipart parts are billed and do not appear in a normal object listing. One lifecycle rule eliminates the category.
• Data transfer in is free. Uploading from EC2 to S3 in the same region costs nothing in transfer; avoid cross-region log buckets unless you have a specific durability requirement.

The payoff on the server side is that disk usage becomes bounded by one day of logs plus the retry backlog, rather than growing without limit. That converts an eventual, certain outage — disk full — into a fixed capacity requirement you provision for once. Reliability improves along three axes: the rotation is non-disruptive (no restart, no dropped lines); durability jumps from a single EBS volume to S3's multi-AZ storage, so an instance loss no longer means log loss for anything older than the current day; and every failure mode keeps the data — upload fails, file stays; verification fails, file stays; process restarts, the next run sweeps the backlog. The only irreversible action in the pipeline is gated on a verified success.

Common mistakes#

• Deleting the local file before confirming the upload — the most expensive one-line bug in this design.
• Forgetting the file descriptor reopen. Rotation appears to work; archives silently contain the wrong day.
• Using copytruncate and accepting the race. Fine for access logs nobody reads, unacceptable for audit trails.
• Scheduling in local time. DST will corrupt exactly two days a year, and only in production.
• Assuming a single instance. Two processes writing one file and both rotating it produces interleaved, truncated garbage.
• Logging secrets and hoping redaction catches them. It only catches declared paths.
• logger.info(JSON.stringify(obj)) — this defeats the entire structured pipeline. You get a JSON string inside a JSON string, unqueryable by field.
• Logging at info inside a hot loop. Log volume that scales with request work rather than request count is how a 200 MB/day service becomes a 40 GB/day service overnight.
• No requestId. Without correlation, a 500 MB archive is a haystack.
• Never testing the read path. If you have never once pulled an archive from S3 and answered a real question with it, you do not have a logging system — you have a backup of files nobody can use. Test it before the incident.
• Storing logs "just in case," forever. That is a growing bill and a growing liability, not a strategy.

Practices worth keeping#

• Emit NDJSON, always. Pretty-print only in development.
• Stamp service, env, version, and requestId on every line via base and child loggers.
• Propagate a correlation ID with AsyncLocalStorage so any code path can log with context without threading a logger parameter through every function.
• Use log levels with discipline: error for things a human must act on, warn for degraded-but-handled, info for state transitions, debug for development. If everything is error, nothing is.
• Sample high-volume, low-value routes (health checks, static assets) rather than dropping the level globally.
• Log the decision, not just the event: not "payment failed", but "payment failed" with the provider code, idempotency key, attempt number, and correlation ID.
• Keep logs immutable and append-only. Never edit an archive.
• Alert on metrics, not on log volume. Logs are for investigation; metrics are for detection.
• Encrypt at rest, restrict at the IAM layer, and enforce retention in infrastructure rather than in application code.

Where this goes next#

• Hourly rotation with an hourly key prefix — reduces the worst-case data-loss window from 24 hours to 1 and produces smaller, faster-to-scan objects, at 24× the PUT requests (still negligible).
• Query in place with Athena — the date-partitioned prefix layout means adding a Glue table over s3://bucket/logs/ gives you SQL over the whole archive, paying only per byte scanned. Partition pruning makes single-day queries cheap. This is the highest-value next step for most teams.
• Ship to OpenSearch or a vendor for the hot window while keeping S3 as the cold, cheap, long-term tier: hot search for 7 days, archive for 90.
• Replace the file + cron pipeline with a sidecar (Fluent Bit or Vector) once you run more than a couple of instances — log to stdout and let the collector handle buffering, batching, and multi-destination fanout. That is the natural evolution path, and this architecture is explicitly the pre-scale version of it.
• Adopt OpenTelemetry to unify traceId across logs, metrics, and traces, so one ID in a log line jumps straight to a distributed trace.
• Emit metrics from the archive job itself — last successful upload timestamp, archive size, backlog file count — then alert when the last successful upload is older than 26 hours. A silent archiving job is indistinguishable from a working one until you need the data.
• Object Lock plus a separate audit account for logs with genuine compliance value.

Lessons learned#

The reopen step is the whole game. Everything else in this pipeline is mechanical. The rename/reopen interaction is the one place where the intuitive implementation is silently wrong, and the failure only surfaces when you go looking for a specific day's logs — which is always during an incident.

Make the irreversible step the last step, and gate it on verification. Ordering operations by reversibility is a general principle worth internalizing: compress (reversible), upload (reversible), verify (read-only), delete (irreversible). Any failure before the last step is a no-op you can retry.

Push retention into infrastructure. Every retention job written in application code eventually breaks, and no one notices because success is silent. Lifecycle rules do not have that failure mode, and they let you remove delete permissions from the application entirely — a security win disguised as an ops convenience.

Design the read path before the write path. Partitioned prefixes and structured JSON cost nothing on day one and determine whether the archive is queryable on day 400. Most log pipelines are optimized entirely for writing and are miserable to read.

And know when to stop using this. The architecture is correct for a single VM or a small fleet with per-instance keys. The moment you run ephemeral containers, autoscale, or need sub-minute searchability, local files stop making sense and a collector agent becomes the right answer — the same threshold logic that took SRVJ from one EC2 box to Kubernetes. Recognizing that boundary early is more valuable than making the file-based approach survive one more scaling step.

The best log pipeline is the one that is still working — and still affordable — eighteen months after the person who built it stopped thinking about it. Boring, verified, and declaratively bounded wins.