Deep Dive · Real-time
SRVJ
A real-time collaborative diagram editor: CRDT synchronization, presence, and persistence built from protocol level up.
- Role
- Solo — design and implementation, backend and frontend
- Timeline
- —
- Stack
- TypeScript Express.js Yjs (CRDTs) PostgreSQL Prisma MongoDB Redis BullMQ Docker Vue.js ...
01 — Overview
SRVJ is a collaborative diagram editor where multiple people edit the same board at the same time — live cursors, presence, and conflict-free merging of concurrent edits. The interesting work is invisible: keeping every client convergent without a central lock, and persisting a document that is technically a binary CRDT state.
I built it solo, end to end: the synchronization server, authentication and access control, the persistence layer, notifications, background processing, and the Vue frontend. The goal was to understand real-time collaboration infrastructure by building it from the protocol level up, not by wrapping a hosted service.
02 — What I Built
CRDT synchronization server
- Problem
- Concurrent edits from multiple clients must converge to the same document without locking or a "last write wins" data loss.
- Approach
- Yjs CRDTs carry the conflict resolution; the server’s job is transport and fan-out, not merging.
- Implementation
- A custom WebSocket server implementing the Yjs sync and awareness protocols. Each board gets a dedicated room, so collaboration sessions are isolated and a busy board cannot leak updates or presence into another.
- Outcome
- Live cursors, presence, and seamless multiplayer editing, with convergence guaranteed by the CRDT rather than by server-side coordination.
Authenticated, authorized WebSocket connections
- Problem
- WebSockets bypass the usual per-request middleware chain, so authentication and permissions have to be enforced at the connection boundary.
- Implementation
- Connections authenticate with PASETO v4 tokens, and project-level role-based access control is enforced before a client joins a board room.
- Outcome
- A client that should not see a board never receives its state or its awareness traffic.
Dual-snapshot persistence
- Problem
- The authoritative document is a Yjs binary state — lossless, but opaque to queries. An API that needs to list, search, or render previews cannot work against a binary blob.
- Implementation
- Every snapshot is stored twice: the authoritative Yjs binary state for lossless recovery, and a denormalized JSON representation for efficient querying and API reads.
- Outcome
- Recovery replays exact CRDT state; read paths never touch or decode the binary format.
Split storage: PostgreSQL and MongoDB
- Implementation
- PostgreSQL with Prisma holds the relational data — users, projects, memberships, sharing — where integrity and joins matter. MongoDB with Mongoose absorbs the high-frequency diagram mutation writes.
Scalable notifications and background jobs
- Problem
- Server-sent notification streams are pinned to one process; with multiple instances behind a load balancer, an event raised on one instance must reach clients connected to another.
- Implementation
- Notifications go out over SSE with Redis Pub/Sub fanning events out across instances. BullMQ workers handle asynchronous email and notification processing off the request path.
- Outcome
- Notification delivery scales horizontally, and slow work never blocks an API response.
Reliability and operations baseline
- Implementation
- Zod validation on inputs, Helmet and CORS, rate limiting, centralized error handling, structured logging with Pino, and Prometheus metrics. Collaboration behavior is verified with integration tests, and the platform runs containerized via Docker Compose behind Nginx.
03 — Engineering Decisions
Two snapshot formats instead of one
The Yjs binary state is the only lossless representation of the document, but it cannot be queried. Rather than force one format to do both jobs badly, each read path gets the format built for it: binary for recovery, denormalized JSON for the API.
Tradeoff
Every snapshot is written and stored twice, and the two representations must be kept consistent.
PostgreSQL and MongoDB side by side
Identity, membership, and sharing are relational problems — foreign keys and constraints catch real bugs there. Diagram mutations are high-frequency, schema-light writes that fit a document store.
Tradeoff
Two databases mean two operational surfaces: separate backups, migrations, and failure modes.
SSE for notifications, WebSockets for collaboration
Board collaboration is genuinely bidirectional, so it runs over WebSockets. Notifications are one-way server-to-client, and SSE with Redis Pub/Sub fan-out delivers them across horizontally scaled instances without holding a second full-duplex socket per user.
04 — Security
PASETO v4 tokens
WebSocket connections are secured with PASETO v4 authentication before any board state is exchanged.
Project-level RBAC
Role-based access control is enforced per project, and collaboration sessions are isolated into dedicated board rooms.
Input and transport hardening
Zod validates inputs at the boundary; Helmet, CORS, and rate limiting cover the transport layer; errors flow through one centralized handler so failures never leak internals.
Further Reading
- CRDTs, Yjs, and the Day I Stopped Writing Conflict-Resolution CodeWhy I stopped writing conflict-resolution code for SRVJ's collaborative diagrams — CRDTs from first principles (G-Counter, LWW-Register, OR-Set, sequence types), then Yjs and the authenticated WebSocket relay that keeps every editor converged.
- Server-Sent Events (SSE): Real-Time Notifications in SRVJHow SRVJ delivers real-time notifications with Server-Sent Events, BullMQ, Redis Pub/Sub, and PostgreSQL — a persist-then-fan-out pipeline that scales horizontally without sticky sessions.
- JWT vs PASETO: Choosing the Right Token for the JobI shipped JWT in production, got burned, and switched to PASETO for auth and payments — but the real lesson is token taxonomy: signed vs encrypted vs opaque, and which job each one actually belongs to.
- From One EC2 Box to Kubernetes: How SRVJ Actually Deploys on AWSHow SRVJ's deployment grew up in three acts — a single EC2 box with NGINX and pm2, Docker Compose, and finally a Kubernetes cluster with kustomize, an NGINX ingress, and an HPA. Plus the S3 patterns that outlived every stage.