Inside foony.io’s Realtime Architecture

Howdy! foony.io is our realtime pub/sub service: WebSockets, channels, presence, and message history, the same shape as Ably or Pusher. It started as the backbone for Foony’s multiplayer games, and we productized it once it was carrying our own traffic.

This post walks through how it actually works. Not the marketing diagram, the real one: what runs, what each piece owns, and the two or three decisions that made the rest of the system fall into place. The diagrams are interactive, so click around.

The shape of the system

The whole service is one Go module that ships three binaries.

The split is deliberate. edge is the only thing on the hot path. control exists so the dashboard can never hurt the data plane. metering exists so billing writes can be slow, batched, and replayable without anyone’s message waiting on Postgres.

A few properties worth calling out before we go deeper:

• Edges are stateless. Any pod can serve any connection. Deploys and crashes reshuffle connections, nothing more.
• The WebSocket framing is binary because we measured it. One Go package defines every frame, and the SDKs mirror it. JSON is still supported where it’s convenient (the REST API, and message payloads are plain JSON values), but the framing itself is binary: accepting binary publishes cut edge CPU per message by about 42% (the inbound decode went from ~20% of edge CPU to ~2%), and encoding a delivered message benchmarks about 3.9x faster than json.Marshal.
• NATS JetStream is the only stateful system on the publish path. Postgres holds control-plane data (apps, keys, plans, placement) and answers cached lookups on connect. If Postgres is down, connected clients keep publishing.

A connection’s life

Every WebSocket runs as an isolated little state machine inside the edge:

1. The socket upgrades.
2. The client gets 5 seconds to send its first frame, which must be auth (a JWT or an API key). Anything else, or silence, closes the socket.
3. On success the edge replies connected, and the loop begins: sub, unsub, pub, presence frames, hist, fetch, and ping, one frame at a time.
4. An unknown frame gets an error reply instead of a hangup, so an older SDK can degrade gracefully.
5. On close, the edge drops every NATS subscription the connection owned.

Here’s that life on the wire, frame by frame:

There’s also a REST API for backends that don’t want to hold a socket open: publish, history, presence, and token minting over plain HTTP. It’s served by the same listener as the WebSocket, on the same pods, through the same quota and metering code. A REST publish is indistinguishable from a WebSocket publish, to subscribers and to the bill.

Cells: sharding by app, not by machine

One NATS cluster can’t be the whole product forever, so the unit of horizontal scale is a cell: a self-contained NATS JetStream cluster that owns a slice of apps, holding everything those apps need (the message log, presence, retained values, and the ownership leases).

Which cell owns which app lives in a Postgres table. The edge caches those lookups with refresh-ahead: an entry nearing expiry gets refreshed in the background, so an app that’s actively connecting never blocks on the database. An app with no placement row falls back to the default cell, which means existing apps needed no backfill when cells landed.

Here’s a cell opened up, and a whole app migration, step by step:

One edge pod can serve apps on different cells at the same time, and “serve” is very literal: a cell is just a NATS cluster, so the placement row carries its NATS URL, and the edge dials it lazily the first time a connection needs it, then caches that cell’s whole resource set (publisher, history reader, presence, leases). The pods stay stateless, and the cells stay independent: a bad day in one cluster is one slice of apps, not the product.

The hard part: ordering without consensus

Everything above is fairly standard. This is the part I’d actually defend in a design review.

foony.io promises that each channel’s messages have a contiguous serial: 1, 2, 3, with no gaps and no reordering. Serials are what make resume exact. A client that reconnects anywhere, on any pod, says “I have serial 41” and gets exactly 42 onward. A gap tells the SDK it missed something. A duplicate tells it to drop the frame.

Numbering messages 1, 2, 3 with no mistakes needs a single counter per channel, which means one machine has to be in charge of each channel’s counter. The classic answer is consensus, with a Raft group per shard. We didn’t build that, because NATS JetStream already replicates the message log for us. What we needed was much smaller: agreement on who holds the counter, and a way to survive being wrong about it.

Channels hash into partitions (fifteen per cell), and each partition has exactly one owner pod at a time. The sequencer is edge code, but everything it trusts is scoped to the app’s cell: the leases live in that cell’s KV bucket, and the counters are recovered from that cell’s log.

• Ownership is a lease, an entry in that KV bucket that expires after a few seconds. The owner keeps re-writing its entry to stay owner. If the pod dies, it stops re-writing, the entry expires, and another pod claims the partition.
• Claims are fair-share, not first-come-take-all. Every pod also keeps a liveness key in the bucket, so each pod holds at most its share of the partitions given the live pod count. A pod that’s over its share (because a new pod joined) releases the excess.
• A new owner needs no handoff. Every stored message carries its own serial, so when a new owner gets its first publish for a channel, it reads that channel’s newest stored message and continues counting from there. The log is the source of truth, and the counter in memory is just a cache of recently active channels (the ones idle longest get evicted). Fifty million idle channels cost nothing. An evicted channel that wakes up pays one read.

Here’s the whole lifecycle, live. Kill a pod and watch its leases decay and get reclaimed:

pod	1	2	3	4	5	6	7	8	9	10	11	12
edge-a
edge-b
edge-c

Kill a podAdd a podReset

Leases bound how long two pods can disagree about ownership, they don’t make it impossible. Here’s the race. Pod A owns a partition and then stalls for a few seconds, maybe a garbage-collection pause, maybe a network blip. Its lease expires, pod B claims the partition and starts writing. Then pod A wakes up. It has noticed nothing. It still believes it’s the owner, and it has a write already on its way out the door. Nothing about the lease can stop that write.

That’s what the fence is for, and it’s my favorite detail in the system. Every append carries a condition. When the owner assigns a serial, it also remembers which stored message is currently the newest one on that channel. The append then tells JetStream: store this only if that message is still the newest. The check and the write happen as one atomic step inside the stream itself (JetStream’s Nats-Expected-Last-Subject-Sequence header, if you want to look it up). So when pod A’s late write finally arrives, pod B has already appended, A’s condition is false, and the stream refuses the write. A re-reads the log, discovers the world moved on, and steps down. Serial order and stored order cannot diverge, because the only way to append is to correctly name the current end of the log.

Two smaller decisions round this out:

• Reserve, then commit. The counter doesn’t move forward until the append succeeds, so a failed append burns nothing. Subscribers never see a gap for a serial that was assigned but never stored.
• Dedupe by message id. Every stored publish carries an id the client chose. The stream remembers the ids it has seen recently, and a second publish with the same id is dropped instead of stored. So when the SDK’s socket dies right after sending and it resends on reconnect, or a publish being passed to the owner pod gets retried, only one copy ever lands in the log.

Here’s the full path a durable publish takes, step by step:

Note what’s absent: no Paxos, no Raft of our own, no coordinator service. One KV bucket for leases, one conditional append for safety, and a log that already knew how to be replicated.

The read side

Fan-out is the easy half, so it stays simple. Each edge bridges NATS subjects to its local WebSocket subscribers. History reads come from the stream, paged by serial. Two features deserve a note:

• Presence is a JetStream KV bucket where every member of a channel holds one key, with per-key TTLs. A connection that vanishes without saying goodbye stops refreshing its key, the key expires, and that expiry becomes the leave event everyone sees. Ghost detection is a property of the store, not a sweeper job we had to write.
• Retained last value is a channel rule for state-shaped channels (a scoreboard, a sensor reading). The bucket keeps only the latest message per channel, and a client that attaches gets that one value replayed immediately, without reading history. KV’s last-write-wins per key is exactly the semantics needed, so the feature is a thin wrapper.

Here’s presence doing its thing. The bars are key TTLs, and the re-PUT flashes are the refresh heartbeats:

presence bucket · room-1

room-1.alice.c1

refreshing

room-1.bob.c2

refreshing

room-1.carol.c3

refreshing

watchers see: alice · bob · carol

Someone joinsClean leaveKill someone's wifiReset

Which channels persist history at all is a per-app ruleset, like Ably’s channel rules: the longest matching channel prefix wins, and an empty prefix is the default. Ephemeral channels (cursors, typing indicators) skip durable storage entirely and never pay the sequencing cost.

History that ages like an LSM tree

Thirty days of history for every persisted channel can’t live inside NATS. The first version tried to make NATS do retention anyway, with a TTL stamped on every durable message, and it taught us a lesson: each per-message TTL became a task in JetStream’s internal timer wheel, and iterating that wheel grew to roughly 80% of nats-server CPU. The fix was to delete the concept. Each tier is now its own stream with a plain MaxAge (a cheap oldest-message check), and real retention moved out of NATS entirely.

What runs instead borrows the shape of an LSM tree, the storage design under databases like RocksDB: new data lands in a small fast buffer, gets written out to files that never change again, and a background job merges small files into bigger ones. Ours looks like this:

• The hot stream is the fast buffer (LSM people call it the memtable). Each persist stream keeps only a short window, around 10 minutes, on the machine’s local disk.
• The archiver is the flush. It drains each partition stream in order, seals batches into compressed segments in R2 (Cloudflare’s object storage, think S3), records them in an index, and only then purges the hot stream. That order is the whole safety story: nothing is ever deleted before its copy is confirmed in R2, so a stalled archiver can never lose history. It just stops purging and the stream grows toward its backstop.
• The compactor is, well, compaction. A quiet channel leaves lots of tiny segments, so once a channel has 32 or more index rows, runs of small segments merge into slices of about 256 KiB. The trigger is row count, not age: a one-message-a-day channel compacts about once a month, a busy one compacts continuously.
• A history read is a stitched scan. The recent tail comes from the hot stream, older pages come from R2 through the index, and serials line the two up exactly.

The 30-day MaxAge on the hot stream is deliberately a backstop, not the mechanism. It only fires if the archiver is fully dead, as a disk-fill guard, and it sits far above the archiver’s window so a lagging archiver can never let NATS delete something that hasn’t reached R2 yet.

The Spanner ideas this steals, and the ones it skips

A lot of this design is Spanner-shaped on purpose. The parts that survived contact with a team much smaller than Google’s:

Spanner	foony.io
Placement metadata (which shard holds which directory)	The placement table (which cell holds which app)
Paxos leader leases	Partition-ownership leases in the cell’s KV bucket
Epoch fencing on writes	The expected-sequence fence on every durable append
The replicated log as the source of truth	JetStream streams, recovered from on every failover

And the parts deliberately skipped, starting with TrueTime and commit wait. TrueTime is Spanner’s answer to the fact that no two servers agree on what time it is. Ask a normal clock and you get one number that might be wrong. Ask TrueTime and you get an interval, [earliest, latest], guaranteed to contain the true time, kept a few milliseconds wide by GPS receivers and atomic clocks in every datacenter. Spanner stamps a transaction with latest, then deliberately waits until earliest on a fresh reading has passed that stamp before revealing the commit. That little wait is what lets two transactions on opposite sides of the planet come out in the order they really happened (the Spanner paper tells it well). We skip all of it because each channel’s order comes from a single owner pod, so no clocks ever need comparing. We also skip consensus of our own, because JetStream already runs Raft under every stream and KV bucket, one level down. And we skip ordering promises between different channels, because no realtime customer has ever asked for them.

Counting the money without touching the hot path

Billing is where realtime systems quietly go wrong, because the naive version puts a database write next to every message. Ours never touches Postgres on the publish path:

1. Edges count usage in memory and emit rollups into a JetStream stream.
2. The metering binary consumes the rollups and applies them to Postgres idempotently, so a crashed writer replays without double billing.
3. The billed unit is a message of up to 5 KiB. Bigger payloads count as multiple messages, rounded up.

The dashboard’s live numbers cheat in the same spirit: each pod flushes per-app counters aggregated per pod, so the write cost scales with the number of pods, not the number of connections.

What I’d tell past me

The log is the source of truth, and everything else is a cache. This one sentence generated half the design. The sequencer’s memory is an LRU over the log. A new partition owner recovers from the log. A fenced-out pod re-reads the log. Every time we asked “what if this state is lost or stale”, the answer was already sitting in the stream.

Lease plus fence beats consensus you don’t need. We got single-writer ordering with a KV bucket and a conditional append. The trick is accepting that the lease will occasionally be wrong, and making the storage layer reject the loser instead of trying to make the lease perfect.

Keep the blast radii separate. Stateless edges mean deploys are boring. Cells mean a sick cluster is a slice of apps. A control plane that’s never on the hot path means the dashboard can fall over during your best traffic day and nobody publishing a message will know.

If you want to poke at the real thing, the docs live at foony.io. And if you’ve built ordering on top of a replicated log a different way, I’d genuinely like to hear how it went. I’m @theacorn1010.