building a durable crdt sync gateway on ursula

i spent the last few weeks building a system around a durability problem i wanted to understand properly. it's called quorum loro gateway, and it's a rust server that connects loro's crdt sync protocol to ursula, a self hosted, quorum replicated durable stream server built by tonbo io. this is the story of why i built it, what actually broke while i was building it, and how i learned enough to build it in the first place.

i came across ursula a few weeks ago and got curious about what the tonbo team behind it was building: durable, replayable streams over plain http, with writes committed through a quorum before they are acknowledged. i ended up building on top of it, then writing about both the gateway and the ursula behavior that shaped its design.

the problem i wanted to solve

loro is a crdt library. if you don't know what that means, here's the short version. imagine two people editing the same document on two laptops, both offline, both making changes, then both coming back online. crdts are the math that lets those edits merge back together correctly, no matter what order they arrive in, without a central server having to referee anything in real time.

loro comes with its own reference server, but its persistence model was not the guarantee i wanted to build around. i wanted each successful acknowledgement to follow a replicated commit, rather than relying on one server process and its local persistence path.

i wanted an ack to actually mean something. not "a server received this and will try to save it soon." something closer to "this crossed a real durability boundary."

why ursula

the broader idea behind ursula is to give each document, room, session, task, or agent run its own durable timeline. applications can append to it, replay it from an offset, or follow it live over plain http.

for this project, the part that mattered was the durability boundary. recent writes go through ursula's replicated hot path and are acknowledged after a quorum commit, while older stream data can move to s3 later. in the three voter setup i tested, one voter could fail without losing an acknowledged update.

this is where ursula comes in, and i'll be honest, it's a genuinely great fit for this problem. ursula is a self hosted, quorum replicated event stream server. when you write to it, that write gets replicated to a majority of voters before ursula says it is committed. in the three voter setup i tested, one voter could fail without losing an acknowledged update.

ursula also fits this workload because it does not interpret an operation's timestamp or impose an event-time lateness cutoff on ordinary appends. an update created by an offline device can still be appended when that device reconnects; ursula stores it in append order instead of trying to reorder or reject it based on when it was created.

that lines up naturally with loro. ursula is responsible for storing the bytes in durable append order, while loro decides whether an old update can merge into the current document. the gateway does not add a separate staleness rule between them.

what i built

quorum loro gateway is a small rust server that sits between official loro clients and ursula. it speaks the official loro websocket synchronization protocol on one side and stores each accepted update in an ursula stream on the other.

before writing anything, the gateway validates the loro update and wraps the exact received bytes in a versioned, checksummed frame. receiving the update is not enough to acknowledge it. the gateway waits until ursula reports the frame committed before returning Ack(Ok).

it also handles the less clean case where ursula may have committed a write but the response never reached the gateway. the gateway retries using the same producer information and verifies the committed bytes before acknowledging. if it still cannot prove what happened, the room fails closed instead of guessing.

the gateway keeps no durable local database of its own. after a crash, a fresh instance can rebuild the room from the checkpoint and active delta stored in ursula.

how one update moves through the system

a client begins by sending a normal loro document update over the websocket connection. the gateway validates the update, preserves its exact bytes, and encodes them into one frame containing version, length, digest, and checksum information.

the frame is appended to the room's active delta stream. ursula replicates it to a quorum and reports it committed. only after that does the gateway send Ack(Ok) back to the client.

if the commit succeeds but its response disappears, the gateway retries the same frame with the same producer tuple. an ursula duplicate response is not enough by itself: the gateway reads the committed range back and compares the bytes before treating the update as durable.

if the gateway process later crashes, another instance reads the room state from ursula, restores the latest checkpoint, replays the active delta generation, and reconstructs the loro document.

the actual hard part

the normal path is easy to describe: the client sends an update, the gateway writes its frame to ursula, ursula commits it, and the gateway sends an acknowledgement. the difficult case is not a clean success or a clean failure. it is a write whose result becomes unknown.

imagine the gateway sending a frame to ursula. ursula commits it, but the network response disappears on the way back. from the gateway's point of view, the same timeout could also mean that the frame never committed. it cannot safely acknowledge the update, but blindly submitting it again could create another physical copy.

at first i thought an ursula duplicate result would be enough proof. it was not. producer deduplication tells the gateway that the same producer sequence was already handled, but the gateway still needs to prove that the committed bytes match the frame it is currently holding. it therefore reads the returned range back and compares it byte for byte before acknowledging.

if the gateway still cannot prove what happened, the room fails closed. it does not turn a timeout, malformed response, or unresolved retry into Ack(Ok). guessing at that boundary would make the acknowledgement meaningless.

the same ambiguity becomes dangerous during generation rotation. if the gateway switches streams while an append to the old generation is unresolved, that update could commit behind the checkpoint after recovery has already moved on. the room actor therefore cannot rotate until the append is proven committed, proven not committed, or the room has failed closed.

the other hard part was making the gateway disposable. it keeps no durable local database, so a fresh process has to reconstruct the document entirely from ursula after a crash.

to keep that recovery from depending on the document's complete lifetime history, i added an explicit generation system. each document uses named manifest,checkpoint/{generation}, and delta/{generation} streams, and generation names are never reused.

i deliberately did not use ursula's built in snapshot mechanism because publishing one advances the stream's replay boundary and makes older history eligible for trimming. the gateway instead restores the latest checkpoint and replays only the active delta generation. the previous generation is treated as sealed because the gateway stops writing to it, not because ursula closes the stream.

what the failure tests proved

i did not want the acknowledgement rule to exist only as a comment in the code. the useful tests were the ones that forced the gateway through the failure windows where it would be easiest to accidentally claim success.

with one voter unavailable in the three voter cluster, an acknowledged update was still committed by a majority. when the cluster lost quorum, the gateway did not turn the failure into a successful acknowledgement.

i also tested the more awkward case where ursula committed an append but its response was lost. the retry located the earlier committed range, and the gateway acknowledged only after reading that range back and verifying the exact frame.

finally, i killed the gateway and started a fresh instance. the new process reconstructed the room from ursula rather than relying on state left behind by the old process. these tests do not make the prototype production ready, but they gave the central guarantee something concrete to stand on.

the benchmark that mattered most to me

once bounded recovery was working, i ran a simple in memory comparison. replaying a room's full history versus loading a checkpoint plus the small tail after it.

full replay kept getting slower as the room history grew, reaching about 4.8 seconds at 250,000 small updates. that was enough to show the problem clearly: recovery should depend on the latest checkpoint and active delta, not on replaying the room's entire lifetime.

how i actually learned to build this

none of this came from a course. i actually started out writing go, and rust didn't click for me right away, so the first couple days were just working through rustlings until the syntax stopped feeling foreign. i leaned on ai a fair bit early on to get past the parts that weren't intuitive coming from go, then worked through database internals on my own time, and got comfortable enough with distributed systems concepts, quorum, consensus, replication, to know what questions were even worth asking. i learn by picking a real system, poking at it until something breaks or confuses me, then going and finding out exactly why.

for this project specifically that meant reading ursula's protocol docs closely enough to notice an open, unassigned issue about its snapshot mechanism, understanding loro's crdt model well enough to know where the real coordination problems would show up, and being honest enough with myself to flag the one thing i couldn't verify alone instead of just assuming it and shipping anyway.

it's a research prototype, not a production system. it doesn't handle multiple gateway writers yet, there's no auth, and there's no retention policy for old generations. i know exactly what's missing because i wrote it down on purpose. that felt more honest than pretending it was finished.

so what does this all mean

honestly, i'm not sure i have one clean takeaway yet. this is still a research prototype, not a finished sync platform. what it demonstrates is narrower: an acknowledgement can be tied to a real replicated commit, and recovery can start from a checkpoint plus the active delta instead of replaying a document's entire lifetime.

the interesting part was not just getting loro and ursula to talk to each other. it was finding the places where a design that looks correct can quietly lie: a response can disappear after a commit, a retry can succeed without proving which bytes were stored, and a rotation can cross a write whose outcome is still unknown. most of the project became about refusing to guess at those boundaries.

so maybe that is the takeaway. the system is small and unfinished, but its central guarantee is now something i can explain, test, and point at. for a research prototype, that feels like enough.