i spent the last few weeks building something nobody asked for. it's called quorum loro gateway, and it's a rust server that connects loro's crdt sync protocol to ursula, a self hosted, quorum replicated durable stream server built by tonbo io. this is the story of why i built it, what actually broke while i was building it, and how i learned enough to build it in the first place.
the problem i wanted to solve
loro is a crdt library. if you don't know what that means, here's the short version. imagine two people editing the same document on two laptops, both offline, both making changes, then both coming back online. crdts are the math that lets those edits merge back together correctly, no matter what order they arrive in, without a central server having to referee anything in real time.
loro comes with its own reference server. it works, but it has a weak spot. it holds the document in memory and periodically saves a backup to local disk. if that one process crashes between backups, whatever got acknowledged since the last save is just gone. the app told you "saved" and it lied.
i wanted an ack to actually mean something. not "a server received this and will try to save it soon." something closer to "this survived, no matter what happens next."
why ursula
this is where ursula comes in, and i'll be honest, it's a genuinely great fit for this problem. ursula is a self hosted, quorum replicated event stream server. when you write to it, that write gets copied to multiple machines, and only once a majority of them confirm they have it does ursula tell you it's committed. one machine dying five seconds later doesn't matter, because the data already exists somewhere else.
ursula has this great property that most databases don't bother with: it doesn't impose a deadline on when data can arrive. most streaming systems assume events show up roughly in order, close to real time, and anything too late gets rejected or dropped. ursula just doesn't do that. an event that shows up a week late is treated exactly the same as one that shows up a second late. i asked tonbo's founder about this directly, and he confirmed it's deliberate, not a missing feature. he said flat out that for workloads like chats, agent runs, and collaborative documents, there's no need for that restriction, and he doesn't want to add one.
that one design choice turns out to be exactly what a crdt sync system needs. loro's whole job is merging operations correctly no matter when they arrive. ursula's whole job is storing them durably no matter when they arrive. neither system cares about lateness, and that's not a coincidence you can plan around after the fact, it's a foundation you get to build straight on top of.
the actual hard part
the easy version of this project is: client sends an update, gateway writes it to ursula, gateway sends an ack. that part isn't hard.
the hard part is making the gateway itself disposable. any instance should be able to crash at any moment and get replaced by a fresh one that rebuilds the document purely by reading from ursula. no local state that matters survives anywhere except ursula.
to make recovery fast, the gateway checkpoints periodically instead of replaying a document's entire history every time it restarts. i deliberately didn't use ursula's built in snapshot feature for this, because it's single slot and destructive, publishing a new one throws away your ability to recover anything before it. instead i built my own generation system: each document rotates through named streams, manifest, checkpoint/{generation},delta/{generation}, none of which ever get reused.
the sharpest edge in the whole design showed up at the exact boundary where one generation closes and the next one opens. picture a client's write landing right as the gateway decides to seal that generation. the network hiccups, the response gets lost, and now the gateway genuinely doesn't know if that write committed before the stream closed or not. get this wrong and you either lose an acknowledged write or accidentally save it twice.
i designed around it by never letting an ambiguous write move forward blindly. the gateway retries the exact same write against the generation that might have just closed, using ursula's own duplicate detection to resolve it, before it's ever allowed to consider the next generation. but the whole thing rested on one assumption i couldn't verify from documentation alone: does ursula check for a duplicate before or after it checks if the stream is closed. get that ordering wrong and the entire mechanism falls apart quietly, not loudly.
so i asked. tonbo's founder wrote back and confirmed it directly, dedup runs before the closed stream check, a retry of a write that actually committed returns the original result even after closure, and it's documented behavior in their spec. that one answer is the difference between a design that sounds right and one that actually is.
the benchmark that mattered most to me
once bounded recovery was working, i ran a simple in memory comparison. replaying a room's full history versus loading a checkpoint plus the small tail after it.
full replay took 261 milliseconds. bounded recovery took 8. same room, same 2,000 updates, same final state. it's not a real cluster benchmark, i said as much in the readme, but it proved the thing i actually needed proof of: recovery cost now depends on how recent your last checkpoint is, not on how old the document is.
how i actually learned to build this
none of this came from a course. i actually started out writing go, and rust didn't click for me right away, so the first couple days were just working through rustlings until the syntax stopped feeling foreign. i leaned on ai a fair bit early on to get past the parts that weren't intuitive coming from go, then worked through database internals on my own time, and got comfortable enough with distributed systems concepts, quorum, consensus, replication, to know what questions were even worth asking. i learn by picking a real system, poking at it until something breaks or confuses me, then going and finding out exactly why.
for this project specifically that meant reading ursula's protocol docs closely enough to notice an open, unassigned issue about its snapshot mechanism, understanding loro's crdt model well enough to know where the real coordination problems would show up, and being honest enough with myself to flag the one thing i couldn't verify alone instead of just assuming it and shipping anyway.
it's a research prototype, not a production system. it doesn't handle multiple gateway writers yet, there's no auth, no automatic rotation, no retention policy for old generations. i know exactly what's missing because i wrote it down on purpose. that felt more honest than pretending it was finished.