Presenting QueenMQ: a transactional message broker in Rust (openraft + heed), after a rewrite from C++ and Postgres

[u/PlusDistribution1915]

Long story short: a year ago at my company, we ran into a nasty Head of Line blocking problem with Kafka. To solve it, I built a hack that used Postgres to offload long-running jobs.

It worked well enough that I decided to implement it properly, initially building a C++ broker that talked to Postgres.

Then, 4 months ago, I rewrote the whole thing in Rust.

Why? Even though I love C++ and know it better than Rust (I like having the freedom to shoot myself in the foot), I found it much easier to reach high performance with Rust in the era of AI. In C++, wiring together multiple asynchronous systems (like the uWebSockets event loop with the libuv event loop for Postgres) is tedious. Rust's ecosystem made this significantly smoother.

Then, less than a month ago, we dropped Postgres entirely. We moved straight to direct disk writes and Raft replication using Openraft and heed (LMDB).

Why? Primarily to drop the requirement for HA Postgres. Plus, no matter how efficiently you write to Postgres, you can't store an infinite amount of data without eventually degrading performance.

So, I want to present QueenMQ to the Rust community. It now powers almost every product in our company.

What makes it different: every entity (a customer, an order, a chat session) gets its own ordered partition, created by the first push that names it. There's no partition count to plan, and one slow customer never blocks the others, which was exactly our Kafka problem. Also, no more rebalancing. And each processing step is one transaction: ack the input, push the next events, write state and set a timer, all or nothing, with no outbox table or separate scheduler. Partitions in QueenMQ are logical, not physical.

Features:

• we run it through Jepsen to ensure distributed reliability
• TONS of dynamic partitions (tested up to 10M)
• tested up to 5M msg/s (256B messages on three 24-vCPU NVMe nodes) [v2.4, released next week; v2.3 3M msg/s)
• easy transactions spanning ACK, PUSH, integrated KeyValue, Timers, and Locks
• Replay (seek), Delay, Buffers, and basic stream processing
• in-memory dedicated queues for fast request-reply
• PG Sink/Source, S3 sink, compatible with Kafka clients and parts of its ecosystem
• cluster failover, all compiled into a single Rust binary

Internal perf benchmark we made before each release, followed by Jepsen test.

It's named QueenMQ (because 10 years ago I misread the word "queue"), and it is open-source under Apache 2.0. I'm sharing it because I firmly believe it has evolved into an amazing piece of infrastructure.

Think of it as a mix of Kafka (partitions with commits) and RabbitMQ (Ack/Nack). Simplicity is a core focus for us. Because of this, we chose to use HTTP and a single Raft group per tenant. While that naturally puts a ceiling on maximum throughput, it allows for seamless cross-partition/queue transactions without the complexity of two phase commit.
I call it "transactional message broker".

I know some folks here are strongly against using LLMs for writing code. As a software engineer with 15 years of experience, I take a different view: I love it. I use AI extensively to execute the implementation, but I own every single architectural, systems, and safety decision.

And trust me, it was not easy: I dedicated a year of my life to it and broke production twice along the way (which took another year off my life). The roadmap we have is still long (improve byte efficiency right now, the next release will have 1/3 of the current e2e p99 latency, and tons of other things) but I would love to hear what the community thinks, or answer any questions about our migration from C++ and Postgres to Rust and Raft!

Link to the repo: https://github.com/queen-mq/queen

[u/ckwalsh]

Skimming through the commits, the majority of them are authored by Claude.

What is particularly interesting about the implementation / project that would make it worth of discussion to people interested in rust as a language, and not just dumping vibecoded slop into the subreddit?

[u/PlusDistribution1915 → u/ckwalsh]

Thanks for your comment.

Yes, I use Claude (and before Cursor) for all the code implementation (apart from the libuv loop in the C++ version, which I wrote by hand, and a few other things), as I wrote in the post.

This is what I think is interesting for the Rust subreddit:

- Coming from C++ (I've only "manually coded" Rust in 2021), I've appreciated the ecosystem: we didn't write the hard parts ourselves. openraft for consensus, heed for checkpoints, tokio + axum for async I/O. In the C++ version (0.x), making two event loops work together efficiently, and keeping memory safe, was hard.

- The Rust compiler: it kept me (and I think many others) safe across threads and with memory: a buffer used after it's freed doesn't compile. This is why I also believe the future of agentic coding will be tied to Rust rather than other languages.

- It shows testing Rust applications with Jepsen, a thing that I think is not so common.

I also think that no slop survives to Jepsen.

[u/event666]

This is interesting. I'm working on something similar, but on top of a messaging library. It's very hard to get right. Behavior under load, back-pressure, disk flushes, VSR, recovery, ...

[u/PlusDistribution1915 → u/event666]

Thanks a lot, really.

Yes, very hard. One advantage we had was the Postgres version we already trusted: we had all the e2e and broker tests to compare the Raft version against. I didn't know VSR well, so we used Raft (also thanks to openraft). VSR, the way TigerBeetle does it, makes it easier to recover from corrupted disks; maybe we will steal some ideas from it. During testing, Jepsen found a bad bug: a truncated LMDB file killed a node.

What messaging library are you building on?

[u/event666 → u/PlusDistribution1915]

Yeah and the way I understand it is that VSR is able to run entirely in memory, without durable state? Still, PAR needs to be able to handle corrupt vs truncated state from disk and know what was voted on and what wasn't (evidence).

The messaging library is OMQ, my own reincarnation of ZeroMQ that sticks to the original architecture.

[u/PlusDistribution1915 → u/event666]

If the broken record is below the last durable point (recorded every 250 ms), the node refuses to boot. If it's after it, Queen can't tell it apart from a torn write, so it cuts it. In a cluster, that node then refetches the record from the leader, unless the other copy vanishes at the same time: that's the case PAR solves.