June in TigerLand

Jul 17, 2025

Dear friends,

We hope your June was joyous. TigerBeetle got Jepsen’d! We also made consensus more resilient to metastability failures, landed fixes to our RabbitMQ CDC connector, and went on tour to Amsterdam for the third edition of Systems Distributed!

Let’s go!

Spending time to get things exactly right is worth it, because the code is an asset, it will pay for itself many times over during its lifetime. matklad

A key property of a replicated system is its ability to mask failures. Specifically, a replicated system must provide safety and liveness in the presence of fail stop failures (where replicas fail permanently), and gray failures (where replicas exhibit degraded performance or intermittent errors), so long as the number of failures are within the system’s fault tolerance limits. In practice, the system must also guard against subtle metastability failures, wherein the root cause is not a specific hardware failure or a software bug, but emergent behaviour that causes self-sustaining congestive collapse in response to a transient stressor (e.g. a load surge).
In June, we made TigerBeetle more resilient to metastability failures! Typically, a TigerBeetle cluster makes progress at the pace of the fastest quorum of replicas. The other replicas may lag a bit, but eventually catch up with the rest of the cluster during quieter periods. Before this enhancement, if the cluster were under high load, lagging replicas would lag more and more, fall into state sync, and ultimately enter a potentially perpetual state of lag. Now, the primary applies backpressure to the client, which in turn backpressures the application, allowing lagging replicas to catch up with the rest of the cluster.
For safety, replicas undergoing state sync withhold prepare_ok messages while they are syncing table data blocks from the other replicas. The VOPR (our deterministic simulator) discovered that replicas were being too conservative, also withholding these messages while syncing client replies , which are always safe to overwrite. We fixed this by relaxing the conditions under which the replicas withhold prepare_ok messages!
TigerBeetle metrics provide fine-grained observability into the state of the cluster - for example, what commit, view, operation number, checkpoint the replica is on. Recently, we added an important coarse-grained metric, the end-to-end client request latency, to measure how much latency the application actually observes! Additionally, we added counters for the different types of messages that a replica sends and receives (for example, prepare, prepare_ok, commit, etc.) over the network.
We fixed a bug found by Antithesis, wherein our MessageBus wasn’t handling the ECANCELED error, when shutdown is invoked concurrently with send.
We landed various fixes to TigerBeetle’s Change Data Capture (CDC) connector for streaming transfers and balance updates to AMQP-compatible message queues.
TigerBeetle CDC now supports streaming transfers created before version 0.16.29, provided both the debit and credit accounts have the history flag! Earlier, replicas would crash upon encountering objects created with the old schema.
We fixed an overzealous assertion in the AMQP connector that overlooked the possibility of receiving an asynchronous basic.ack from RabbitMQ, whilst publishing a batch of messages.
Additionally, we improved compatibility with the AMQP specification by adding support for signed integer types. While this new encoding is handled transparently by RabbitMQ/AMQP clients, code changes might be necessary if consumers explicitly rely on the unsigned data types.
Finally, we fixed a crash in the CDC runner, by virtue of the metrics being too coarse-grained. Specifically, we were capturing metrics with millisecond precision, even though we were publishing events at a sub-millisecond scale! We now capture events with nanosecond precision.
We also got to the bottom of a mysterious SIGSEGV on our fuzzing infrastructure, one without any trace of a stack overflow! Specifically, we observed that despite our fuzzer using only half a megabyte of the stack, a heap allocation was being memory mapped into the stack region. With the help of a friend from the Zig community, we discovered that Zig’s PageAllocator uses mmap with an address hint (a preference for which address to use in the virtual address space). It turns out this is inherently incompatible with the way Linux’s memory manager manages the stack and the heap growth. Read this for a masterclass in systems debugging!
Finally, we landed some enhancements to our benchmark and CI infrastructure. Less is more , so we condensed the output of our benchmark to only print key latencies: the p1, p50, p99, and p100 (as opposed to a much higher granularity output). Additionally, we reduced the number of entry points in our GitHub CI checks. Finally, --transfer-batch-delay-us (the time that a client waits before sending the next request) now uses an async timeout, making it compatible with --clients. Previously, users would have to spin up two separate benchmark processes to achieve this!

Kirill helped us realize that a TODO to perform exponential backoff for client to replica pings was unnecessary, since those pings are very rare. The best code is indeed no code at all… Thank you for your contribution, Kirill!
Shoutout to rbino and alvinyan-bond for taking CDC for a spin and reporting issues, your feedback is helping us make it rock solid.

Last month, we walked through workload generation for the VOPR, and squashed another VOPR bug in realtime… or rather, in simulated time, flowing backwards! We also discussed how we use our deterministic simulation infrastructure to write unit tests for TigerBeetle’s consensus, and live-debugged a performance issue using deterministic performance testing.

Join us live every week on Twitch or catch up on the TigerTube!

Boredom Over Beauty: Why Code Quality is Code Security (June 3) Some of the most devastating vulnerabilities stem from complexity. This post by John Saigle, referencing TigerStyle, explains why predictable, well-formed code is the foundation of security.

Jepsen Report, TigerBeetle 0.16.11 & Fuzzer Blind Spots (June 6) Jepsen found a correctness bug in what appeared to be a well-fuzzed component of the TigerBeetle code: the query engine. Read the report, as well as our companion post. Thank you, Kyle for the collaboration!

Systems Distributed ’25 (June 19-20) It was our delight to present the third edition of Systems Distributed in Amsterdam last month. Over two hundred participants from around the world enjoyed cinematic experiences, stroopwafels, and sparkling IJ river scenes. SD25 online launches on YouTube next week , including live discussion with each speaker. Do join us!

Deterministic upgrades with multiversion binaries at JOTB25 (June 26) Federico Massimiliano Lorenzi’s J on The Beach talk is available! Dive into the nitty gritty of how TigerBeetle handles cluster upgrades safely, by bundling multiple releases into a single binary trenchcoat. Bellissimo!

SD25 online starts next week! Systems Distributed talks will be released as online premieres twice weekly on YouTube from July 21 into September. See you at 10am PT / 7pm CET on Mondays and Wednesdays for live chat and discussion, including the speakers.

‘Till next time… la vita è musica e fantasia!

The TigerBeetle Team