May in TigerLand

    Dear friends,

    Hope your May was majestic! We improved latency and throughput during repair, abstracted the physical, non-deterministic interface for AOF, and landed our very first CDC connector targeting RabbitMQ. We also walked the Caminito Del Ray, beat jet lag as the #AnimalDatabaseAlliance, and celebrated just one month until Systems Distributed in Amsterdam!

    Keep an eye on our blog today…

    Let’s go!

    _ “No man is an island, Entire of itself, Every man is a piece of the continent, A part of the main.__”_ John Donne

    • As a financial transactions database, TigerBeetle is designed to serve as the system of record and source of truth for transactional data, since it excels * at handling OLTP workloads, which necessarily have high “power law” contention. However, for the entire application to function, other databases must coexist, sometimes handling complementary data (e.g., relational OLGP databases, document and object storage) or mirroring different views of the same transactions committed to TigerBeetle (e.g., OLAP databases, data warehouses, legacy systems, and event-driven workflows).

    One pattern for interconnecting different systems is Change Data Capture (CDC). A technique where the source database tracks and streams only the changes (such as inserts, updates, and deletes) as they happen. Since financial transactions are immutable and changes are append-only (with no updates or deletions), CDC is particularly efficient for TigerBeetle to process.

    We landed our very first CDC connector targeting RabbitMQ, a popular open-source message broker widely adopted across many industries (notably in the financial sector). *Pun intended, but yes, users can now use TigerBeetle’s builtin CDC to ultimately feed their favorite spreadsheets.

    • TigerBeetle includes a RabbitMQ client bundled with the main executable, compatible with both the latest RabbitMQ version and the older 3.x releases, as well as other message brokers that support the AMQP 0.9.1 protocol.

    The CDC job runs as a stateless process, separate from the database, that polls the TigerBeetle cluster for incremental changes and publishes them to RabbitMQ as they occur.

    • Like everything else in TigerBeetle, the RabbitMQ client implementation follows TigerStyle and also statically allocates all the memory it will ever need up front. It is built using TigerBeetle’s asynchronous I/O interface, processing concurrent network requests in a single-threaded process. While publishing to RabbitMQ, it fetches the next batch of event changes from the TigerBeetle cluster and then swaps buffers in a producer-consumer fashion.

    For more details on setting up the CDC job, please refer to the documentation.

    • This month, we also focused on making TigerBeetle’s repair protocol more efficient, introducing a collection of features and fixes that not only makes repairs faster but benefits replication in general.

    • We implemented an adaptive replication routing protocol that detects and selects the most responsive replica, taking into account changes in network topology (such as partitions, latency and packet loss) as well as slow or crashed replicas. The new replication topology is more efficient because the primary does not rely on retries and absolute timeouts to replace a failing replica.

    • When a failing replica recovers, it needs to catch up with the rest of the cluster, either by replaying requests (if it isn’t too far behind) or by requesting data blocks from other replicas that contain the missing state. There was low-hanging fruit we could optimize here. First, we increase the maximum write IOPS, as we were inadvertently throttling repairs by not allocating enough resources. Then, we avoid backpressure by preventing the recovering replica from overwhelming healthy ones by limiting the number of missing blocks it can request for each replica.

    • Another interesting effect during recovery in a cluster under high load is that a lagging replica would request a missing block that was already obsolete by the time a healthy replica received the request. This led to two consequences: many repair requests were dropped, and the recovering replica remained indefinitely behind the cluster, unless new requests ceased to come in. To solve this problem, we now advertise whenever a checkpoint is made durable, so that lagging replicas can proactively sync to this durable checkpoint.

    • Designing away the problem of not having enough IOPS allocated, we redesigned the MessageBus abstraction to allow the replica to pull (and select) from the list of received messages rather than pushing events to the replica. This approach enables the replica to “suspend” certain messages and return to them later when it has enough IOPs, avoiding potential retries caused by messages being dropped due to insufficient resources.

    • We also introduced a new CLI subcommand, tigerbeetle recover, to safely format an empty file and recover a replica that has been permanently lost (e.g., if the SSD fails).

    • TigerBeetle’s release is now a two-step process! We moved the release cut to every Friday, spending a weekend on the CFO fleet (Continuous Fuzzing Orchestrator). In other words, you’ll still be getting a new release every Monday, but the date of the release will be set to Friday. This setup allows extra time for fuzzers to find problems in the specific commit we are trying to release.

    Can you run your distributed system on a single laptop? In a single process? On a single thread?! Last month in IronBeetle, we started out with logical and physical DST, live-coded the third incarnation of tickless [VOPR](https://github.com/tigerbeetle/tigerbeetle/pull/2956, as live-coded on Twitch!), and covered fault injection, including near-byzantine disk. Finally, by rubberducking, we discovered a missing assert in our state checker!

    In Brian’s TigerBeetle Stream, an important milestone: Brian began the test suite for the TigerBeetle Rust client, learning some Rusty testing and concurrency techniques along the way, and then landing the initial Rust client PR! Final steps for the project were discussed. Kudos to Brian for everything invested into 100% live coding this client on Twitch.

    Join us live every week on Twitch or catch up on the TigerTube!

    J on The Beach (May 14-16) We were proud to attend the ninth edition of this splendid event, which included a walk along the Caminito Del Ray for Federico Massimiliano Lorenzi! Catch the recap.

    Elixir Conf (May 15-16) At Elixir Conf in Krakow, Riccardo Binetti unveiled live how his tigerbeetlex client fuses Elixir and Zig for blazing-fast financial transactions! From BEAM internals to NIFs and clean APIs, a full-stack performance boost… we’re looking out for the recording!

    New blog post: Asserting Implications (May 26) Most programming languages don’t have special syntax for implication, as you don’t often branch based on implications. Our latest post was about asserting implications.

    Systems Distributed (June 19-20) It’s two weeks till we meet in Amsterdam for the third edition of Systems Distributed. We’re ready and set for talks, learning, and spending time together with you. If you’re still considering joining us, some final tickets remain! And if you can’t join us this time, SD talks are (always) published on YouTube (we’ll put them up as soon as we can, once post-production is done after the conference).

    Tweet Tweet Tweet Tweet Tweet Tweet Tweet Tweet Tweet

    ‘Till next time… take me down to the Paradise City!

    The TigerBeetle Team

    RSS iconRSS
    An idling tiger beetle Speech bubble says hi