August in TigerLand

    Dear friends,

    We hope your August was awesome. Last month, we launched TigerBeetle Cloud! We also wrote about protocol-aware deterministic simulation testing, improved our Ruby, Python, and Rust clients, and reworked our state machine unit tests. Systems Distributed ‘26 launched on YouTube, and we celebrated four trips around the sun as a company. Time flies!

    Let’s go!

    “All that is gold does not glitter,
    Not all those who wander are lost;
    The old that is strong does not wither,
    Deep roots are not reached by the frost.”
    — J.R.R. Tolkien

    Log-structured merge (LSM) trees are write-optimized data structures composed of an in-memory table and on-disk levels of exponentially increasing capacity. To amortize write IO, LSM trees first batch data in the in-memory table. When the in-memory table gets full, it is flushed to disk, and compacted down to lower levels as they get full.

    While designing LSM trees, it is crucial to measure amplification across three dimensions: read, write, and space. Read/write amplification is the ratio of IO performed to the IO required to serve a read/write operation. Space amplification is the ratio of disk space consumed to disk space required to store the data. The lower the amplification, the better.

    • In practice, it is infeasible for the in-memory table to always be full before it is written to disk. Specifically, it must be flushed when the write-ahead log (WAL) gets full, otherwise we risk data loss on crash (as the WAL starts recording new data, overwriting old data). In TigerBeetle, if an application submits tiny batches, this flush-on-checkpoint behavior would result in tiny on-disk tables and high space amplification. We alleviate this by greedily coalescing small tables on Level 0, ensuring tables are full by the time they’re compacted down to lower levels.
      • Last month, we discovered that while greedy coalescing improved space amplification, it resulted in high write amplification and disk bandwidth utilization, as we were reading, coalescing, and writing back the same data repeatedly. We fixed this by deferring table coalescing until Level 0 is full, merging all the tiny tables in a single pass.
    • TigerBeetle on macOS got ~7x faster, from ~30k TPS to ~200k TPS! The key insight was that LSM tree blocks needn’t be flushed to disk on every write; instead, flushing can be deferred to the end of compaction. On Linux and Windows, O_DSYNC is efficient enough to perform on every write, whereas macOS’s F_FULLFSYNC benefits from amortization via deferred flushing (since it requires the drive to flush all buffered data to disk).
    • While opening a TCP socket, TigerBeetle replicas configure the kernel send buffer and receive buffer size. Last month, we discovered that Linux could silently clamp these socket options to net.core.wmem_max / rmem_max, leading to mysterious performance degradation. Per TigerStyle, everything in TigerBeetle has a limit (including the send queue in our MessageBus), so the kernel send buffer being too small was causing TigerBeetle replicas to drop egress messages! We now verify that the socket options actually take effect, and warn the operator if clamped.

    Batching is central to getting the most out of TigerBeetle. Each TigerBeetle client allows for at most one in-flight request (for static allocation and to simplify state management). While waiting on the response for its in-flight request, the client automatically batches (auto-batches) operations of the same type. This is valuable for applications that can’t batch naturally, for instance a web server where each request is handled by a separate thread. Here, multiple threads can share a single TigerBeetle client and take advantage of auto-batching.

    • Last month, we fixed a bug in the prefetch of auto-batched create_transfers and create_accounts, where only the first operation of the batch was used to decide whether all operations were imported. So, if non-imported and imported operations were auto-batched, they were erroneously assumed to be all non-imported, causing the state machine to skip prefetching some records. Thank you to @shioyama for prompting this change!
    • This bug inspired us to rework our state machine unit tests; we now run every unit test with and without auto-batching, and added an example-based test for this scenario.
    • In addition, our client integration tests were revamped; they now use the real client instead of a mock EchoClient, which exposed the same types as the real client but didn’t exercise production behavior.

    Last month also saw improvements across our language clients ranging from defense-in-depth against overflows and supply-chain attacks to better code generation.

    • Our Ruby and Python clients now impose stricter integer checks. In the Python client, we discovered a silent u128 overflow in lookup_accounts and lookup_transfers, where an ID greater than (2128 - 1) was masked to a u128, causing a query for the wrong ID. In the Ruby client, rb_integer_pack silently truncated floats for any passed integer field. In both cases, we now error out.
    • TigerBeetle’s Ruby client is published using Trusted Publishing, where GitHub Actions authenticates with RubyGems.org using short-lived tokens. Unfortunately, the RubyGems API docs don’t specify the expected HTTP codes for their responses. To guard against potential supply-chain attacks, our HTTP client now omits logging the response for an unexpected code, as it may leak tokens.
    • Finally, the Rust client was refactored to remove duplicate type definitions and custom conversion functions by expanding our code generation.

    Each line of TigerBeetle code is handcrafted, independently reviewed, and understood by at least one other engineer. In the spirit of investing in our understanding, tidy (our tool to test various non-functional properties of the code) now fails on LLM-authored contributions, using commit metadata to detect LLM usage.

    Last month on IronBeetle’s ‘Pragmatics of Consensus’, Matklad and Tobi tackled the questions: How do backups construct JoinView messages? How does a prospective primary disambiguate between what is committed and what is uncommitted? Watch the View Change Pragmatics episode to learn! They also dove into State Sync, the subprotocol lagging replicas use to get up to date.

    And a bonus episode! On Zig Resilient Parser (ZIRP) syntax trees, a project that Matklad started at Zig Day in Boston.

    Join us live every Thursday at 5pm UTC on Twitch, YouTube, and X!

    Jonas Rocks A New Track Record In Round 5! Aug 5
    We’re nearing the finals of the North America Radical Cup! In the penultimate round, Jonas set a new lap record for a Radical at Lime Rock: 51.289s! Watching his overtake moment in Race 2 was a delight. Next up, COTA for Round 6 this weekend, and then the world finals in Barcelona! Jonas—champion!

    Watch Systems Distributed On YouTube! Aug 19
    The calibre of this year’s Systems Distributed speakers was remarkable, and we’re excited to share the talks more widely. During the first five premieres, released last month, we enjoyed hearing from just how far and wide you tuned in—Cape Town and California to Lisbon, Oslo, and Rio de Janeiro—and cherished the chance to chat to each speaker for Q&A in live chat.

    Benchmarking Performance Gains With Btrfs On Linux 7.3, Aug 25
    TigerBeetle was designed with predictable performance in mind. So, it’s always a pleasure when Phoronix uses TB as an application for benchmarking file-system performance! Last month, they published “absolutely wild” benchmarks for TigerBeetle atop Btrfs on Linux 7.3, with significant throughput and latency improvements compared to Btrfs on Linux 7.2.

    Protocol-Aware Deterministic Simulation Testing (DST), Aug 25
    TigerBeetle is known for DST, but what magic lies beyond? Last month, Chaitanya, who studied under the authors of the seminal “Protocol-Aware Recovery”, Ram Alagappan and Aishwarya Ganesan, wrote about pushing past “determinism” to “protocol-aware” DST, and the mechanics, methods, and merits of doing so. To dive deeper, you can also catch his talk on the same topic at Bug Bash (24 min) earlier this year. Kudos, Chaitanya!

    TigerBeetle Launches Cloud! Aug 28
    Financial-grade systems, trillion scale, high-stakes support, ultimate success. Last month, we announced TigerBeetle Cloud! Pricing and plans (Startup, Growth, Enterprise) are now globally available for the fastest transactions database, operated by its authors.

    Systems Distributed ‘26 On YouTube Continues! September
    For the remainder of this year’s lessons in systems thinking from Zooko Wilcox-O’Hearn, Dominik Tornow, Aishwarya Ganesan, Ram Alagappan, Hillel Wayne, Andrew Kelley, and James Cowling: join us on YouTube every Monday and Wednesday at 9am PT for the month of September. Not to forget: Lightning Talks! In which the quack-timer had to make some noise… Hit ‘notify me’ for YouTube to send you reminders. See you in the live chat!

    Georg Kreuzmayr At P99Conf (Online), Oct 21–22
    TigerBeetle needed object storage—the official SDKs didn’t compose with our completion-based event loop or DST—so we built it ourselves in Zig. The result is 4–10× faster than the official GCP and AWS SDKs, and in his talk at P99 Conf, Georg will cover the architecture, DST infrastructure, and how to apply TigerStyle to cloud I/O from first principles. Register for free.

    Tweet Tweet Tweet Tweet Tweet Tweet Tweet Tweet Tweet

    ’Till next time… it’s your life!

    The TigerBeetle Team

    An idling tiger beetle Speech bubble says hi