August in TigerLand
Dear friends,
Hope you had a super August! This month, we premiered the Systems Distributed ’24 talks on YouTube, landed several major new database features, optimized state sync performance by an order of magnitude, and took SimTigerBeetle into the classroom to teach the next generation of computer scientists.
Let’s go!
This month, we introduced some exciting new features to TigerBeetle, inspired by some interesting use cases shared by users: 🎉
- Transfers between accounts used to require the amount to be greater than zero. However, we learned that there are use cases where a business transaction consisting of multiple line items (i.e. multiple transfers) may have one of these line items set to zero (e.g. booking a ride share trip, with the city tax line item set to zero).
Starting from version 0.16.0, TigerBeetle will therefore accept zero-amount transfers, introducing breaking changes for clients (not replicas) in situations where past versions of TigerBeetle interpreted zero as a sentinel value:
- Two-phase transfers allow the application to reserve a certain amount through a pending transfer that can later be posted (moving an amount less than or equal to the original reserved amount) or voided (reverting the reserved amount). Previously, when posting a pending transfer, amount=0 could be used as a shortcut to say “post the full pending amount “, so that the application could send a transfer with the amount zeroed to confirm the pending transfer without having to know (or query) the original amount.
Now, zero simply means “post zero “, since zero can be a meaningful amount.
To achieve the previous behavior for posting the total of the pending amount without knowing the value, we introduced a new sentinel value defined by the constant AMOUNT_MAX, which does what it says: “ post the maximum amount ”.
Kudos to Phil Davies for suggesting this approach on our Community Slack!
- Balancing transfers allow the application to create what is known in accounting as a closing entry—a transfer that moves the net balance of one account to another, effectively equalizing debits and credits in the account. This can be useful where the net balance needs to be calculated e.g. when generating a monthly invoice or at the end of a fiscal period.
Previously, when balancing an account, if debits and credits were already equalized, a balancing transfer with amount=0 would fail because zero-amount transfers were not allowed!
Now, the application can express and record the intent to balance any account, even if there’s no net balance to move.
The breaking API change here is that applications could previously use amount=0 as an implicit sentinel value to request balancing “as much as possible “. Similar to two-phase transfers, we now instead use the new sentinel value, AMOUNT_MAX, to express this behavior explicitly.
- When upgrading an existing cluster to TigerBeetle >=0.16.0, be sure to review if your application uses zeroed amounts for any of these features, and if so, update your application to use the new AMOUNT_MAX sentinel instead.
Subscribe to this tracking issue to stay informed about breaking changes that may affect clients!
- Speaking of closing entries , it’s also common practice at the end of a period, upon account termination, or for audit purposes to “freeze” an account, and prevent any further movements. TigerBeetle now supports this with a new feature that allows applications to close an account, making it easier to express and enforce accounting policies clearly.
Accounts can be closed through a special two-phase transfer, marked with the new closing_debit and closing_credit flags. This ensures that once an account is closed, it rejects any new transfers, except for voiding or expiring existing pending transfers. By making closing transfers pending, we ensure that accounts can be unclosed anytime, with an auditable reversal operation tied directly to the original closing transfer — safeguarding against undesired locks and any unexpected close-and-unclose cycles.
These primitives can be combined in even more interesting use cases. For instance, an application can “suspend” an account for a specified period of time by setting a timeout on the closing transfer, which will automatically restore the account when the timeout expires!
Check out the documentation for our new recipe on closing accounts. 📖
- Another exciting feature introduced in 0.16.0 is the ability to provide a user-defined timestamp for Transfers and Accounts. This is crucial for importing historical events into TigerBeetle while preserving the original timestamp.
With the new imported flag, users can now specify that TigerBeetle should not use the cluster’s time as the timestamp. Instead, the timestamp provided by the application is validated against the following invariants:
The user-defined timestamp must be a 64-bit integer representing the number of nanoseconds since UNIX epoch.
It must be a past date, and never ahead of the cluster clock at the time the request arrives at the cluster.
It must be unique across the cluster (no two events may share the same timestamp).
It must be strictly increasing in relation to the last timestamp assigned to events of the same type (Account or Transfer).
Accounts must always predate Transfers.
Even though imported events are inherently past events , these rules ensure thattheir effects cannot alter transactions already recorded. This preserves the integrity of the history, with no retroactive changes that could impact the current state.
For more information, please refer to our documentation on importing accounts and transfers, along with the samples for Java, C#, Go, and Node.js. 📚
To prepare the query engine for these new features, we added support for indexing zeroed values, which were previously treated as optional indexes. Additionally, we introduced the ability to index a single bit in a flag, to allow querying by the new imported and closed flags.
We rolled out a new state sync protocol that significantly improves liveness in various scenarios discovered during deterministic simulation testing! We recommend reading matklad’s excellent writeup for the details, but to summarize: the original state sync protocol was a separate protocol, orthogonal to the VSR protocol. While VSR handled replicating the log of operations across a quorum of replicas, the state sync protocol was responsible for repairing and updating a lagging replica’s storage.
Although this approach was elegant, it led to suboptimal liveness due to inconsistencies between the log and checkpoint, requiring constant patches to address various manifestations of the same problem. The new protocol fully integrates, to make VSR aware of the checkpoint and solve the root cause.
- To quote Tolkien: “ One does not simply change the VSR protocol ”! 🧙♂️
In order to introduce these changes for existing clusters without disturbing clients and replicas during the upgrade, we needed a strategy:
1. In version 0.15.5, replicas support both protocols, enabling old versions to talk with the new ones during rolling upgrades.
2. In version 0.15.6, the new protocol was effectively introduced.
3. In version 0.16.0, the old protocol is deprecated.
TigerBeetle’s upgrade protocol takes care of this upgrade path automatically, requiring all replicas of the cluster to “visit” the intermediate version before upgrading to newer versions. This ensures that the VSR protocol change is transparent to both clients and the cluster operator.
- Not only is the new protocol simpler, it’s also faster: another PR boosted state sync performance by an order of magnitude!
This means that syncing a lagging TigerBeetle replica is now almost as fast as moving the same amount of data through the network using tools like rsync and iperf, thanks to the same insight: TigerBeetle’s deterministic storage guarantee means that we need only transfer the missing data blocks, rather than blindly copying the entire file! ⚡
And once again, deterministic simulation testing saved countless hours of debugging by unraveling a complex set of conditions where a lagging replica would falsely contribute to the cluster’s durability by sending prepare_ok messages while still syncing its own storage! We fixed this latent bug that could lead to liveness issues in the presence of storage faults. 🤯
In last month’s newsletter, we introduced TigerBeetle’s new multiversion binary upgrade process. We’ve now completed the integration of multiversion binaries with the release infrastructure.
Starting with version 0.15.5, each new release includes everything needed to seamlessly trigger the upgrade process when it detects that the binary on disk has been updated to a newer version.
- To ensure that we don’t release anything with a broken upgrade process, we’ve added smoke tests to validate the full upgrade cycle for an entire cluster.
Clusters running 0.15.4 and 0.15.3 still require a slightly more involved upgrade procedure. Please refer to the upgrading documentation for more details!
- TigerBeetle often uses 128-bit unsigned integers in its API, as this versatile data type opens up a broad range of possibilities for applications. For example, it’s ideal for UUIDs, encoding custom user data, or recording account balances and transfer amounts. However, while many client platforms do not yet natively support this data type, users need to resort to alternative representations like BigIntegers and GUID/UUID.
We’ve therefore added sanity tests for coherence across all language implementations:
The in-memory representation of a 128-bit unsigned integer in Zig matches the decimal representation of the same value as a BigInteger in Java, C#, Go, and Node.js.
A little-endian 128-bit integer preserves its byte order when converted to/from a big-endian UUID in Java or a GUID in C#.
Thanks to James Kulubya for pointing out the endianness issue for C# GUIDs!
We’ve also made several developer experience improvements, including a simpler install script, a cleanup of the root directory (less is more!), and a series of code refactorings to streamline things.
These highlights also include updates to TIGERSTYLE, which now encourages composing multiple simple assertions rather than a single complex assert, to provide more information if a condition fails.
There’s also a recommended pattern for in-place initialization of large data types, which ensures pointer stability and immovable types while eliminating unnecessary copy-move allocations.
We kicked off the SD’24 live premieres on August 12, and are now halfway through. If you missed the in-person conference in NYC or want to rewatch your favorite speakers and engage with them in the live chat, be sure to add the upcoming premiere schedule to your calendar. All online premieres start at 10am PT / 1pm ET 7pm CET. See you online!
Every 730 days (or so!), something special takes place in California… as researchers from around the world, from universities such as Stanford, TU-Munich, UC Santa Cruz and Johns Hopkins University, and from companies such as Amazon, Google, Microsoft and Oracle, gather to workshop on High Performance Transaction Systems (HPTS).
HPTS is a storied conference, begun in 1985 by the legendary Jim Gray, and a conference almost as old as the field of Online Transaction Processing (or OLTP) itself.
This year, we’re delighted to share that TigerBeetle was invited to attend, and so Joran will be jumping on a jet plane to be there. We can’t wait to listen and learn from many of our heroes who will also be there (some of whom for their 20th time!).
This year, P99 Conf takes place online from October 23-24. Join as Matklad dives into the implementation of TigerBeetle’s just-in-time compaction algorithm for LSM-tree storage, using only static memory allocation and guaranteeing deterministically identical data files across all replicas for faster recovery. Register free today to reserve your spot!
We’re going to be doing our 2nd hackathon soon!
Join the TigerBeetle team at the Interledger Hackathon, taking place Saturday, October 19 and Sunday, October 20, to see how you can use Interledger, the Open Payments API, and Web Monetization to power payments and infrastructure for the future. Register here to take part.
The official Interledger Summit will also kick off the following weekend on October 26 at the Century City Conference Venue in Cape Town. Book your tickets here.
Till next time… keep it classical!
The TigerBeetle Team
To celebrate three years since Joy Machs drew the first TigerBeetle (left), Joy updated our little superhero (right) and shared the backstory of the process.
RSS
