ACID Transactions Are Hard At Scale … Part 2

K.S. Bhaskar

… But YottaDB Does the Hard Work for You

Part 1 discussed what ACID transactions are and why they are hard to scale. This post discusses how YottaDB does the hard work for you.

Concurrency Control

At a very high level, there are two types of concurrency control: pessimistic (known as locking) and optimistic.

In pessimistic concurrency control, as the transaction logic executes, it “locks” the data it accesses. To ensure fully ACID properties, this lock must prevent concurrent processes executing their transaction logic from both reading as well as writing the data. Reading is blocked so that the other processes do not see the intermediate states of data. The consequent performance loss has led databases to provide transaction variants that are not fully ACID, such as multiversion concurrency control (MVCC; also known as “stable reads”) where data a process has read within a transaction will not change for that process as a result of a commit by another transaction. While this suffices for some types of business logic (in the balance transfer example, if there is a concurrent change to a service charge, it is probably acceptable for the transaction to use the prior service charge), it is unacceptable for others (such as two concurrent transactions withdrawing funds from the same account).

In optimistic concurrency control (OCC), a process executing a transaction keeps track of the data that it reads, but does not update the database until the time comes to commit the transaction. If no data that it has read has changed (data that it intends to update must first be read), it commits the transaction; if any data has changed (“collided”), it starts over from the beginning and re-attempts the transaction. Optimistic concurrency control when implemented in hardware, is called transactional memory. Software transactional memory also exists.

The primary pitfall of pessimistic concurrency control is deadlock: if process 1 locks data A and process 2 locks data B, then as they execute program logic process 1 finds it needs B and process 2 finds that it needs A. Deadlocks can of course be more complex: for example, A needs a resource that B has locked, B has a resource that C has locked, and C has a resource that A has locked. In a complex environment where there might be thousands or tens of thousands of concurrent transactions, there might be a deadlock involving a few transactions. The system is busy, and the application is making progress overall, but those few transactions are mutually unable to progress further. Much research has gone into detecting and preventing deadlock, but the problem remains a hard one to solve at scale. We know of at least one database that provides Atomicity and Durability, but makes application software responsible for Consistency and Isolation.

The primary pitfall of optimistic concurrency control is livelock: if two processes update the same data, one of them will update it first. The other will detect a collision and retry the transaction, thus executing the transaction logic twice. At scale, the system is busy, but wasting resources because processes are executing transaction logic multiple times for each successful commit; at least they are not deadlocked.

While most databases pessimistic concurrency control and ameliorate conflicts with MVCC, MVCC is unacceptable for real-time core-banking systems. YottaDB uses OCC with mechanisms to detect and limit livelock.

YottaDB Concurrency Control

Every database region has a transaction number that is incremented on every update.¹ Every database block that is updated during a transaction has its transaction number set to the transaction number of the region, which is incremented in anticipation of the next transaction.

A process inside a transaction notes the transaction number of every database block it reads (a block must be read before it can be updated). Updates are kept in process-private memory. When the process commits the transaction, it checks whether one or more blocks it has read were updated since it read them, and if no block has changed, it commits the transaction. If even one block has changed, it restarts the transaction.

To prevent persistent livelock, if a transaction is forced to restart the transaction thrice, on the fourth attempt it locks other processes from updating the database region(s) in the transaction, and completes the transaction.

As the probability of multiple concurrent transactions updating the same blocks is low, this works well in practice. Nevertheless, YottaDB has mechanisms for applications to detect and avoid livelock. Each database region has a count of the number of first, second, third, and subsequent² transaction restarts. In the typical scenario of random collisions, one would see a low rate of first restarts, a lower rate of second restarts, and virtually no third restarts. A pattern other than this means that there are pathological restarts resulting from application design that creates “hot” data areas where concurrent transactions update the same data. YottaDB has tools to enable application developers to identify such hot data areas, so that they can avoid pathological restarts.

To avoid problems such as that described in Vendor’s secret ‘fix’ made critical app unusable during business hours, YottaDB also provides for transaction timeouts: a transaction that exceeds the specified time will abort the transaction with an error. Since the transaction will not have been committed, no other concurrent transaction is affected.

YottaDB’s OCC has additional benefits over locking:

• It is computationally easier to restart a transaction that has made no updates than to roll back a transaction that has.
• It is hard for pessimistic concurrency control to lock data that does not exist. With YottaDB’s OCC index blocks are part of the read set of a process executing a transaction. If another process adds a record when a process is inside a transaction, an index block will have a new transaction number, resulting in a restart. In other words, YottaDB transactions ensure Consistency and Isolation not just for the existence of data but also for the absence of data.
• Programming ACID transactions in YottaDB is extremely simple. For example:
  • There is no need to WATCH for changed data. YottaDB does the watching automatically with no need for application program action.
  • YottaDB automatically rolls back and restarts transactions, carrying them through to being committed, with no explicit programming required on the part of the application.

Yes, but does it really scale?

The proof of the pudding, as they say, is in the eating. YottaDB, and its upstream code base, GT.M (whose transaction processing implementation YottaDB inherits and improves), are in daily production at the world’s largest real-time core-banking systems, with tens of millions of accounts, and peak ACID transaction rates of tens of thousands of transactions per second.

YottaDB does the hard work for you by providing a clean, simple ACID transaction APIs in a variety of languages.

Contact us to discuss what YottaDB can do for your transaction processing application.

Footnotes

¹ YottaDB treats single updates as “mini transactions,” which simplifies both explaining and implementing how the database engine works.

² There are circumstances under which a transaction can restart more than three times. But that would be getting lost in the weeds for the level of detail of this blog post.

Credits

1. Photo of ATMs in Croatia by Gewild who dedicated this work to the public domain.
2. Photo of savings bank box of Second National Bank, Allentown PA is in the public domain.

Published on January 28, 2026

See All General