Understanding How Transactions Work in Cross Data Center Replications (XDCR)

Couchbase offers an impressive array of powerful tools and features within its platform services. Notably, Cross Data Centre Replication ensures seamless data replication across various geographies, while ACID transactions robustly support transaction workloads, enhancing both reliability and efficiency.

Many customers often encounter a common question: Why is there a difference in document counts between the source and destination clusters when using transactions and Cross Data Centre Replication (XDCR)? Certain types of documents never appear on the destination cluster, leading to confusion about whether this issue is related to XDCR. Before delving into the underlying mechanisms, let’s first clarify some key terms.

What is XDCR?

XDCR facilitates the replication of data between databases or buckets that may reside in different clusters, cloud providers, or data centres. XDCR also supports intra-cluster replication, allowing data replication between different databases within the same cluster.

Designed for geographically distributed database clusters, XDCR protects against data centre failures and supports high availability with active-active cluster configurations. The underlying protocol used by XDCR is the Data Change Protocol (DCP), which is also employed for intra-cluster replication, ensuring low-latency memory-to-memory replication.

XDCR offers unidirectional and bidirectional operations and supports active-active replication with automatic conflict resolution. It also allows filtered replication to replicate subsets of documents based on the target cluster’s needs.

What are Transactions?

A transaction is a single logical unit of work consisting of multiple database operations that either execute as a whole or not at all. Couchbase transactions enable ACID (atomic, consistent, isolated, and durable) actions on the database. Couchbase supports distributed multi-document, multi-node ACID transactions at scale without sacrificing performance and high availability.

What is an Active Transaction Record?

In Couchbase, data within a database or bucket is divided among logical containers called vbucket, each residing on a single node. Every Couchbase server bucket has 1024 vbucket (64 on MacOS). Active Transaction Records (ATR) are metadata documents on each vbucket that log every active transaction attempt, indicating whether an attempt has been committed. ATR entries serve as switches for marking transactions as committed. ATRs are created and maintained by Couchbase automatically and can be easily identified by their prefix _txn:atr-. These records can be viewed but should not be altered by users or applications.

How Does XDCR Replicate Transaction Documents?

In Couchbase, transactions are scoped to a single primary cluster only and transactions are not supported for active-active configuration. A transaction can involve multiple attempts, each creating an entry in an ATR document. These entries are crucial as single sources of truth for the attempts. ATRs reside in the default collection of the bucket of the first mutated document unless specified otherwise. Each collection used for ATRs will eventually contain 1,024 ATR documents. During the prepare phase, mutations within a transaction are staged in a document’s extended attributes (XATTRs) property, remaining invisible to the Couchbase cluster until the commit phase. The write intent in a transaction is specified in document XATTRs and it acts as a write lock, preventing other clients from modifying the same document until the transaction is either committed or aborted. These write intents function as locks exclusively for the primary cluster.

XDCR replicates data from the source to the destination cluster asynchronously, supporting eventual consistency for transactional updates. That’s why, a commit on the source cluster does not guarantee that the transaction has been replicated over XDCR. Once a transaction is committed on the source cluster, updates are replicated to the destination cluster one by one. This means a committed transaction on the source cluster does not guarantee immediate commitment on the destination cluster. In case of failover, a committed transaction can be lost if it does not commit to the destination before the failover, so applications must wait for all outstanding requests to complete or abort requests before failing over to secondary cluster.

Transaction Replication Steps

The following steps outline the transaction logic and data replication using XDCR:

1. 1. Transaction Attempt Initiation: Every transaction attempt by the application (SDK) creates an entry in the ATR, functioning as a virtual lock. Done on both nodes but shown on one in the picture below for simplicity. 
   2. Staging Changes: Transactional changes are staged in the XATTRs of the target documents, not affecting the document bodies. This can be across multiple nodes and documents. These stage changes act as a lock against any other transactions on these documents. 
   3. Commitment: Once the transaction logic executes completely, the transaction attempt is committed, updating the attempt entry in the ATR (done on both nodes but shown on one in the picture below for simplicity) and the list of document ids involved in the transaction is updated. Transactional actors can read the updated information from the XATTRs if needed.
   4. Finalizing Changes: Transactional changes are moved from document XATTRs to document bodies (done by the SDK but shown directly in the picture below for simplicity)
   5. Completion and Cleanup: The transaction attempt is marked as ‘Completed’ and removed from the ATR.
   6. Replication: The newly updated document changes are replicated one by one to the target cluster using XDCR.

Conclusion

XDCR is a powerful tool for replication across different data centres and geographies, supporting both unidirectional and bidirectional replication with eventual consistency for transactional changes. By design, uncommitted changes in a transaction and metadata for recording transactions are never sent to target clusters, ensuring data integrity and consistency across replicated environments.

Understanding these mechanisms can help clarify why certain documents might not appear in the destination cluster immediately, as they rely on the transactional states and replication processes described above.