Data Replication
Murtadha Al Hubail
Project Team:
Motivation & Goals
NC 1
Cluster
Controller
NC2
NC3
•
AsterixDB typical cluster consists of a master node (Cluster Controller) and
multiple node controllers (NC).
•
Data is hashed partitioned across NCs based on primary key and stored only in
one node.
•
In case of NC failure, all data stored on that node is permanently lost.
•
Goal:
–
Implement an efficient data replication protocol to prevent permanent data loss.
Related Work
•
Synchronous Replication
–
All Replicas Identical
–
Two Phase Commit
–
Group Communication (Postgres-R)
•
Asynchronous Replication
–
Primary-Backups Model
–
Strong Consistency
•
Middleware Messaging Service
•
Publish/Subscribe (Yahoo’s PNUTS)
•
Distributed File System (Google’s Bigtable)
•
Using Paxos for consensus on failures (Google’s Spanner, Spinnaker)
–
Eventual Consistency
•
Amazon's Dynamo (Cassandra)
System Architecture
•
Each NC has a number of NCs designated as backup
nodes based on a replication factor (# of copies).
Primary and backup NCs based on replication factor = 3
System Architecture
•
Implement data replication by sending transactions REDO logs, and LSM
disk components to back NCs via Replication Channel over the network.
Transaction logs flow from NC1 to backups in NC2 and NC3
RMI vs. Socket Programming
Testing & Evaluation
1. Deploy AsterixDB on a cluster consisting for two or more nodes and
enable data replication.
2. Evaluate data replication performance impact compared to no
replication.
3. Disconnect a node from the cluster and erase all data to simulate
hardware failure.
4. Reconnect the failed node.
5. Check that all data has been restored.
Testing Results
•
Environment
–
2 Nodes Cluster
–
Gigabit Ethernet
–
Replication Factor = 2
•
Results:
With an average of %12.97 throughput decrease, two copies of the data could be
maintained.
The project aims to implement a robust data replication protocol to avoid permanent data loss in a clustered environment. By replicating data across multiple nodes and utilizing primary and backup nodes effectively, the system ensures data durability and availability in case of node failures. Various replication strategies and technologies are explored, and testing results indicate the impact of replication on system performance and data resilience.
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
You are allowed to download the files provided on this website for personal or commercial use, subject to the condition that they are used lawfully. All files are the property of their respective owners.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.
E N D
Presentation Transcript
Data Replication Project Team: Murtadha Al Hubail
Motivation & Goals AsterixDB typical cluster consists of a master node (Cluster Controller) and multiple node controllers (NC). Data is hashed partitioned across NCs based on primary key and stored only in one node. In case of NC failure, all data stored on that node is permanently lost. Goal: Implement an efficient data replication protocol to prevent permanent data loss. Cluster Controller NC 1 NC3 NC2
Related Work Synchronous Replication All Replicas Identical Two Phase Commit Group Communication (Postgres-R) Asynchronous Replication Primary-Backups Model Strong Consistency Middleware Messaging Service Publish/Subscribe (Yahoo s PNUTS) Distributed File System (Google s Bigtable) Using Paxos for consensus on failures (Google s Spanner, Spinnaker) Eventual Consistency Amazon's Dynamo (Cassandra)
System Architecture Each NC has a number of NCs designated as backup nodes based on a replication factor (# of copies). Primary and backup NCs based on replication factor = 3
System Architecture Implement data replication by sending transactions REDO logs, and LSM disk components to back NCs via Replication Channel over the network. Transaction logs flow from NC1 to backups in NC2 and NC3
Testing & Evaluation 1. Deploy AsterixDB on a cluster consisting for two or more nodes and enable data replication. 2. Evaluate data replication performance impact compared to no replication. 3. Disconnect a node from the cluster and erase all data to simulate hardware failure. 4. Reconnect the failed node. 5. Check that all data has been restored.
Testing Results Environment 2 Nodes Cluster Gigabit Ethernet Replication Factor = 2 Results: With an average of %12.97 throughput decrease, two copies of the data could be maintained.
