Framework for One-sided RDMA Multicast in Data Center Group Communications

This paper explores the implementation of one-sided RDMA multicast in data center group communications, highlighting the benefits of offloading network I/O from CPUs and unique connection parameters for efficient multicast operations. The research addresses the challenges of enabling one-sided RDMA multicast in fixed-function hardware setups, proposing solutions for enhanced performance in group communication patterns within data centers.

• RDMA
• Multicast
• Data Center
• Group Communications
• Networking

emch_rri Follow

Uploaded on Sep 15, 2024 | 0 Views

Download Presentation

Please find below an Image/Link to download the presentation.

The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. Download presentation by click this link. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.

Presentation Transcript

1. Towards a Framework for One-sided RDMA Multicast Xin Zhe Khooi Xin Zhe Khooi, Cha Hwan Song, Mun Choon Chan National University of Singapore EuroP4 2021 EuroP4 2021

2. Group Communications Receivers Sender Various applications exhibit group communication patterns in DCs: File system replication2 Distributed coordination3 Virtual tenant intra-networking4 Multicast offers efficient link utilization Multicast accelerates application performance5 R2 R1 Unicast R3 Multicast Receivers Multicast Sender R2 R1 [1] Mike McBride and Olufemi Komolafe. 2020. Multicast in the Data Center Overview. Internet-Draft draft-ietf-mboned-dc-deploy-09. Internet Engineering Task Force. https://datatracker.ietf.org/doc/html/draft-ietf-mboned-dc-deploy-09. Work in Progress. [2] Shvachko et al. 2010. The Hadoop Distributed File System. IEEE MSST. [3] Li et al. 2016. Just Say NO to Paxos Overhead: Replacing Consensus with Network Ordering. USENIX OSDI. [4] Shahbaz et al. 2019. Elmo: Source Routed Multicast for Public Clouds. ACM SIGCOMM. [5] Xia et al. 2015. Blast: Accelerating High-performance Data Analytics Applications by Optical Multicast. IEEE INFOCOM. Multicast R3 XZ Khooi et al. Towards a Framework for One-sided RDMA Multicast. EuroP4 2021. 2/11

3. Remote Direct Memory Access (RDMA) RDMA - offload network I/O from CPU1 Data center operators are shifting towards RDMA2,3 RDMA Operations One-sided: READ/ WRITE/ ATOMIC No CPU involvement, direct memory access Unique connection parameters per receiver, per connection <QPN, PSN, RKEY, RADDR> Multicast not supported Two-sided: SEND/ RECV Like other kernel bypasses, e.g., DPDK, XDP CPU actively involved Supports multicast App Buffer Can we have both zero-CPU overhead AND multicast support? OS TCP/IP RDMA NIC [1] Illustration adapted from https://forum.huawei.com/enterprise/en/whare-are-the-rdma-and-ib/thread/767731-891. [2] Guo et al. 2016. RDMA over Commodity Ethernet at Scale. ACM SIGCOMM. [3] Gao et al. 2021. When Cloud Storage Meets RDMA. USENIX NSDI. [4] RDMA Aware Networks Programming User Manual. https://www.mellanox.com/related-docs/prod_software/RDMA_Aware_Programming_user_manual.pdf XZ Khooi et al. Towards a Framework for One-sided RDMA Multicast. EuroP4 2021. 3/11

4. One-sided RDMA Unicast Unicast WRITEs To: Receiver A <QPNA, PSNA, RKEYA, RADDRA> To: Receiver B <QPNB, PSNB, RKEYB, RADDRB> Sender Receiver A Receiver B <QPNA, PSNA, RKEYA, RADDRA> <QPNB, PSNB, RKEYB, RADDRB> XZ Khooi et al. Towards a Framework for One-sided RDMA Multicast. EuroP4 2021. Connection setup & exchange parameters 4/11

5. One-sided RDMA Multicast Multicast WRITEs? Multicast Group Members X A, B To: Group X <QPNX, PSNX, RKEYX, RADDRX> How can we workaround the fixed function RDMA H/W to enable one-sided RDMA multicasts? I can only accept <QPNB, PSNB, RKEYB, RADDRB> Fixed Function Fixed Function Fixed Function Sender Receiver A Receiver B <QPNB, PSNB, RKEYB, RADDRB> <QPNA, PSNA, RKEYA, RADDRA> XZ Khooi et al. Towards a Framework for One-sided RDMA Multicast. EuroP4 2021. 5/11

6. Augmenting Fixed Function RDMA H/W with P4 Multicast Group Members Group Receiver Connection Params X A, B X A <QPNA, PSNA, RKEYA, RADDRA> X B <QPNB, PSNB, RKEYB, RADDRB> To: Group X To: To: Receiver A Receiver B <QPNX, PSNX, RKEYX, RADDRX> <QPNA, PSNA, RKEYA, RADDRA> <QPNB, PSNB, RKEYB, RADDRB> Sender Receiver A Receiver B <QPNB, PSNB, RKEYB, RADDRB> <QPNA, PSNA, RKEYA, RADDRA> XZ Khooi et al. Towards a Framework for One-sided RDMA Multicast. EuroP4 2021. 6/11

7. A Framework for One-sided RDMA Multicast 3. Mapping Rule Configuration 1. Multicast Transfer Request 2. Get Connection Parameters 4. Clear to send! Sender Receiver A Receiver B <QPNA, PSNA, RKEYA, RADDRA> <QPNB, PSNB, RKEYB, RADDRB> XZ Khooi et al. Towards a Framework for One-sided RDMA Multicast. EuroP4 2021. 7/11

8. A Framework for One-sided RDMA Multicast 7. Teardown Session 6. Done! 5. Multicast Transfer Sender Receiver A Receiver B <QPNA, PSNA, RKEYA, RADDRA> <QPNB, PSNB, RKEYB, RADDRB> XZ Khooi et al. Towards a Framework for One-sided RDMA Multicast. EuroP4 2021. 8/11

9. RDMA NICs with P4 Programmable Pipelines1 I will do multicast only! We can do the header Memory is scarce, and finite mapping! P4 programmable packet processing engines Sender Receiver A Receiver B [1] Mellanox BlueField2 DPU SmartNICs. https://store.mellanox.com/categories/dpu.htm XZ Khooi et al. Towards a Framework for One-sided RDMA Multicast. EuroP4 2021. 9/11

10. Future Directions/ Challenges One-sided RDMA Multicast-as-a Service API Interaction between Multicast and PFC How to prevent PFC deadlocks1? Multicast Congestion Control How to handle packet retransmissions? How to adapt the transmission rate accordingly? Integration with Scalable Multicast Routing Source routing2? [1] Guo et al. 2016. RDMA over Commodity Ethernet at Scale. ACM SIGCOMM. [2] Shahbaz et al. 2019. Elmo: Source Routed Multicast for Public Clouds. ACM SIGCOMM. XZ Khooi et al. Towards a Framework for One-sided RDMA Multicast. EuroP4 2021. 10/11

11. Questions? khooixz (at) comp (dot) nus (dot) edu (dot) sg * This slide deck utilizes icons from flaticon.com. XZ Khooi et al. Towards a Framework for One-sided RDMA Multicast. EuroP4 2021. 11/11