Exploring Wireless Technologies for Datacenters
Alternative Switching Technologies:
Wireless Datacenters
Hakim Weatherspoon
Assistant Professor, Dept of Computer Science
CS 5413: High Performance Systems and Networking
October 22, 2014
Slides from the “On the Feasibility of Completely Wireless Datacenters” at the ACM/IEEE
Symposium on Architectures for Networking and Communications Systems (ANCS),
October 2012.
•
Overview and Basics
•
Data Center Networks
–
Basic switching technologies
–
Data Center Network Topologies (today and Monday)
–
Software Routers (eg. Click, Routebricks, NetMap, Netslice)
–
Alternative Switching Technologies
–
Data Center Transport
•
Data Center Software Networking
–
Software Defined networking (overview, control plane, data
plane, NetFGPA)
–
Data Center Traffic and Measurements
–
Virtualizing Networks
–
Middleboxes
•
Advanced Topics
Where are we in the semester?
Goals for Today
•
On the Feasibility of Completely Wireless
Datacenters
–
J. Y. Shin, E. G. Sirer, H. Weatherspoon, and D. Kirovski,
IEEE/ACM Transactions on Networking (ToN)
, Volume
21, Issue 5 (October 2013), pages 1666-1680.
Conventional Datacenter
…
Going Completely Wireless
•
Opportunities
–
Low maintenance : no wires
–
Low power: no large switches
–
Low cost: all of the above
–
Fault tolerant: multiple network paths
–
High performance: multiple network paths
Which wireless technology?
60GHz Wireless Technology
•
Short range
–
Attenuated by oxygen
molecules
•
Directional
–
Narrow beam
•
High bandwidth
–
Several to over 10Gbps
•
License free
–
Has been available for
many years
6
Why now?
•
CMOS Integration
-
Size < dime
-
Manufacturing cost < $1
60 GHz Antenna Model
•
One directional
–
Signal angle between
25° and 45°
–
Maximum range < 10 m
–
No beam steering
•
Bandwidth < 15Gbps
–
TDMA (TDD)
–
FDMA (FDD)
•
Power at 0.1 – 0.3W
How to integrate to datacenters?
Designing Wireless Datacenters
•
Challenges
–
How should transceivers and racks be oriented?
–
How should the network be architected?
–
Interference of densely populated transceivers?
Completely Wireless Datacenters
•
Motivation
•
Cayley
Wireless Datacenters
–
Transceiver placement and topology
•
Server and rack designs
–
Network architecture
•
MAC protocols and routing
•
Evaluation
–
Physical Validation: Interference measurements
–
Performance and power
•
Future
•
Conclusion
Transceiver Placement:
Server and Rack Design
•
Rack
•
Server
3D View
How do racks communicate with each other?
Cayley Network Architecture:
Topology
Masked Node Problem and MAC
•
Most nodes are hidden terminals to others
–
Multiple (>5) directional antennae
=> Masked node problem
–
Collisions can occur
•
Dual busy tone multiple access [Hass’02]
–
Out of band tone to preserve channels
–
Use of FDD/TDD channels as the tone
12
Cayley Network Architecture: Routing
•
Geographical Routing
•
Inter rack
–
Diagonal XYZ routing
•
Turn within rack
–
Shortest path turning
•
Within dst rack to dst
server
–
Up down to dst story
–
Shortest path to dst server
Completely Wireless Datacenters
•
Motivation
•
Cayley
Wireless Datacenters
–
Transceiver placement and topology
•
Server and rack designs
–
Network architecture
•
MAC protocols and routing
•
Evaluation
–
Physical validation: Interference measurements
–
Performance and power
•
Future
•
Conclusion
Hardware Setup for Physical
Validation
•
Use of a conservative platform
•
Real-size datacenter floor plan setup
•
Validation of all possible interferences
Intra-rack communications
Inter-rack communications
Physical Validation: Interference Evaluation
(Signal angle
θ
= 15° )
Physical Validation: Interference Evaluation
(Signal angle
θ
= 15° )
Evaluation
•
Performance
: How well does a Cayley datacenter
perform and scale?
–
Bandwidth and latency
•
Failure tolerance
: How well can a Cayley
datacenter handle failures?
–
Server, story, and rack failure
•
Power
: How much power does a Cayley
datacenter consume compared to wired
datacenters
•
Simulate 10K server datacenter
–
Packet level: routing, MAC protocol, switching delay, bandwidth
•
Conventional datacenter (CDC)
–
3 Layers of oversubscribed switches (ToR, AS, CS)
•
(1, 5, 1), (1, 7, 1) and (2, 5, 1)
•
Latency: 3-6us switching delay
•
Bandwidth: 1Gbps server
•
FAT-tree: Equivalent to CDC (1,1,1)
•
Cayley wireless datacenter
–
10Gbps bandwidth
–
1 Transceiver covers 7 to 8 others
–
Signal spreading angle of 25°
–
Low latency Y-switch (<< 1us)
Evaluation Setup
Top of Rack
Aggregate
Core
Evaluation Setup
•
Uniform random
–
Src and dst randomly selected in entire datacenter
•
MapReduce
–
Src sends msg to servers in same row of rack
–
Receiver sends msg to servers in same column of rack
–
Receivers send msg to servers inside same pod with
50% probability
Cayley datacenters have the most bandwidth
Bandwidth
s
•
Burst of 500 x 1KB packets per server sent
Latency
•
Uniform random benchmark
•
MapReduce benchmark
Cayley datacenters typically performs the best
Fault Tolerance
Cayley datacenters are extremely fault tolerant
Power Consumption to Connect 10K Servers
•
Conventional datacenter (CDC) *
–
Depending on the oversubscription rate
58KW to 72KW
•
Cayley datacenter
–
Transceivers consume < 0.3W
–
Maximum power consumption:
6KW
•
Less than 1/10 of CDC power consumption
* Cost and spec of Cisco 4000, 5000, 7000 series switches
Discussion and Future Work
•
Only scratched the surface
–
How far can wireless datacenters go with no wires?
•
Need larger experiment/testbed
–
Interference and performance of densely connected
datacenter?
•
Scaling to large datacenters (>100K servers)?
•
Scaling to higher bandwidth (> 10Gbps)?
Conclusion
•
Completely wireless datacenters
can be
feasible
•
Cayley wireless datacenters exhibit
–
Low maintenance
–
High performance
–
Fault tolerant
–
Low power
–
Low cost
References
•
S. Pinel, P. Sen, S. Sarkar, B. Perumana, D. Dawn, D.
Yeh, F. Barale, M. Leung, E. Juntunen, P. Vadivelu, K.
Chuang, P. Melet, G. Iyer, and J. Laskar. 60GHz single-
chip CMOS digital radios and phased array solutions
for gaming and connectivity. IEEE Journal on Selected
Areas in Communications, 27(8), 2009.
•
Z.J. Hass and J. Deng. Dual busy tone multiple access
(DBTMA)-a multiple access control scheme for ad hoc
networks. IEEE Transactions on Communications,
50(6), 2002.
•
PEPPM. Cisco Current Price List.
http://www.peppm.org/Products/cisco/price.pdf,
2012.
27
Related Work
Before
Next time
•
Project Interim report
–
Due Monday, October 27.
–
And meet with groups, TA, and professor
•
Lab3 – Packet filter/sniffer
–
Due
yesterday
, Tuesday, October 21. But, 24 hour grace period.
•
Lab1/2 redux
due Friday, October 24
•
Fractus Upgrade: SAVE ALL YOUR DATA
–
Fractus will be upgraded from October 28
th
to 30
th
–
Can use Red Cloud during upgrade period, then switch back to
F
ractus
•
Required review and reading for Wednesday, October 22
–
Data center TCP (DCTCP), M. Alizadeh, A. Greenberg, D. A. Maltz, J. Padhye, P. Patel, B.
Prabhakar, S. Sengupta, and M. Sridharan. ACM SIGCOMM Computer Communication Review
(CCR), Volume 40, Issue 4 (October 2010), pages 63-74.
–
http://dl.acm.org/citation.cfm?id=1851192
–
http://www.sigcomm.org/sites/default/files/ccr/papers/2010/October/1851275-1851192.pdf
•
Check piazza: http://piazza.com/cornell/fall2014/cs5413
•
Check website for updated schedule
This presentation delves into the feasibility and advantages of completely wireless datacenters, focusing on alternative switching technologies, goals, conventional setups, opportunities, 60GHz wireless technology, antenna models, design challenges, and the motivation behind moving towards wireless datacenters. It covers various aspects like transceiver placement, network architecture, MAC protocols, and performance evaluations.
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
You are allowed to download the files provided on this website for personal or commercial use, subject to the condition that they are used lawfully. All files are the property of their respective owners.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.
E N D
Presentation Transcript
Alternative Switching Technologies: Wireless Datacenters Hakim Weatherspoon Assistant Professor, Dept of Computer Science CS 5413: High Performance Systems and Networking October 22, 2014 Slides from the On the Feasibility of Completely Wireless Datacenters at the ACM/IEEE Symposium on Architectures for Networking and Communications Systems (ANCS), October 2012.
Goals for Today On the Feasibility of Completely Wireless Datacenters J. Y. Shin, E. G. Sirer, H. Weatherspoon, and D. Kirovski, IEEE/ACM Transactions on Networking (ToN), Volume 21, Issue 5 (October 2013), pages 1666-1680.
Conventional Datacenter Core Switch Aggregate Switch Top of Rack Switch
Going Completely Wireless Opportunities Low maintenance : no wires Low power: no large switches Low cost: all of the above Fault tolerant: multiple network paths High performance: multiple network paths Which wireless technology?
60GHz Wireless Technology Short range Attenuated by oxygen molecules Directional Narrow beam High bandwidth Several to over 10Gbps License free Has been available for many years Why now? Rx Tx CMOS Integration - Size < dime - Manufacturing cost < $1 7 mm 5 mm [Pinel 09] 6
60 GHz Antenna Model One directional Signal angle between 25 and 45 Maximum range < 10 m No beam steering Bandwidth < 15Gbps TDMA (TDD) FDMA (FDD) Power at 0.1 0.3W How to integrate to datacenters?
Designing Wireless Datacenters Challenges How should transceivers and racks be oriented? How should the network be architected? Interference of densely populated transceivers?
Completely Wireless Datacenters Motivation Cayley Wireless Datacenters Transceiver placement and topology Server and rack designs Network architecture MAC protocols and routing Evaluation Physical Validation: Interference measurements Performance and power Future Conclusion
Transceiver Placement: Server and Rack Design 3D View Rack Server 3-way switch (ASIC design) Inter-rack space Intra-rack space 2D View How do racks communicate with each other?
Cayley Network Architecture: Topology
Masked Node Problem and MAC Most nodes are hidden terminals to others Multiple (>5) directional antennae => Masked node problem Collisions can occur Dual busy tone multiple access [Hass 02] Out of band tone to preserve channels Use of FDD/TDD channels as the tone 12
Cayley Network Architecture: Routing Geographical Routing Inter rack Diagonal XYZ routing Turn within rack Shortest path turning 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18D 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 Within dst rack to dst server Up down to dst story Shortest path to dst server 18S 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18
Completely Wireless Datacenters Motivation Cayley Wireless Datacenters Transceiver placement and topology Server and rack designs Network architecture MAC protocols and routing Evaluation Physical validation: Interference measurements Performance and power Future Conclusion
Hardware Setup for Physical Validation Use of a conservative platform Real-size datacenter floor plan setup Validation of all possible interferences Intra-rack communications Inter-rack communications
Physical Validation: Interference Evaluation (Signal angle = 15 ) Intra-Rack Space (Tx on server 0) -40 Error free Default noise -45 -50 -55 RSS (dB) -60 -65 -70 -75 -80 10 9 8 7 6 5 4 3 2 1 Server ID of Rx
Physical Validation: Interference Evaluation (Signal angle = 15 ) Edge of signal: can be eliminated Orthogonal Inter-Rack Space Orthogonal Inter-Rack Space Orthogonal Inter-Rack Space (Tx on Rack D) (Tx on Rack D) (Tx on Rack D) Diagonal Inter-Rack Space (Tx on Server 2 of Rack D) Non-Adjacent Inter-Rack Space (Tx on Rack D) (Tx on Rack D) (Tx on Rack D) Non-Adjacent Inter-Rack Space Non-Adjacent Inter-Rack Space -40 -40 -40 -40 -40 -40 -40 Error free Default noise Tx: server 1 Error free Error free Error free Tx: server 2 Tx: server 4 Default noise Default noise Default noise Tx: server 3 Error free Default noise Error free Tx: server 0 Tx: server 0 Tx: server 2 Default noise Default noise Tx: server 1 Error free Tx: server 0 -45 -45 -45 -45 -45 -45 -45 Tx: server 2 Tx: server 2 Tx: server 3 -50 -50 -50 -50 -50 -50 -50 -55 -55 -55 -55 -55 -55 -55 RSS (dB) RSS (dB) RSS (dB) RSS (dB) RSS (dB) RSS (dB) RSS (dB) -60 -60 -60 -60 -60 -60 -60 -65 -65 -65 -65 -65 -65 -65 -70 -70 -70 -70 -70 -70 -70 -75 -75 -75 -75 -75 -75 -75 -80 -80 -80 -80 -80 -80 -80 10 10 10 9 9 9 8 8 8 7 7 7 6 6 6 15 14 13 12 11 10 15 15 15 14 14 14 13 13 13 12 12 12 11 11 11 10 10 10 Server ID of Rx on Rack A Server ID of Rx on Rack A Server ID of Rx on Rack A Server ID of Rx on Rack B Server ID of Rx on Rack C Server ID of Rx on Rack C Server ID of Rx on Rack C Potential Interference: can be blocked using conductor curtains
Evaluation Performance: How well does a Cayley datacenter perform and scale? Bandwidth and latency Failure tolerance: How well can a Cayley datacenter handle failures? Server, story, and rack failure Power: How much power does a Cayley datacenter consume compared to wired datacenters
Evaluation Setup Simulate 10K server datacenter Packet level: routing, MAC protocol, switching delay, bandwidth Conventional datacenter (CDC) 3 Layers of oversubscribed switches (ToR, AS, CS) (1, 5, 1), (1, 7, 1) and (2, 5, 1) Latency: 3-6us switching delay Bandwidth: 1Gbps server FAT-tree: Equivalent to CDC (1,1,1) Cayley wireless datacenter 10Gbps bandwidth 1 Transceiver covers 7 to 8 others Signal spreading angle of 25 Low latency Y-switch (<< 1us) (1,5,1) (1,7,1) (2,5,1) (1,1,1) Core 1.4 2 1 10 Aggregate 10 10 5 10 Top of Rack 10 10 10 10
Evaluation Setup Uniform random Src and dst randomly selected in entire datacenter MapReduce Src sends msg to servers in same row of rack Receiver sends msg to servers in same column of rack Receivers send msg to servers inside same pod with 50% probability
Bandwidth Burst of 500 x 1KB packets per server sent Maximum Aggregate Bandwidth Normalized to Fat-tree 1.6 fat-tree CDC 171 Cayley CDC 151 CDC 251 1.4 1.2 1 0.8 s 0.6 0.4 0.2 0 Uniform Rand Hops: CDC < 6, Cayley > 11 MapReduce Hops: CDC < 6, Cayley > 8 Cayley datacenters have the most bandwidth
Latency Uniform random benchmark Uniform Random (4KB Packet) Uniform Random (16KB Packet) 200 10000 fat-tree CDC 251 8000 150 Latency (us) Latency (us) CDC 171 CDC 151 6000 100 4000 Cayley 50 2000 0 0 100 200 300 400 500 100 200 300 400 500 Packet Injection Rate (Packet/Second/Server) Packet Injection Rate (Packet/Second/Server) MapReduce benchmark MapReduce (4KB Packet) MapReduce (16KB Packet) 600 2500 2000 Latency (us) Latency (us) 400 1500 1000 200 500 0 0 100 200 300 400 500 100 200 300 400 500 Packet Injection Rate (Packet/Second/Server) Packet Injection Rate (Packet/Second/Server) Cayley datacenters typically performs the best
Fault Tolerance Preserved connectivity among live nodes 25% 55% 77% 99% 100 Preserved connectivity (%) 80 60 Node 40 Story Rack 20 0 0 10 20 30 40 50 60 70 80 90 Failed components (%) Cayley datacenters are extremely fault tolerant
Power Consumption to Connect 10K Servers Conventional datacenter (CDC) * Switch Type Typical Power Top of rack switch (ToR) 176W Aggregation switch (AS) 350W Core switch (CS) 611W Depending on the oversubscription rate 58KW to 72KW Cayley datacenter Transceivers consume < 0.3W Maximum power consumption: 6KW Less than 1/10 of CDC power consumption * Cost and spec of Cisco 4000, 5000, 7000 series switches
Discussion and Future Work Only scratched the surface How far can wireless datacenters go with no wires? Need larger experiment/testbed Interference and performance of densely connected datacenter? Scaling to large datacenters (>100K servers)? Scaling to higher bandwidth (> 10Gbps)?
Conclusion Completely wireless datacenters can be feasible Cayley wireless datacenters exhibit Low maintenance High performance Fault tolerant Low power Low cost
References S. Pinel, P. Sen, S. Sarkar, B. Perumana, D. Dawn, D. Yeh, F. Barale, M. Leung, E. Juntunen, P. Vadivelu, K. Chuang, P. Melet, G. Iyer, and J. Laskar. 60GHz single- chip CMOS digital radios and phased array solutions for gaming and connectivity. IEEE Journal on Selected Areas in Communications, 27(8), 2009. Z.J. Hass and J. Deng. Dual busy tone multiple access (DBTMA)-a multiple access control scheme for ad hoc networks. IEEE Transactions on Communications, 50(6), 2002. PEPPM. Cisco Current Price List. http://www.peppm.org/Products/cisco/price.pdf, 2012. 27
Related Work Link Technology Modifications Required Working Prototype Optics w/ WDM 10G-180G (CWDM) 10G-400G (DWDM) Switch Software Glimmerglass, Fulcrum Helios (SIGCOMM 10) Optics (10G) Host OS Emulation c-Through (SIGCOMM 10) Wireless (1G, 10m) Unspecified Flyways (SIGCOMM 11, HotNets 09) Optics (10G) Host Application; Specific to Stream Processing Calient, Nortel IBM System-S (GLOBECOM 09) Optics (10G) Host NIC Hardware HPC (SC 05)
Before Next time Project Interim report Due Monday, October 27. And meet with groups, TA, and professor Lab3 Packet filter/sniffer Due yesterday, Tuesday, October 21. But, 24 hour grace period. Lab1/2 redux due Friday, October 24 Fractus Upgrade: SAVE ALL YOUR DATA Fractus will be upgraded from October 28th to 30th Can use Red Cloud during upgrade period, then switch back to Fractus Required review and reading for Wednesday, October 22 Data center TCP (DCTCP), M. Alizadeh, A. Greenberg, D. A. Maltz, J. Padhye, P. Patel, B. Prabhakar, S. Sengupta, and M. Sridharan. ACM SIGCOMM Computer Communication Review (CCR), Volume 40, Issue 4 (October 2010), pages 63-74. http://dl.acm.org/citation.cfm?id=1851192 http://www.sigcomm.org/sites/default/files/ccr/papers/2010/October/1851275-1851192.pdf Check piazza: http://piazza.com/cornell/fall2014/cs5413 Check website for updated schedule
