Automatic Load Balancing Proposal for PMD Port Queue Assignment

This proposal discusses dynamic port queue assignment strategies for PMDs in network environments. It covers both static and dynamic assignment methods, such as round-robin and cycles-based algorithms, to ensure uniform load distribution. The proposal introduces an automatic load balancing feature, detailing additional statistics and configuration parameters for improved PMD overload detection and reassignment triggers. Emphasis is placed on maintaining load distribution efficiency while considering factors like Rx drops and Q drops thresholds.

• Load Balancing
• PMD
• Dynamic Assignment
• Network Management
• Automatic Reassignment

maclean_c Follow

Uploaded on Sep 15, 2024 | 0 Views

Download Presentation

Please find below an Image/Link to download the presentation.

The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. Download presentation by click this link. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.

Presentation Transcript

1. PMD Auto Load Balancing Nitin Katiyar Pradeep Venkatesan Jan Scheurich 2018-12-04 | Page 1 (20)

2. Agenda Background Auto Load Balancing Proposal Example Summary Questions 2018-12-04 | Page 2 (20)

3. Background Port Rx queues assignment to PMDs is realized statically by configuration dynamically - on demand or when certain events occur Static port queue assignment: Simple Works best if traffic patterns are well understood and unchanging 2018-12-04 | Page 3 (20)

4. Background(contd) Dynamic port queue assignment: triggered by port addition, deletion etc. triggered by CLI pmd-rxq-rebalance command configurable assignment algorithm i.e. cycles or roundrobin can better handle unpredictable traffic patterns but still not adaptive 2018-12-04 | Page 4 (20)

5. Background(contd) Round Robin Assignment Simple algorithm Load distribution not uniform Reassignment may or may not result in better load distribution Cycles based Assignment Assignment aims to keep load distribution uniform across PMDs Each PMD keeps track of the processing cycles spent for each Rx Queue Six snapshots of the queue processing cycles each spanning 10 seconds are maintained When reassignment requested, the queue statistics are used to get a uniform distribution 2018-12-04 | Page 5 (20)

6. Automatic Load Balancing - Proposal Trigger cycles based assignment automatically (disabled by default) Maintain additional per-queue statistics to help detect if any PMD is overloaded Additional configuration parameters to fine tune the behavior Reassignment will be triggered only if such a reassignment will result in better load distribution 2018-12-04 | Page 6 (20)

7. Automatic Load Balancing - Proposal Details Additional per-queue statistics to be maintained Number of Rx drops for physical port queues Number of times vhost ports exceeded the qfill threshold New configuration parameters Enable/Disable automatic load balancing PMD Overload Threshold (utilization %) Whether Q Drops should factor into PMD Overload determination Minimum improvement in load distribution expected as a result of reassignment 2018-12-04 | Page 7 (20)

8. Automatic Load Balancing - Proposal Details (contd ) Modifications to Cycles based reassignment function A new option termed "dry-run" Dry-run option will return the poll list resulting from running the algorithm withoutactually performing the reassignment. 2018-12-04 | Page 8 (20)

9. Automatic Load Balancing - Details (contd ) A PMD will consider itself overloaded, if for six consecutive 10- second intervals: the PMD utilization is more than the configured Overload threshold AND if Q drops are to be factored, number of Rx queue drops or number of times qfill- threshold exceeded is non-zero 2018-12-04 | Page 9 (20)

10. Automatic Load Balancing - Details (contd ) OVS Main thread changes: if any PMD considers itself overloaded, perform a dry-run of the reassignment if dry-run poll list identical to current poll list, nothing more to do if dry-run poll list different from current poll list, reassignment may offer benefits 2018-12-04 | Page 10 (20)

11. Automatic Load Balancing - Details (contd ) OVS Main thread changes: Evaluate if the reassignment will meet the configured minimum improvement as follows Calculate the total processing cycles for each PMD based on current poll list of RX queue Calculate the average variance in processing cycles for each PMD Repeat above calculations using dry-run poll list If dry-run average variance is less, the reassignment will likely result in a more uniform load distribution - Calculate the percentage improvement of the new average variance over the old average variance - Trigger reassignment if the difference is more the configured improvement threshold 2018-12-04 | Page 11 (20)

12. Automatic Load Balancing - Details (contd ) PMD Thread PMD Thread Record processing cycles and checks if PMD is overloaded Record processing cycles and checks if PMD is overloaded Monitor PMD load Monitor PMD load Continue monitoring PMD load Main Thread Is any of the PMD thread overloaded? No Yes Reload PMD with new RX queue mapping Reload PMD with new RX queue mapping Initiate PMD dry run Does dry run indicate improvement ? No Yes Initiate data path reconfiguration 2018-12-04 | Page 12 (20)

13. Automatic Load Balancing - Example PMD Processing Load 100 35 80 60 30 40 20 25 10 20 5 0 PMD 1 PMD 2 Initial Configuration Configured pmd-auto-lb-thresh = 80% Processing load of PMD1 = 90% (Q0=25%, Q1=30%, Q2=35%) Processing load of PMD2 = 35% (Q3=10%, Q4=20%, Q5=5%) Mean load = 62% and Variance = 756 2018-12-04 | Page 13 (20)

14. Automatic Load Balancing - Example(contd ) PMD Processing Load 100 80 5 20 60 30 10 40 35 25 20 0 PMD 1 PMD 2 PMD Dry Run Processing load of PMD1 = 60% (Q0=25%, Q1=30%, Q5=5%) Processing load of PMD2 = 65% (Q2=35%, Q3=10%, Q4=20%) Mean load = 62% and Variance = 6 Improvement = ((756 6) * 100) / 756 = 99% 2018-12-04 | Page 14 (20)

15. OVS CLI changes We propose to have following knobs for Automatic PMD load balancing. ovs-vsctl set open_vswitch . other_config:pmd-auto-lb="true" ovs-vsctl set open_vswitch . other_config:pmd-auto-lb-thresh= <>" ovs-vsctl set open_vswitch . other_config:pmd-auto-lb-min- improvement= <>" ovs-vsctl set open_vswitch . other_config:pmd-auto-lb-drop- check= <> 2018-12-04 | Page 15 (20)

16. RFC Patch RFC Patch v1 is available in ML: https://patchwork.ozlabs.org/patch/982444/ 2018-12-04 | Page 16 (20)

17. Future Enhancements Frequent change in RX queue to PMD assignment can be avoided by adding a dampening logic in main thread. Time interval and frequency for PMD auto load balancing can be made configurable via CLI. The threshold for RX Drops for physical ports and RX qfill for vhost user ports can be made configurable via CLI. 2018-12-04 | Page 17 (20)

18. Summary PMD auto load balancing feature provides a mechanism to efficiently utilize the PMDs if traffic pattern changes after RX queue assignment to PMDs. This feature can be enabled/disabled dynamically via OVS CLI and no additional cost in processing when this feature is disabled. Provides various tunable parameters via OVS CLI to setup automatic load balancing according to the needs. 2018-12-04 | Page 18 (20)

19. Questions 2018-12-04 | Page 19 (20)

20. 2018-12-04 | Page 20 (20)