Importance of Hot-Plug and Error Handling for NVMe
undefined
P
C
I
e
H
o
t
-
P
l
u
g
a
n
d
E
r
r
o
r
H
a
n
d
l
i
n
g
f
o
r
N
V
M
e
2
0
1
9
N
V
M
e
™
A
n
n
u
a
l
M
e
m
b
e
r
s
M
e
e
t
i
n
g
a
n
d
D
e
v
e
l
o
p
e
r
D
a
y
M
a
r
c
h
1
9
,
2
0
1
9
P
r
e
p
a
r
e
d
b
y
:
A
u
s
t
i
n
B
o
l
e
n
,
S
e
r
v
e
r
S
t
o
r
a
g
e
T
e
c
h
n
o
l
o
g
i
s
t
,
D
e
l
l
E
M
C
C
u
r
t
i
s
B
a
l
l
a
r
d
,
S
t
o
r
a
g
e
T
e
c
h
n
o
l
o
g
i
s
t
,
H
P
E
J
o
e
C
o
w
a
n
,
S
e
n
i
o
r
S
y
s
t
e
m
s
A
r
c
h
i
t
e
c
t
,
H
P
E
Agenda
•
The Importance of Hot-Plug and Error Handling for NVMe™
•
Challenges with NVMe Hot-Plug and Error Handling
•
Solutions to NVMe Hot-Plug and Error Handling Challenges
•
Questions
The Importance of Hot-Plug and
Error Handling for NVMe™
The Importance of Hot-Plug (RASM)
*
https://software.intel.com/en-us/articles/rasm-a-primer-for-isv-applications-engineers
Better RASM = Reduced TCO
Better RASM = Reduced TCO
Customer Requirements
:
•
Surprise/Async hot-plug
- No prepare-to-remove
•
Parity with SAS/SATA or better
•
Handle all PCIe errors, not just
errors due to surprise/async
removal
The Importance of Hot-Plug (Reliability)
*
https://software.intel.com/en-us/articles/rasm-a-primer-for-isv-applications-engineers
Reliability
:
Device reliability is key, however:
•
Small failure rates exacerbated at
scale
•
Hundreds or thousands of
systems per datacenter
•
Many drives per system
•
NAND wears out
Failures will occur HA solutions will
require Hot-Plug
The Importance of Hot-Plug (Manageability)
*
https://software.intel.com/en-us/articles/rasm-a-primer-for-isv-applications-engineers
Manageability
:
•
Monitoring and reporting of
device failure or predicted failure
•
Inventorying for re-provisioning of
storage
The Importance of Hot-Plug (Serviceability)
*
https://software.intel.com/en-us/articles/rasm-a-primer-for-isv-applications-engineers
Serviceability
:
•
Async hot-plug is required for
SAS/SATA equivalent serviceability
for NVMe drives
•
Async/surprise removal eliminates
the need for:
•
Orderly removal software
•
A technician with physical
access to replace drives may
not have access to these
software interfaces
•
Costly orderly removal hardware
(attention buttons, power controllers,
etc.)
The Importance of Hot-Plug (Availability)
*
https://software.intel.com/en-us/articles/rasm-a-primer-for-isv-applications-engineers
Availability
:
•
Hot-plug increases availability by
avoiding costly downtime due to:
•
Replacing failed drives
•
Re-provisioning storage
Challenges with NVMe™ Hot-Plug and
Error Handling
NVMe™ Hot-Plug/Error Handling – Why is it such a heavy lift?
Because it’s an ecosystem issue!
•
NVMe Drive
•
Platform
•
Hardware
•
Firmware
•
BMC
•
PCIe Root Port/Switch
•
Operating System
•
NVMe Driver
•
PCIe Driver
•
ACPI Driver
•
Applications
Each player historically looking at
their own piece. But who is looking at
the whole picture?
Hot-Plug Storage – A High-Level Comparison
Host Software
(Operating System, Drivers,
Applications, UEFI/BIOS)
SAS
Controller
SATA
Controller
NVMe
Controller
SAS
Drive
SATA
Drive
NVMe
Drive
Hot-Plug Barrier
Processor
Hardware above the
barrier
is not
hot pluggable
Hardware below the
barrier
is
hot pluggable
SAS Bus
SATA Bus
PCIe Bus
•
SAS/SATA drivers bind to
controllers above the hot
plug barrier
•
Protocol conversion
provides software isolation
•
Physical layer conversion
provides hardware isolation
•
NVMe™ drivers bind to
controllers below the hot plug
barrier
•
No protocol translation == No
software isolation
•
No physical layer conversion
== No hardware isolation
The PCIe Hot-Plug Eras
(Where we’ve been, Where we are)
•
T
h
e
S
t
a
n
d
a
r
d
H
o
t
-
P
l
u
g
C
o
n
t
r
o
l
l
e
r
(
S
H
P
C
)
E
r
a
–
Timeframe: PCI/PCI-X, Early PCIe
–
Complex (196 page specification)
–
Orderly insertion/removal only
–
Async insert/removal likely to crash system
–
Additional hardware (expensive)
–
Power Controllers
–
Power/Attention Indicators/Buttons
–
M
e
c
h
a
n
i
c
a
l
R
e
t
e
n
t
i
o
n
L
a
t
c
h
(
M
R
L
)
•
T
h
e
H
o
t
-
P
l
u
g
S
u
r
p
r
i
s
e
(
H
P
S
)
E
r
a
–
Timeframe: Starting with new form factors like PCIe storage and Thunderbolt to present day
–
New form factors demand a simplified user experience that eliminates orderly removal overhead
–
For NVMe, mimic SAS/SATA hot-plug model
–
Surprise insertion/removal
–
Surprise removal not supported by most OSes
–
Software or hardware initiated orderly removal typically required
Hot-Plug Issues Persist After SHPC and HPS
•
System crashes are still possible
•
Errors if orderly removal process not followed with SHPC
•
Synthesized all 1’s data during errors - not always handled correctly by software
•
No strict model for interaction of stack components - leads to race conditions causing
crashes and deadlocks
•
Other issues
•
Timely detection of removal and insertion (detection while in low power state)
•
Mechanical insert/remove issues (slow insert, angled insert, etc.)
•
Issues often require changes outside the component under test (OS, switch, etc.)
•
SHPC and HPS aren’t robust enough for complex use cases
Solutions to NVMe™ Hot-Plug and
Error Handling Challenges
Key Design Tenets
•
Create a hot-plug and error handling/recovery “toolbox”
-
Allow for flexibility in solution
-
Systems, Form Factors, OSes all have different needs
-
Support all PCIe use cases, not just NVMe
-
Tools to handle unforeseen issues
•
Fix known issues
•
Leverage and reach parity with existing solutions
-
SAS/SATA model
Eliminate need for orderly insertion/removal
-
Proprietary PCIe error recovery models
•
Multi-phase approach with incremental improvements
•
Error recovery mechanisms must be extensible to
all
PCIe errors
-
Surprise/async removal errors
-
Minimize the chance of issue due to accidental removal of wrong device
-
Errors unrelated to hot-plug
Hot-Plug
&
Error Handling
Key Design Tenets
•
Hooks for time-to-market
•
System hardware/firmware changes should be
sufficient for:
•
New system designs and form factors
•
Fixing defects/unforeseen issues
•
Avoid/minimize need for:
•
Future OS changes
•
Future PCIe Root Port/Switch changes
Industry Alignment
•
Alignment/Feedback from OEMs
•
Dell EMC
•
HPE
•
Lenovo
•
Oracle
•
Alignment/Feedback from PCIe Root Port and
Switch Vendors
•
AMD
•
Broadcom
•
Intel
•
Microsemi
•
OSVs
•
Microsoft
•
VMWare
•
Linux distributors/kernel developers
Standards-Based Solution
CER Era
Host SW/FW
(Operating System,
Drivers, Applications, UEFI/BIOS)
PCIe Root
Port w/ DPC
NVMe
Drive
Processor
PCIe Bus
E
r
r
o
r
PCIe Root
Port w/ DPC
Switch
Upstream
Port
Switch
Downstream
Port w/ DPC
Switch
Downstream
Port w/ DPC
NVMe
Drive
NVMe
Drive
Asyn
c Removal or
other errors d
etected
by the Root Port or
Switch
DPC in Root Port or
Switch contains errors
by forcing/keeping
PCIe link down
The Root Port or
Switch notifies FW or
host OS
FW and/or host OS
entities attempt to
recover from the error
PCIe Bus
PCIe Bus
A
s
y
n
c
R
e
m
o
v
e
Host OS releases
DPC and restarts
device if present and
recovered
PCIe
Switch
•
T
h
e
C
o
n
t
a
i
n
m
e
n
t
E
r
r
o
r
R
e
c
o
v
e
r
y
(
C
E
R
)
E
r
a
–
Timeframe: Transitioning now
–
Replaces HPS
–
The term “async” replaces “surprise” (i.e.
async removal/insertion instead of surprise
insertion/removal) in PCIe specs
–
CER software/firmware model can be used
to recover from many PCIe errors – not
just errors due to async removal
–
Utilizes Downstream Port Containment
(DPC) hardware in PCIe root ports and
switch downstream ports to contain errors
including async remove related errors
–
Two CER modes: Native OS Controlled
and Firmware First
›
Firmware First mode requires ACPI changes
in OS and BIOS/UEFI
–
Based on tried-and-true proprietary models
System Firmware Intermediary Era
Host Software
(Operating System,
Drivers, Applications, UEFI/BIOS)
SAS
Controller
SATA
Controller
NVMe
Controller
SAS
Drive
SATA
Drive
NVMe
Drive
Hot-Plug Barrier
Processor
Hardware above the
barrier
is not
hot pluggable
Hardware below the
barrier
is
hot pluggable
SAS Bus
SATA Bus
PCIe Bus
System Firmware
Intermediary (SFI)
•
SFI isolates PCIe hot-plug
events from the OS, drivers,
and applications
for hot-plug
-
does not alter data path.
•
Hardware isolation in PCIe
Root Ports and Switch
Downstream Ports
•
Provides options to invo
ke
system firmware (BIOS, UEFI,
BMC, etc.) for hot-plug events
•
Particularly useful for complex
out-of-band (independent of
host OS) platform config of
hot-inserted devices (e.g.,
unlocking TCG drives or
device authentication)
•
T
h
e
S
y
s
t
e
m
F
i
r
m
w
a
r
e
I
n
t
e
r
m
e
d
i
a
r
y
(
S
F
I
)
E
r
a
–
Timeframe: Silicon support will arrive over next several years
–
Does not replace DPC/CER - works alongside DPC/CER
–
Adds hardware/firmware layer between OS and devices for hot-plug
Hot-Plug Parameter Extensions (_HPX)
•
_
H
P
X
e
x
i
s
t
s
a
c
r
o
s
s
a
l
l
h
o
t
-
p
l
u
g
e
r
a
s
•
_
H
P
X
a
l
l
o
w
s
s
y
s
t
e
m
f
i
r
m
w
a
r
e
t
o
p
r
o
v
i
d
e
s
y
s
t
e
m
-
s
p
e
c
i
f
i
c
P
C
I
e
c
o
n
f
i
g
s
p
a
c
e
s
e
t
t
i
n
g
s
t
o
O
S
–
Not just for hot-inserted device; also used if device is reset at runtime
•
N
e
w
_
H
P
X
S
e
t
t
i
n
g
R
e
c
o
r
d
(
T
y
p
e
3
)
d
e
f
i
n
e
d
i
n
A
C
P
I
s
p
e
c
i
f
i
c
a
t
i
o
n
–
Previous setting records only worked for pre-defined registers
–
New registers required spec update an OS change
–
New Type 3 record can specify
any
register with offset relative to offset 0h of:
–
The start of configuration space
–
A Capability Structure
–
An Extended Capability Structure
–
A Vendor-Specific Extended Capability
–
A Designated Vendor-Specific Extended Capability
•
H
a
n
d
l
e
d
i
f
f
e
r
e
n
t
r
e
v
i
s
i
o
n
s
o
f
c
a
p
a
b
i
l
i
t
y
s
t
r
u
c
t
u
r
e
s
–
Apply changes to any revision of the capability structure
–
Apply changes to a specific revision of the capability structure
–
Apply changes to capability structures with revision greater than or equal to
the specified revision
•
S
u
p
p
o
r
t
s
s
i
m
p
l
e
i
f
-
t
h
e
n
-
e
l
s
e
c
o
n
d
i
t
i
o
n
a
l
g
r
a
m
m
a
r
–
E
.
g
.
,
t
o
s
e
t
P
C
I
e
c
o
n
f
i
g
u
r
a
t
i
o
n
s
p
a
c
e
r
e
g
i
s
t
e
r
s
t
o
p
r
e
f
e
r
r
e
d
v
a
l
u
e
b
a
s
e
d
o
n
d
e
v
i
c
e
c
a
p
a
b
i
l
i
t
y
•
L
i
g
h
t
w
e
i
g
h
t
a
l
t
e
r
n
a
t
i
v
e
t
o
S
F
I
f
o
r
s
i
m
p
l
e
c
o
n
f
i
g
s
p
a
c
e
s
e
t
t
i
n
g
s
E
x
a
m
p
l
e
P
s
e
u
d
o
c
o
d
e
–
S
e
t
C
o
m
p
l
e
t
i
o
n
T
i
m
e
o
u
t
(
C
T
O
)
V
a
l
u
e
b
a
s
e
d
o
n
d
e
v
i
c
e
’
s
C
o
m
p
l
e
t
i
o
n
T
i
m
e
o
u
t
R
a
n
g
e
s
S
u
p
p
o
r
t
e
d
:
If CTO Range B supported then
Set CTO
Value
to 65 ms to 210 ms
Else if CTO Range C supported then
Set CTO Value to 260 ms to 900 ms
Else if CTO Range D supported then
Set CTO Value to 4 s to 13 s
Else
Set CTO Disable
Next Steps
•
P
C
I
e
R
o
o
t
P
o
r
t
s
a
n
d
S
w
i
t
c
h
e
s
-
Add support for DPC/eDPC
-
Add support for SFI
•
O
p
e
r
a
t
i
n
g
S
y
s
t
e
m
s
a
n
d
O
E
M
s
-
Add support for async removal in HPS mode as a stop-gap until CER can be fully implemented
-
Add support for Containment Error Recovery Model defined by PCI-SIG
Native OS controlled and Firmware First models
-
Review/contribute to open source effort
DPC Containment Error Recovery patches submitted to Linux kernel
o
Also called Error Disconnect Recover (EDR) after the ACPI method used in DPC CER model
_HPX patches submitted to Linux kernel
•
C
o
n
n
e
c
t
o
r
s
/
F
o
r
m
F
a
c
t
o
r
s
-
D
e
s
i
g
n
f
o
r
a
s
y
n
c
h
o
t
-
p
l
u
g
-
Prevent damage to I/O pins on hot-insert typically by making ground pins longer than other pins
-
Limit current surge on hot-insert
Pre-charge pin for each voltage rail which is second to mate or
Soft start/hot-plug circuits for each rail
-
Physical presence mandatory
Should be shortest pin so platform knows when device is fully inserted
May need a presence pin on each end of connector unless you can guarantee connector cannot mate at an angle
-
Make sure pins can’t cross-connect on insert
-
Consider issues with pin wipe b/c higher frequencies demand shorter pin lengths making it difficult to support pins of different length
-
Form factors should allow for stable insert/removal
-
Form factors should allow adequate mount points
Resources
* Requires member access to the relevant standards body website
Linux Enablement
Q
u
e
s
t
i
o
n
s
?
Delve into the critical aspects of hot-plug and error handling for NVMe technology, focusing on challenges, solutions, customer requirements, and the significance of reliability, manageability, and serviceability. Learn how these elements impact device performance and mitigate failures at scale. Explore the need for async hot-plug features to enhance serviceability and reduce total cost of ownership for storage systems.
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. Download presentation by click this link. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
E N D
Presentation Transcript
Architected for Performance PCIe Hot-Plug and Error Handling for NVMe 2019 NVMe Annual Members Meeting and Developer Day March 19, 2019 Prepared by: Austin Bolen, Server Storage Technologist, Dell EMC Curtis Ballard, Storage Technologist, HPE Joe Cowan, Senior Systems Architect, HPE
Agenda The Importance of Hot-Plug and Error Handling for NVMe Challenges with NVMe Hot-Plug and Error Handling Solutions to NVMe Hot-Plug and Error Handling Challenges Questions
The Importance of Hot-Plug and Error Handling for NVMe
The Importance of Hot-Plug (RASM) Customer Requirements: Surprise/Async hot-plug - No prepare-to-remove Parity with SAS/SATA or better Handle all PCIe errors, not just errors due to surprise/async removal Better RASM = Reduced TCO * https://software.intel.com/en-us/articles/rasm-a-primer-for-isv-applications-engineers
The Importance of Hot-Plug (Reliability) Reliability: Device reliability is key, however: Small failure rates exacerbated at scale Hundreds or thousands of systems per datacenter Many drives per system NAND wears out Failures will occur HA solutions will require Hot-Plug * https://software.intel.com/en-us/articles/rasm-a-primer-for-isv-applications-engineers
The Importance of Hot-Plug (Manageability) Manageability: Monitoring and reporting of device failure or predicted failure Inventorying for re-provisioning of storage * https://software.intel.com/en-us/articles/rasm-a-primer-for-isv-applications-engineers
The Importance of Hot-Plug (Serviceability) Serviceability: Async hot-plug is required for SAS/SATA equivalent serviceability for NVMe drives Async/surprise removal eliminates the need for: Orderly removal software A technician with physical access to replace drives may not have access to these software interfaces Costly orderly removal hardware (attention buttons, power controllers, etc.) * https://software.intel.com/en-us/articles/rasm-a-primer-for-isv-applications-engineers
The Importance of Hot-Plug (Availability) Availability: Hot-plug increases availability by avoiding costly downtime due to: Replacing failed drives Re-provisioning storage * https://software.intel.com/en-us/articles/rasm-a-primer-for-isv-applications-engineers
Challenges with NVMe Error Handling Hot-Plug and
NVMe Hot-Plug/Error Handling Why is it such a heavy lift? Because it s an ecosystem issue! NVMe Drive Platform Hardware Firmware BMC PCIe Root Port/Switch Operating System NVMe Driver PCIe Driver ACPI Driver Applications It s a fan! It s a wall! It s a rope! It s a snake! It s a spear! It s a tree! Each player historically looking at their own piece. But who is looking at the whole picture?
Hot-Plug Storage A High-Level Comparison SAS/SATA drivers bind to controllers above the hot plug barrier Processor Protocol conversion provides software isolation Host Software (Operating System, Drivers, Applications, UEFI/BIOS) Physical layer conversion provides hardware isolation Hardware above the barrier is not hot pluggable SAS SATA Controller Controller PCIe Bus Hot-Plug Barrier NVMe controllers below the hot plug barrier drivers bind to SATA Bus SAS Bus Hardware below the barrier is hot pluggable No protocol translation == No software isolation NVMe Controller NVMe Drive SAS Drive SATA Drive No physical layer conversion == No hardware isolation
The PCIe Hot-Plug Eras (Where we ve been, Where we are) The Standard Hot-Plug Controller (SHPC) Era Timeframe: PCI/PCI-X, Early PCIe Complex (196 page specification) Orderly insertion/removal only Async insert/removal likely to crash system Additional hardware (expensive) Power Controllers Power/Attention Indicators/Buttons Mechanical Retention Latch (MRL) The Hot-Plug Surprise (HPS) Era Timeframe: Starting with new form factors like PCIe storage and Thunderbolt to present day New form factors demand a simplified user experience that eliminates orderly removal overhead For NVMe, mimic SAS/SATA hot-plug model Surprise insertion/removal Surprise removal not supported by most OSes Software or hardware initiated orderly removal typically required
Hot-Plug Issues Persist After SHPC and HPS System crashes are still possible Errors if orderly removal process not followed with SHPC Synthesized all 1 s data during errors - not always handled correctly by software No strict model for interaction of stack components - leads to race conditions causing crashes and deadlocks Other issues Timely detection of removal and insertion (detection while in low power state) Mechanical insert/remove issues (slow insert, angled insert, etc.) Issues often require changes outside the component under test (OS, switch, etc.) SHPC and HPS aren t robust enough for complex use cases
Solutions to NVMe Error Handling Challenges Hot-Plug and
Key Design Tenets Create a hot-plug and error handling/recovery toolbox - Allow for flexibility in solution - Systems, Form Factors, OSes all have different needs - Support all PCIe use cases, not just NVMe - Tools to handle unforeseen issues Hot-Plug & Error Handling Hot Hot- -Plug & Error Handling Error Handling Fix known issues Leverage and reach parity with existing solutions - SAS/SATA model Eliminate need for orderly insertion/removal - Proprietary PCIe error recovery models Plug & Multi-phase approach with incremental improvements Error recovery mechanisms must be extensible to all PCIe errors - Surprise/async removal errors - Minimize the chance of issue due to accidental removal of wrong device - Errors unrelated to hot-plug
Key Design Tenets Hooks for time-to-market System hardware/firmware changes should be sufficient for: New system designs and form factors Fixing defects/unforeseen issues Avoid/minimize need for: Future OS changes Future PCIe Root Port/Switch changes
Industry Alignment Alignment/Feedback from OEMs Dell EMC HPE Lenovo Oracle Alignment/Feedback from PCIe Root Port and Switch Vendors AMD Broadcom Intel Microsemi OSVs Microsoft VMWare Linux distributors/kernel developers
Standards-Based Solution ECN Sponsors Standards Bodies Specifications Proposal Standard Stage Description System Firmware Intermediary (SFI) PCIe Base Spec Ratified. ECN Published to PCI-SIG Website. Adds system firmware layer between OS and PCIe devices for hot-plug. Containment Error Recovery (CER) PCIe Base Spec Ratified. ECN Published to PCI-SIG Website. Defines software/firmware PCIe error recovery model built on top of Downstream Port Containment hardware. ACPI Spec Released In ACPI 6.3 PCI Firmware Specification Ratified. ECN Published to PCI-SIG Website. Hot-Plug Extensions (_HPX) ACPI Spec Released In ACPI 6.3 Allows system firmware to tell OS how to set PCIe Configuration Space for hot-inserted PCIe devices. PCI Firmware Specification Member Review Complete. Should be ratified shortly.
5 CER Era Host OS releases DPC and restarts device if present and recovered Processor Host SW/FW (Operating System, Drivers, Applications, UEFI/BIOS) The Containment Error Recovery (CER) Era Timeframe: Transitioning now Replaces HPS The term async replaces surprise (i.e. async removal/insertion instead of surprise insertion/removal) in PCIe specs CER software/firmware model can be used to recover from many PCIe errors not just errors due to async removal Utilizes Downstream Port Containment (DPC) hardware in PCIe root ports and switch downstream ports to contain errors including async remove related errors Two CER modes: Native OS Controlled and Firmware First Firmware First mode requires ACPI changes in OS and BIOS/UEFI Based on tried-and-true proprietary models PCIe Root Port w/ DPC PCIe Root Port w/ DPC 4 FW and/or host OS entities attempt to recover from the error PCIe PCIe Bus 3 Switch Upstream Port The Root Port or Switch notifies FW or host OS Switch Switch Downstream Port w/ DPC Switch Downstream Port w/ DPC Error DPC in Root Port or Switch contains errors by forcing/keeping PCIe link down 2 PCIe Bus PCIe Bus Async Remove NVMe Drive Async Removal or other errors detected by the Root Port or Switch 1 NVMe Drive NVMe Drive
System Firmware Intermediary Era SFI isolates PCIe hot-plug events from the OS, drivers, and applications for hot-plug - does not alter data path. The System Firmware Intermediary (SFI) Era Timeframe: Silicon support will arrive over next several years Does not replace DPC/CER - works alongside DPC/CER Adds hardware/firmware layer between OS and devices for hot-plug Hardware isolation in PCIe Root Ports and Switch Downstream Ports Processor Host Software (Operating System, Drivers, Applications, UEFI/BIOS) Provides options to invoke system firmware (BIOS, UEFI, BMC, etc.) for hot-plug events Hardware above the barrier is not hot pluggable SAS SATA Controller System Firmware Intermediary (SFI) Controller Particularly useful for complex out-of-band (independent of host OS) platform config of hot-inserted devices (e.g., unlocking TCG drives or device authentication) PCIe Bus Hot-Plug Barrier SATA Bus SAS Bus Hardware below the barrier is hot pluggable NVMe Controller NVMe Drive SAS Drive SATA Drive
Hot-Plug Parameter Extensions (_HPX) _HPX exists across all hot-plug eras Example Pseudocode Set Completion Timeout (CTO) Value based on device s Completion Timeout Ranges Supported: _HPX allows system firmware to provide system-specific PCIe config space settings to OS Not just for hot-inserted device; also used if device is reset at runtime If CTO Range B supported then Set CTO Value to 65 ms to 210 ms Else if CTO Range C supported then Set CTO Value to 260 ms to 900 ms Else if CTO Range D supported then Set CTO Value to 4 s to 13 s Else Set CTO Disable New _HPX Setting Record (Type 3) defined in ACPI specification Previous setting records only worked for pre-defined registers New registers required spec update an OS change New Type 3 record can specify any register with offset relative to offset 0h of: The start of configuration space A Capability Structure An Extended Capability Structure A Vendor-Specific Extended Capability A Designated Vendor-Specific Extended Capability Handle different revisions of capability structures Apply changes to any revision of the capability structure Apply changes to a specific revision of the capability structure Apply changes to capability structures with revision greater than or equal to the specified revision Supports simple if-then-else conditional grammar E.g., to set PCIe configuration space registers to preferred value based on device capability Lightweight alternative to SFI for simple config space settings
Next Steps PCIe Root Ports and Switches - Add support for DPC/eDPC - Add support for SFI Operating Systems and OEMs - Add support for async removal in HPS mode as a stop-gap until CER can be fully implemented - Add support for Containment Error Recovery Model defined by PCI-SIG Native OS controlled and Firmware First models - Review/contribute to open source effort DPC Containment Error Recovery patches submitted to Linux kernel o Also called Error Disconnect Recover (EDR) after the ACPI method used in DPC CER model _HPX patches submitted to Linux kernel Connectors/Form Factors - Design for async hot-plug - Prevent damage to I/O pins on hot-insert typically by making ground pins longer than other pins - Limit current surge on hot-insert Pre-charge pin for each voltage rail which is second to mate or Soft start/hot-plug circuits for each rail - Physical presence mandatory Should be shortest pin so platform knows when device is fully inserted May need a presence pin on each end of connector unless you can guarantee connector cannot mate at an angle - Make sure pins can t cross-connect on insert - Consider issues with pin wipe b/c higher frequencies demand shorter pin lengths making it difficult to support pins of different length - Form factors should allow for stable insert/removal - Form factors should allow adequate mount points
Resources Resource Link https://uefi.org/sites/default/files/resources/ACPI_6_3_final_Jan30.pdf (DPC EDR) https://mantis.uefi.org/mantis/view.php?id=1939* (_HPX) https://mantis.uefi.org/mantis/view.php?id=1922* ACPI 6.3: Add Error Disconnect Recover mechanism for DPC and new Hot-Plug Parameter Extensions (_HPX) Setting Record (Type 3) PCI Express Base Specification Revision 4.0 Version 1.0 https://members.pcisig.com/wg/PCI-SIG/document/10912?downloadRevision=active* PCIe Base Spec. ECN: Async Hot-Plug Updates (DPC/CER, SFI) https://members.pcisig.com/wg/PCI-SIG/document/12400* https://members.pcisig.com/wg/PCI-SIG/document/12614* PCI Firmware Spec. ECN: Downstream Port Containment related Enhancements https://members.pcisig.com/wg/PCI-SIG/document/12712* PCI Firmware Spec. ECN: _HPX and PCIe Completion Timeout related _OSC Enhancements Dell EMC Tech Note: NVMe Hot-Plug Challenges and Industry Adoption https://downloads.dell.com/manuals/common/dfd_-_nvme_hot- plug_challenges_and_industry_adoption.pdf Implementing Hot-Plug in NVMe Storage Systems https://www.flashmemorysummit.com/English/Collaterals/Proceedings/2018/20180808_NVME- 201-2_Yung.pdf The Modernization of PCIe Hot-Plug in Linux https://lwn.net/Articles/767885/ * Requires member access to the relevant standards body website
Linux Enablement Feature Patch Link Add Error Disconnect Recover (EDR) support https://patchwork.kernel.org/cover/10833723/ DPC Add _OSC based negotiation support for DPC https://patchwork.kernel.org/patch/10833717/ Containment Error Recovery (CER) Add Error Disconnect Recover (EDR) ACPI notifier support https://patchwork.kernel.org/patch/10833725/ Add Error Disconnect Recover (EDR) support https://patchwork.kernel.org/patch/10833721/ Implement support for _HPX Type 3 tables https://patchwork.kernel.org/cover/10843875/ Do not export pci_get_hp_params() https://patchwork.kernel.org/patch/10843877/ Hot-Plug Parameter Extensions (HPX) https://patchwork.kernel.org/patch/10843887/ Remove the need for 'struct hotplug_params Implement Type 3 _HPX record https://patchwork.kernel.org/patch/10843883/ Advertise HPX type 3 support via _OSC https://patchwork.kernel.org/patch/10855469/
Architected for Performance Questions?
