Black Swans and Building Resilient Test Culture
Black Swans are rare unexpected events that can have a significant impact. Resilience is the ability to quickly recover from difficulties and adapt. Understanding these concepts is crucial in building a strong test culture, as demonstrated by Colonel Steven W. Speares. Explore why Verification & Validation (V&V) is essential for complex systems and how failures like the Ariane 5 launch emphasize the importance of thorough testing to prevent catastrophic losses and ensure system integrity.
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
You are allowed to download the files provided on this website for personal or commercial use, subject to the condition that they are used lawfully. All files are the property of their respective owners.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.
E N D
Presentation Transcript
Diversity of Thought: How Black Swans Help Build a Resilient Test Culture Colonel Steven W. Speares Air Force Test & Evaluation Phone: 703 697-0190 or DSN 227-0190 E-mail: steven.speares@us.af.mil I n t e g r i t y - S e r v i c e - E x c e l l e n c e
The Basics Black Swan event A rare, unexpected event that drives large impact or consequence After it occurs, hindsight dictates that it should have been expected, because the data were there but unaccounted for Resilience The capacity to recover quickly from difficulties; toughness The ability of something to spring back into shape; elasticity I n t e g r i t y - S e r v i c e - E x c e l l e n c e
The Stage Setter Part 1 2019 FAA V&V Conference Why Verification & Validation (V&V)? I n t e g r i t y - S e r v i c e - E x c e l l e n c e
Why V&V? High-fidelity simulation Cyber ranges Threat ranges Open-air ranges Software factories Public domain Top-Level Requirements Operational Test System Design System & SoS Test Subsystem Design Subsystem Test Wind tunnels Anechoic chambers Engine test stands Arc heaters Acceleration ranges Test tracks Software integration labs Software factories Hardware in the loop facilities Component Design Component Test Fabrication The Systems Engineering V Test is integral to the systems engineering process Complex systems won t be resilient without testing them I n t e g r i t y - S e r v i c e - E x c e l l e n c e
5 Ariane 5 (4 Jun 1996) First launch of Ariane 5 $500M loss - broke up 37 sec into flight S/W failure commanded errant engine response Ariane 4 software reused on Ariane 5 Inertial reference system (IRS) components not needed, but retained for commonality Validation Failure of S/W systems reqmts No overflow exception handler on unused IRS No test of overflow condition because no reqmt Insufficient systems-level testing Component level tests IRS testing used simulated data, not real (HITL) data Recommendation: Test not just what the system should do, but what it should not 37 sec I n t e g r i t y - S e r v i c e - E x c e l l e n c e
6 Mars Polar Lander (3 Dec 1999) All parts/components worked as designed Software response to transient signals Inadequate testing Flight software not subjected to fault-injection testing Touchdown S/W not tested in landing configuration Recommendation: Reinforce the system-level test principle of test as you fly, and fly as you test. Faster, better, cheaper was the culprit (JPL PM) I n t e g r i t y - S e r v i c e - E x c e l l e n c e
Why Resilient V&V? High fidelity simulation Cyber ranges Threat ranges Open air ranges Software factories Public domain Top-Level Requirements Operational Test System Design System & SoS Test Subsystem Design Subsystem Test Wind tunnels Anechoic chambers Engine test stands Arc heaters Acceleration ranges Test tracks Software integration labs Software factories Hardware in the loop facilities Component Design Component Test Fabrication The Systems Engineering V Every node is a risk vector Need integrated systems test and fault injection I n t e g r i t y - S e r v i c e - E x c e l l e n c e
The Stage Setter Part 2 2019 FAA V&V Conference Resilience lessons learned and takeaways I n t e g r i t y - S e r v i c e - E x c e l l e n c e
This is 2020: The Good Fundamental shift in the nature of test Platform-specific to platform-agnostic DEVSECOPS, software resilience, and chaos engineering I n t e g r i t y - S e r v i c e - E x c e l l e n c e
This is 2020: The Bad Funding challenges Organizational Inertia Lack of relevant expertise I n t e g r i t y - S e r v i c e - E x c e l l e n c e
This is 2020: The Ugly COVID-19 I n t e g r i t y - S e r v i c e - E x c e l l e n c e
The Reality The New Normal I n t e g r i t y - S e r v i c e - E x c e l l e n c e
The Response Physical Organizational Some positives, caveated success Resilience in action I n t e g r i t y - S e r v i c e - E x c e l l e n c e
The Revelation Cultural Resilience is the Key Resilience in Attitude Breaking resident barriers Symptom of the pandemic (the enabling Black Swan) I n t e g r i t y - S e r v i c e - E x c e l l e n c e
The Resilience Equation I n t e g r i t y - S e r v i c e - E x c e l l e n c e
The Resilience Equation Cultural Resilience = Competence x Confidence A product of acumen & attitude I n t e g r i t y - S e r v i c e - E x c e l l e n c e
The Historic Basis Mizuta Masahide Pasteur Churchill Safety Third Apollo 12 I n t e g r i t y - S e r v i c e - E x c e l l e n c e
The Need The 20s are the new 30s Unmanned systems, Space travel Relativistic effects of the modern world Never know when the next Black Swan will arrive I n t e g r i t y - S e r v i c e - E x c e l l e n c e
The Miracle on the Hudson I n t e g r i t y - S e r v i c e - E x c e l l e n c e
Questions? I n t e g r i t y - S e r v i c e - E x c e l l e n c e
