Enhancing Risk Analysis Quality for Effective Decision-Making
This content delves into improving the quality of analyses supporting risk management, emphasizing the importance of defining analysis quality comprehensively for decision makers, fostering a culture of analysis quality, and ensuring practical application on the ground. The Applied Risk Management Specialty Group aims to bridge the gap between risk management and risk analysis practitioners, promoting collaborative research and effective risk management approaches to address complex global issues.
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. Download presentation by click this link. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
E N D
Presentation Transcript
Applied Risk Management Specialty Group Debrief Version: - Participant Responses: Slide 18-20 - Vote Results: Slide 21 How Do We Improve the Quality of Analyses Supporting Risk Management? Three Questions: How do we define analysis quality to cover all we want to cover, and in terms non-risk-analyst decision makers can appreciate and understand? How do we create a Culture of Analysis Quality where non-risk-analyst decision makers insist on that analysis quality? How do we do that in a way that gets applied on the ground, NOT in an unread, stays-on-shelf Standard ? Applied Risk Management Specialty Group SRA Webinar, September 4, 2019 1 Analytic Support for Practical Risk Management
Webinar Outline Applied Risk Management Specialty Group Background and Context Steve Ackerlund, Founding Chair Current Analysis Quality Test Battery Questions and Reflections John Lathrop, Past Chair Ongoing Initiatives in ARMSG Willy Roed, Current Chair 2 Analytic Support for Practical Risk Management
Applied Risk Management Specialty Group Applied Risk Management Specialty Group Seeks to make the expertise of the SRA community more accessible and usable to real-world practitioners Relevant for all SRA members 3 Analytic Support for Practical Risk Management
Bridge the gap towards risk analysis practitioners Applied Risk Management Specialty Group Establish SRA as the preeminent clearinghouse bridging risk management with risk analysis tools Attract risk managers to SRA Encourage collaborative and interdisciplinary research on applied risk management concepts, terminology, and frameworks Promote risk management as an effective approach for addressing a wide range of technically and socially complex problems facing the world today 4 Analytic Support for Practical Risk Management
History of ARMSG and the Analysis Quality Tests Initiative Applied Risk Management Specialty Group 2013: ARMSG Founded 2015: SRA Roundtable Identifying and Promoting Core Knowledge in Risk Management no one ring to rule them all , but There are core principles. There is a need for improved methods of establishing quality SRA can and should provide this global leadership! The Roundtables continue: 2016, 2017, 2018, and the next one in 2019. And the interleaved sequence of webinars 2016, 2017, 2018, and now this one. John Lathrop, chief architect, Analysis Quality Tests Willy to conclude with other synergistic initiatives and an invitation to join in the fun! 5 Analytic Support for Practical Risk Management
Webinar Outline Applied Risk Management Specialty Group Background and Context Steve Ackerlund, Founding Chair Current Analysis Quality Test Battery Questions and Reflections John Lathrop, Past Chair Ongoing Initiatives in ARMSG Willy Roed, Current Chair 6 Analytic Support for Practical Risk Management
Applied Risk Management Specialty Group Title SlideDebrief Version: - Participant Responses: Slide 18-20 - Vote Results: Slide 21 How Do We Improve the Quality of Analyses Supporting Risk Management? Three Questions: How do we define analysis quality to cover all we want to cover, and in terms non-risk-analyst decision makers can appreciate and understand? How do we create a Culture of Analysis Quality where non-risk-analyst decision makers insist on that analysis quality? How do we do that in a way that gets applied on the ground, NOT in an unread, stays-on-shelf Standard ? Applied Risk Management Specialty Group SRA Webinar, September 4, 2019 7 Analytic Support for Practical Risk Management
Now, Diving In To The Analysis Quality Test Battery Applied Risk Management Specialty Group Motivation Participated in 5 Third Party Reviews of major risk assessments, BTRA, RNTRA, RAPID, 2 others (sensitive). And comparison of five rad-nuc terrorism risk assessments. In most cases the risk assessments were blow-you-away fantastic analyses, but in many cases the effectiveness of advice to non-risk-analyst decision makers in their real world could be improved. Many other similar experiences by several members of the specialty group. So we all said Hey! What can we do about this? We re the Applied Risk Management Specialty Group, so . 8 Analytic Support for Practical Risk Management
Start With Question 1: How Define Analysis Quality? to cover all we want to cover, in terms non-risk-analyst decision makers can appreciate and understand? Simple: We just assembled all the shortfalls we have observed, wrote a criteria corresponding to each shortfall, worded to be meaningful to a non-risk-analyst decision maker, re-worded that criteria as a question, Does the analysis meet that criteria, with an answer Yes, No, or Does Not Apply, then combined all those questions. Example: Shortfall: The goal of the analysis is not made clear to all (analysts, decision makers, agencies, stakeholders). Corresponding analysis quality category: AQT 1. Clarity of the goal of the analysis. Is the goal of the analysis clearly described and announced? So that all parties can work toward that same goal without special communication. Is the risk/cost of falling short of that goal described? So that all parties are appropriately motivated to achieve that goal. Example goals: to assure a safe design, to develop a safe design, to select the best design, to demonstrate the level of safety to others, to evaluate insurance or risk management policies. Applied Risk Management Specialty Group For each question (note there are two here): Require one of three answers/answer pairs: Yes (with implications for risk management decision making if called for) No (with implications for risk management decision making required) Does Not Apply (though in cases when there could be an argument whether or not it is DNA, include any necessary explanation) 9 Analytic Support for Practical Risk Management
Start With Question 1: How Define Analysis Quality? to cover all we want to cover, in terms non-risk-analyst decision makers can appreciate and understand? Applied Risk Management Specialty Group Four principles we stumbled across developed as we did that: Full Disclosure: No analysis can satisfy all the requirements we want them to satisfy, so we just want to make decision makers aware of the inevitable analysis shortfalls, and their implications for the decisions the DM is facing. Other Decision Factors: Any real-world decision maker has many factors to factor in to his-her decision. So our job as analysts is to make the risk-related aspects clear and understandable to the DM, so he-she can factor those aspects in together with all the other decision factors. The Dodge: We, at this level, can t hope to cover all aspects, e.g. Validation at a depth adequate for each SRA risk analysis domain, so we word things in Dodge Language, e.g., AQTB # 58: Is the model and analysis fully validated, by normal standards of validation in the domain of practice that applies? Cast the Net Wide: We backed in to a very broad scope of Analysis Quality. Not because we wanted to, but because by the time we addressed all the shortfalls we ve observed, we found ourselves staring at a long list: 10 Analytic Support for Practical Risk Management
Our Extremely Broad Definition of Analysis Quality, p. 1 of 2 Applied Risk Management Specialty Group Cast the Net Wide: We backed in to a very broad scope of Analysis Quality. Not because we wanted to, but because by the time we addressed all the shortfalls we ve observed, we found ourselves staring at a long list: A. Framing the Analysis and Its interface With Decision Making Clarity of goals, of decisions to be supported, diverse perspectives, position in org chart, embedding in decision process, DM focus, decision-aiding results formats, adequate breadth-depth-detail, societal and stakeholder acceptability. B. Capturing the Risk Generating Process (RGP) Comprehensiveness of hazards-events-scenarios, capture complexity, detecting early warnings, consider systems changes. C. Communication integrated into risk analysis, communication science to scope the problem, between analysts, DMs, stakeholders. D. Stakeholder Involvement Identification, consulted, engaged, needs/interests/perspectives, involvement in decision process, in implementation. E. Assumptions and Scope Boundary Issues All assumptions-scope issues listed and their implications for decisions clearly explained. You can download our current draft AQTB from the Applied Risk Management website, right nav bar. F. Pro-Active Creation of Alternative Courses of Action G. Basis of Knowledge Characterize sources of knowledge, strengths, limitations, consider Black Swans and unlikelies, conflicting opinions. H. Data Limitations Analyze data limitations and report their implications, manage data (with QA/QC), verify data, validate data. 11 Analytic Support for Practical Risk Management
Our Extremely Broad Definition of Analysis Quality, p. 2 of 2 Cast the Net Wide: We backed in to a very broad scope of Analysis Quality. Not because we wanted to, but because by the time we addressed all the shortfalls we ve observed, we found ourselves staring at a long list: Applied Risk Management Specialty Group I. Analysis Limitations Describe all analysis limitations and their implications for decision advice, verify all calcs, sensitivity analysis, metric validity. J. Uncertainty Aleatory uncertainty: characterize in DM-understandable terms, propagate them into results, in DM-understandable terms. Epistemic uncertainty: characterize in DM-understandable terms, propagate them into results, in DM-understandable terms. Model uncertainty: characterize in DM-understandable terms, propagate them into results, in DM-understandable terms. Combined uncertainty: characterize in DM-understandable terms, propagate them into results, in DM-understandable terms. Make the implications of that combined uncertainty for decision making clear, understandable to DMs. K. Consideration of Alternative Analysis Approaches Identify any plausible alternative approaches and make clear to DMs the implications of choosing an alternative approach. L. Robustness and Resilience of Action Strategies Examine the need for robustness and resilience, test that recommended strategies include adequate robustness-resilience. M.Model and Analysis Validation and Documentation Validate the model and analysis, document that validation for third party review. N. Reporting Define all key terms, explain and motivate all results without using abstract terms, explain unexpected results. O. Budget and Schedule Adequacy 12 Analytic Support for Practical Risk Management
And Theres More: Domain-Specific Analysis Quality Tests Applied Risk Management Specialty Group Cast the Net Wide: We backed in to a very broad scope of Analysis Quality. Not because we wanted to, but because by the time we addressed all the shortfalls we ve observed, we found ourselves staring at a long list: P. Adversary Modeling Characterize and model adversaries with the most appropriate adversary model. Populate those adversary models with the best available data, SMEs, and elicitation. Model adversary reactions to defensive courses of action. Model adversary acquisition of information and capabilities. Account for Deliberate Black Swans. The Applied Risk Management Specialty Group has other domain-specific AQTs coming: Q. Environmental Restoration (Steve Ackerlund) R. Oil & Gas Development (Willy Roed) S. Cultural Heritage Risk Management (Rob Waller). T. Your Specialty Group s Domain-Specific Analysis Quality Tests Here! 13 Analytic Support for Practical Risk Management
We Define Analysis Quality Extremely Broadly, Since it is Shortfall Based All-Domain Analysis Quality Test Categories Each category has 1-9 criteria, each worded as a question: Does the analysis adequately meet that criteria? Yes-No-DNA. Applied Risk Management Specialty Group A. Framing the Analysis and Its interface With Decision Making B. Capturing the Risk Generating Process (RGP) C. Communication D. Stakeholder Involvement E. Assumptions and Scope Boundary Issues F. Pro-Active Creation of Alternative Courses of Action G. Basis of Knowledge H. Data Limitations I. Analysis Limitations J. Uncertainty K. Consideration of Alternative Analysis Approaches L. Robustness and Resilience of Action Strategies M.Model and Analysis Validation and Documentation N. Reporting O. Budget and Schedule Adequacy Domain-Specific Analysis Quality Tests P. Adversary Modeling Q. Environmental Restoration R. Oil & Gas Development S. Cultural Heritage Risk Management T. Your Specialty Group s Domain-Specific Analysis Quality Tests Here! Roger Flage s list from SRA 2018, his parts of Prudent Risk Analysis: Risk Characterization Risk Perception and Communication Risk Assessment Risk Management and Governance Policy Relating to Risk You can download our current draft AQTB from the Applied Risk Management website, right nav bar. 14 Analytic Support for Practical Risk Management
But Thats Such a Large Test Battery! Cast the Net Wide: We backed in to a very broad scope of Analysis Quality. Not because we wanted to, but because by the time we addressed all the shortfalls we ve observed, we found ourselves staring at a long list: Applied Risk Management Specialty Group Yep. At 63 + questions, PLUS domain-specific questions, the AQTB may seem burdensome. But: These questions were assembled from our long list of shortfalls we ve encountered in our experience. So if we drop any questions we are letting the corresponding shortfalls go unchecked. Do we really want to drop questions and say that we re not checking some shortfalls because it s just too hard to check for them? For any particular analysis, probably a large number of questions will be DNAs. A conscientious and smart review process by analysts involved with the risk analysis should be able to answer all the questions within a reasonable amount of time and effort. The time and effort to answer the AQTB will be quite small relative to the typical stakes of risk management decisions for which risk analyses are commissioned. We want to encourage a Culture of Analysis Quality where the funder (and his reviewers and stakeholders) will feel it is part of Best Practice to set aside funds to complete the AQTB, so it will be performed. (Actually, it should be part of minimum acceptable practice.) 15 Analytic Support for Practical Risk Management
And Thats The Easy Part! Hard Part: Qs 2 & 3 From Title Slide: Applied Risk Management Specialty Group How do we create a Culture of Analysis Quality where non-risk-analyst decision makers insist on that analysis quality? How do we do that in a way that gets applied on the ground, NOT in an unread, stays-on-shelf Standard ? We ve created an Analysis Quality Test Battery. But that could just sit on a shelf. So now what? 16 Analytic Support for Practical Risk Management
Our Plan To Create That Culture, To Avoid Shelf-Sitting: Applied Risk Management Specialty Group We convert the AQTB (available in Word on the ARMSG website) to a computer-based questionnaire. We ve received funding from SRA (thank you) to have that computer-based questionnaire programmed in software, to collect all responses: yes(implications) No(implications) DNA(with justification if called for) and process that into a report. That s great: Then, if the questionnaire is filled out and reported, risk management decision makers will understand all the shortfalls in the analysis and the implications of those shortfalls for their decisions. Wow. OK, so that s the techie side. But it s 63 (+) questions. Who is going to fill that out? How do we get that to happen? Hard Question 1: How do we create Situation 1: Where risk management DMs will want to require an AQTB evaluation for any risk analysis they commission. Hard Question 2: How do we create Situation 2: Where risk analysts will want to offer to conduct an AQTB evaluation. Now that we ve presented all the doable stuff as described in the previous slides, we now have to face the hard questions, Hard Question 1 and Hard Question 2. So we now turn the hard questions over to this webinar s participants: How can SRA help to bring about those two situations, Situation 1 and Situation 2? (and so a Culture of Analysis Quality ) Those are not rhetorical questions. Let s open up this webinar to discussion. Awaiting your answers . 17 Analytic Support for Practical Risk Management
Participant Ideas, All Related to Hard Question 1 Applied Risk Management Specialty Group Hard Question 1: How do we create Situation 1: Where risk management DMs will want to require an AQTB evaluation for any risk analysis they commission. Participant A: Lobby for AQTB to have legal implications so that DM's must use it. We Respond: Great idea but maybe not doable. We ll talk with the Risk Policy and Law Specialty Group about this. Participant B: Perhaps even more important to legal liability is demonstrating reasonable (good quality) risk management. Will this feed into Risk Management Quality Tests? We Respond: Whoof. We ve been so focused on analyses in support of risk management, we haven t thought about risk management quality ITSELF. Not at all a straightforward extension. We ll have to chew on this. Participant A: Show that AQTB users consistently reduce their risk to such a degree that insurance provides will "encourage" their customers to use it. We Respond: Whoof again. Great idea, but also suggests we track AQTB use and outcomes. Depending on how AQTB is implemented, that could be doable (with anonymizing) for AQTB-using cases. BUT we d have to compare AQTB-using risk to AQTB-not-using risk frankly, sounds like too big of an ask, at least until some Phase II. Participant C: Why not share the problem formulation with the decision makers? We Respond: We believe we ve more or less covered that in our Embedding in the Decision Process: Is the risk analysis fully and effectively engaged with the risk management decision makers. Though that is not worded to explicitly make your point. So OK, we ll tweak or add wording to make your point. Good call. 18 Analytic Support for Practical Risk Management
Participant Ideas, All Related to Hard Question 1 Applied Risk Management Specialty Group Hard Question 1: How do we create Situation 1: Where risk management DMs will want to require an AQTB evaluation for any risk analysis they commission. Participant D: Would think regulators would be advocates of this - in the UK finance sector for example, it's principles-based regulation where the organisations have to demonstrate robustness around their approaches for risk assessment and analysis. Anything to support this would likely be welcomed? We Respond: We notice we have Specialty Group chairs with emails with epa.gov, dot.gov, and fda.hhs.gov. So we ll drop them a line about this. Thanks. 19 Analytic Support for Practical Risk Management
Participant Ideas, More Broad Than HQ1: Participant E Applied Risk Management Specialty Group Audio comments made during webinar: Can use the Analysis Quality Test Battery as a guide for developing assessments. In order to encourage people to use the document, in addition to noting it as an evaluation, frame the use of it as a guide for developing assessments. We Respond: Great Idea! We will push for some way for SRA senior officers to push this. I support use of incorporating the questionnaire into a computer format. This way, modules corresponding to the current sections can be developed (e.g., allow users the ability to click-for-more-information or examples). We Respond: Again, great idea. Includes expanding the role of the AQTB to be a living-doc repository of information usefully retrievable. Though SRA acceptance could be a problem, since this will take person-hours and an SRA approval (curation) process. As users answer questions, the tool can also serve to capture and document those responses to prepare a document / file that clearly defines Battery responses. This can serve as the document/file submitted for review (in the event the AQT Battery becomes a requirement. We Respond: We had ideas along this line re an auto-report, but you may be raising the new idea, thank you, of the tool accumulating data across applications of the AQTB, suitably anonymized. Additional comments: Suggested reporting requirements should consider reporting conflict of interests and funding disclosures. We Respond: Good idea. We'll add that. You have just added yourself to the author list. Determine how the tool should be peer reviewed so that it gains acceptance by researchers outside of SRA (so that it is not only reviewed internally). We Respond: Hmm. We'll think about how, exactly, to make that happen. This we'll kick upstairs to SRA senior officers. 20 Analytic Support for Practical Risk Management
Lets Take Some Votes! Much work remains before we are happy with this Analysis Quality Test Battery, but once we are happy with it: Applied Risk Management Specialty Group We will of course at least publish the AQTB on the SRA website. But in addition, should we also set up any combination of the following five modes?: Vote: 56% Yes ; 44% No of 36 votes (of 62 participants) A. Publish the AQTB in every issue of Risk Analysis? Note: B, C, and D all have the benefit of getting analysts used to the AQTB, and so becoming aware of all the considerations raised in the AQTB. B. Require, that any paper submitted to Risk Analysis that describes an analysis to support risk management decision making be reviewed through this AQTB process. That is: Authors must complete the AQTB, then referees must review the AQTB report for acceptability. Vote: 55% Yes ; 45% No of 33 votes (of 62 participants) C. Require, that any presentation submitted to an SRA conference that describes an analysis to support risk management decision making be reviewed through this AQTB process. We wouldn t require that the review be presented, but simply that the AQTB be conducted and the resulting report submitted with the presentation. D. Require, that any poster submitted to an SRA conference that describes an analysis to support risk management decision making be reviewed through this AQTB process. We wouldn t require that the review be presented, but simply that the AQTB be conducted and the resulting report submitted with the presentation. E. Set up a process of SRA accreditation that the AQTB process was completely and validly applied to any specific analysis. That would involve let us say five days of an SRA analyst s time, and so would involve a fee to cover that time, perhaps about $8,000 + travel. It would involve some process of applying an SRA Seal of Approval. Note that the Seal of Approval would not attest that all applicable Analysis Quality Tests scored a Yes or a DNA, but that all implications for risk management decision making of each Yes or No are adequately reported and described to decision makers, and that any arguable DNA is adequately justified. 21 Analytic Support for Practical Risk Management
Now Lets Back Up to More Doable Things: Applied Risk Management Specialty Group This should be an SRA-wide Activity. In fact, it should be a very valuable service performed by SRA. So therefore, the more specialty groups who participate the better. Note that a specialty group can: - comment on or add to the 63 all-domain questions - and contribute its own domain-specific questions. So far, we have four specialty groups signed up to contribute: Said "Yes" 1 Advanced Materials and Technologies James Ede 2 Applied Risk Management 3 Decision Analysis and Risk 4 Dose Response And so we have three questions: What would be the easiest / most effective way for Specialty Groups to participate? For any specialty groups represented at this webinar who have not yet signed up, would you like to sign up now? For those not-yet-responded specialty groups, what can we do to encourage you to participate? Note that if you sign up, all you are signing up to do is to contribute to the Analysis Quality Test Battery. You don t have to help us solve the Hard Questions of Slide 17: How to get DMs to require running an AQTB. Willy R ed Ben Trump Ingrid Druwe Amanda Bailey Diane Henshel Deborah Aiken Hiba Baroud Amina Wilkins Chris Greene Roger Flage Amir Mokhtari Mary O'Reilly Myriam Merad Vanessa Schweizer Jamie Wardman Yvonne Stevens Sweta Chakraborty Cameron MacKenzie 2016 5 Ecological Risk Assessment 8/25/19 6 Economics and Benefits Analysis 7 Engineering and Infrastructure 8 Exposure Assessment 8/29/19 9 Foundational Issues in Risk Analysis 10 Microbial Risk Analysis 11 Occupational Health and Safety 12 Resilience Analysis 13 Risk and Development 14 Risk Communication 8/19/19 15 Risk Policy and Law 16 Security and Defense 22 Analytic Support for Practical Risk Management
Please Send Us Your Ideas! Applied Risk Management Specialty Group As should be clear by now, we need your ideas! So please email those ideas to me at jlathrop@innovativedecisions.com As mentioned, the current draft of the Analysis Quality Test Battery is available on the Applied Risk Management website, right hand nav bar. If you want a copy of these slides, email me at the above email address. 23 Analytic Support for Practical Risk Management
Webinar Outline Applied Risk Management Specialty Group Background and Context Steve Ackerlund, Founding Chair Current Analysis Quality Test Battery Questions and Reflections John Lathrop, Past Chair Ongoing Initiatives in ARMSG Willy Roed, Current Chair 24 Analytic Support for Practical Risk Management
AQT Battery Hearing Process Applied Risk Management Specialty Group Activity Date/ deadline Send the text to SRA members and Specialty Groups Specialty groups: Performed Members: Next newsletter Feedback from members and Specialty Groups October 1st Update of document based on comments November 1st SRA Council discussions and potential approval as an SRA document November Presentation SRA Annual Meeting, December 2019 25 Analytic Support for Practical Risk Management
Ongoing Initiatives in ARMSG Applied Risk Management Specialty Group Activity Status/ Plan Analysis Quality Test Battery document Finished in 2019 (if accepted by council) Analysis Quality Test Battery software tool 2020. (Working on specifications now) Applied Risk Management examples Continuous activity. See publications at www.sra.org/ARMSG Roundtable Annual Meeting 2019 1) How do we best improve quality of analyses supporting risk management decisions? 2) How do we make SRA most useful to real-world, risk management decision makers? 3) How do we attract those decision makers into SRA? LinkedIn Group To be launched at Annual Meeting 2019 26 Analytic Support for Practical Risk Management
Are you interested in Applied Risk Management? Applied Risk Management Specialty Group Do you have suggestions or other feedback? Would you like to contribute to ARMSG? Chair Elect Nomination? (election in December) Executive Committee: Chair: Willy R ed (wr@proactima.com) Past Chair: John Lathrop (jlathrop@innovativedecisions.com) Secretary: Patricia Larkin (plarkin@xplornet.com) Advisors: Steve Ackerlund (WAckerlund@ene.com), Ronald Dyer (ronald.dyer@sheffield.ac.uk) Rob Waller (rw@protectheritage.com) 27 Analytic Support for Practical Risk Management
