NASA Cost Research and Risk Analysis Insights
The NASA Strategic Investment Division (SID) presented on methodologies for calculating project reserve dollars, percentile budgeting, and assessing project and portfolio risk exposure. Their work includes developing tools and metrics to analyze and visualize risk exposure, support fund allocation, and communicate within NASA stakeholders. They consolidated deliverables into an Excel worksheet, updated calculations, and created graphical outputs for better analysis. Validation of estimates versus actuals was planned for January 2018.
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. Download presentation by click this link. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
E N D
Presentation Transcript
Washington Capital Area Chapter of ICEAA Calculating a Project s Reserve Dollars from its S-Curve (cost distribution) A method to estimate the funds needed to cover your project if it exceeds its Point Estimate Presenter: Marc Greenberg Strategic Investment Division (SID) National Aeronautics and Space Administration 13 December 2017
Percentile Budgeting is Only Half the Story Risk analysis provides insight into the cost and schedule range and likelihood of achieving the cost and launch dates. Percentile budgeting (choosing a specific confidence level) uses the risk analysis results to ensure that enough money is available in the budget to protect for that likelihood of success, and/or that enough time has been reserved to protect for the target launch dates. Percentile budgeting establishes a protection level, but there is the other dimension of the risk of overrun. A 70% budget indicates that 30% of the time there is an overrun Overruns affect the portfolio & require re-allocation to ensure that the project can continue Current techniques do not provide insight into the likelihood and magnitude of the potential overruns and how it can affect the portfolio NASA Strategic Investment Division (SID) 2
Cost Research Task (2015) NASA-funded work completed by Tecolote Develop techniques to assess the overall risk exposure a project holds at any given time. What is meant by project risk exposure? How is project risk exposure calculated? What are metrics of risk exposure? Develop methodology and associated metrics to determine the risk exposure a program, theme, or directorate (an Agency portfolio) is carrying at any given time. What does portfolio risk exposure mean? How is portfolio risk exposure calculated? How can portfolio risk exposure be used to support fund allocation? Develop a prototype tool to support calculations and visualization of metrics for a project s risk exposure and support communication within NASA stakeholders Conduct a test case on a sample NASA portfolio NASA Strategic Investment Division (SID) 3
Cost Research Task: Tecolote Team Darren Elliott (Lead) Nick Detore (Analyst) Shu-Ping Hu (Statistician) Matt Blocker (Automation & Visualization) NASA Strategic Investment Division (SID) 4
In-house NASA Cost Research (2017) Consolidated 2015 deliverables into a single Excel worksheet Changed calculations from simulated to deterministic Added a data table for enabling sensitivity analysis Added two graphical outputs to worksheet Added a help worksheet to enable user to estimate: a project s average life cycle cost (Avg LCC) a project s cost risk, measured as its coefficient of variation (CV) Derived look-up cost curves & a single equation to estimate UFE Note: Marked as DRAFT slides because currently being updated Will validate calculator (i.e., estimates versus actuals) in Jan. 2018 NASA Strategic Investment Division (SID) 5
Reserves per NASA Policy (aka UFE) During Formulation, the Decision Memorandum shall establish a target life- cycle cost range (and schedule range, if applicable) as well as the Management Agreement addressing the schedule and resources required to complete Formulation. The Decision Memorandum also documents any additional resources beyond those explicitly estimated or requested by the program/project (e.g., additional schedule margin) when the Decision Authority determines that this is appropriate. This includes Unallocated Future Expenses (UFE), which are costs that are expected to be incurred but cannot yet be allocated to a specific WBS sub- element of a program s or project s plan. Management control of some UFE may be retained above the level of the project (i.e., Agency, Mission Directorate, or program). All projects and single-project programs shall document the Agency s life- cycle cost estimate and other parameters in the Decision Memorandum for Implementation (KDP C), and this becomes the Agency Baseline Commitment (ABC). The ABC is the baseline against which the Agency s performance is measured during the Implementation Phase. The ABC for projects with a life- cycle cost of $250 million or more forms the basis for the Agency s external commitment to OMB and Congress. NASA Strategic Investment Division (SID) 6
NASA Baseline Policy- Project-Simplified Cost Agreement at KDP C This UFE is managed above the Project. Agency Baseline Commitment External Commitments UFE At KDP C and subsequent ABC rebaselines the ABC and the life cycle cost estimate are equal. Management Agreement Managed by Project Manager Not to Scale 7 NASA Strategic Investment Division (SID) 7
NASA Baseline Policy- UFE (i.e., Reserves) Managed Above the Project (Includes Project UFE) Management Agreement (MA) (Includes Project UFE) Program / Mission Directorate (MD) UFE Life Cycle Cost (LCC) The reserves managed above the project Occurs throughout Project Life Cycle MA includes Project Managed UFE and Schedule Margin Reflects the integration of cost, schedule, and risk 8 NASA Strategic Investment Division (SID) 8
Why estimate cost reserves (aka UFE)? Enhance PPBE Process. Reserves calculator can be used to: Determine total amount of reserve dollars to be held at Directorate level Gain insight on de-scope implications to reserves and Give stakeholders additional basis for decisions related to reserves & de-scopes Estimate Reserves by Project. Enables estimating the potential extra dollars needed for a project IF an overrun event should occur Extra dollars = amount of funds in excess of Point Estimate Compare Mutually Exclusive Projects. Can do apples-to-apples cost comparison between two or more competing projects Estimate reserves for Portfolio. Enables ability to estimate potential extra dollars needed for portfolio IF overrun events occur in more than one project within the portfolio Extra Dollars = sum of each project s UFE Total reserve amount can then be calculated as % of total project funds Awareness of a project s cost overrun implications (of being at a specific Confidence Level) provides useful information to determine the amount of UFE and/or Technical De-Scope NASA Strategic Investment Division (SID) 9
METHODOLOGY & EXAMPLES NASA Strategic Investment Division (SID) 10
Project / Portfolio Analysis Process Inputs, Analysis & Outputs for Cost Reserves Estimation Method Initial State-of-Knowledge Captured just prior to any of the following milestones: SRR: System Readiness Review PDR: Preliminary Design Review CDR: Critical Design Review SIR: System Integration Review KDP-D: Key Decision Point (just prior to launch) Replan: Can occur at any point between SRR & Launch Inputs Point Estimate (PE) * Confidence Level (CL) Coefficient of Variation (CV) Average Life Cycle Cost (LCC) Analysis Per Project Overall Portfolio Phase 1 Phase 2 New State-of-Knowledge Key Outputs Likelihood of a Cost Overrun for each Project Cost Reserves for each Project ** Effective Confidence Levels for each Project Cost Reserves for the Portfolio ** Effective Confidence Level of the Portfolio ** Optional Iterations: Enter Cost Mitigating Impacts of De-scopes. * Assumes Point Estimate (PE) includes project management reserve ** Assumes technical de-scopes, if any, were exercised NASA Strategic Investment Division (SID) 11
Example for Notional Project * Point Estimate (PE) = $50M at 25% Confidence Level (CL) ** PE = $50M PE = $50M Likelihood of Cost Overrun: 1 CL = 75% For all Costs over $50M, the Average = $70M If overrun occurs This is the Conditional Project Cost (CPC) Ave = $70M 25% 75% 25% LCC ($ x 10,000) LCC ($ x 10,000) Average Cost Overrun = CPC PE = $70M $50M = $20M Expected Average Cost Overrun = Likelihood of Cost Overrun x Average Cost Overrun Expected Average Cost Overrun = 75% x $20M = $15M Recommended Cost Reserve = $15M *** If the project cost exceeds its $50M PE (i.e., exceeds 25% CL), then assigning the PE with $15M in reserve results in an effective PE of $65M with an effective 38% CL *** * Given a Coefficient of Variation (CV) = 17.75% ** $50M Point Estimate (PE) includes project management reserve *** This result precludes mitigating impacts from technical de-scopes NASA Strategic Investment Division (SID) 12
How the reserves calculator was developed 1. Create detailed CDF ( S-curve) where inputs are fit-space values for (i) Average Life Cycle Cost (Avg LCC) and (ii) Standard Deviation (SD) Step a: SD = (Avg LCC) x (Coefficient of Variation) [i.e., Values are unit space to start off] Step b: Fit-space Variance = LN(1 + (SD / Avg LCC) 2) Step c: Fit-space SD = SQRT (Fit-space Variance) Step d: Fit-space Avg LCC = LN(Avg LCC)-0.5*Fit-space Variance Step e: =LOGNORM.INV (Confidence Level, Fit-space Avg LCC, Fit-space SD) Produces dollar values based upon 10,000 confidence levels (sorted from 0% to 100%) Step f: Plot confidence levels versus respective dollars to create a CDF (aka S-curve ) Create detailed histogram to approximate a PDF. 1000 equal intervals along the given lognormal distribution Add a conditional column on histogram. can be used to: If the mid-point value, V(i), of the given interval > PE, then V(i) PE, else $0. Take the SUMPRODUCT of #2 and #3 to estimate cost reserve $ This is the expected on average dollar amount to hold in reserve above the project Obtain confidence levels of PE and (PE + Reserves) using: LOGNORM.DIST(PE,LN(Avg LCC),(Fit-space SD/Fit-space Ave LCC)*LN(Avg LCC),TRUE),1) Create data table to see UFE sensitivity from changes in PEs and CVs 2. 3. 4. 5. 6. NASA Strategic Investment Division (SID) 13
Overview of the LCC & CV calculators 1. The Average Life Cycle Cost Calculator (excludes Phase E & F costs) is based upon three cost estimating relationships (CERs): Planetary Missions CER Earth Science and Astrophysics Missions CER (same equation) Heliophysics Missions CER 2. The Cost Risk Ballpark Calculator is based upon methods described in a 2007 paper entitled Estimating Cost Uncertainty when only Baseline Cost is Available by LaserLight Networks. a The paper included an example by Abramson and Book: applying a three point method to produce a beta cost distribution of a technology 3. The Cost Risk Reference List is an approximation of the coefficient of variation for each of 50 NASA projects. Each approximation is calculated based upon cost growth from formulation through launch. Determine total amount of reserve dollars to be held at Directorate level (a) Within USAF Methodology section of Estimating Cost Uncertainty when only Baseline Cost is Available. Can be downloaded at: http://www.laserlightnetworks.com/Documents/Estimating%20Cost%20Uncertainty%20when%20Only%20Baseline%20Cost%20is%20Available.pdf NASA Strategic Investment Division (SID) 14
Example Inputs and Outputs (1 of 3) Assume you ve been provided Project s PE of $3,500 (RY$M) Assume an Independent Cost Estimate (ICE) provides you an S-curve which enables you to obtain: the project s average life cycle cost (Avg LCC) = $4,000 (RY$M) the project s cost risk ( coefficient of variation , CV) = 40% (High Risk) Enter values in GREEN cells! Budget UFE Reserves (BUFER) Calculator for All Costs are in RY$M Electro-Nuclear Geosynchronous Observation Instrument (E-NGOI) Coef of Var (CV): 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% >50% ------------> ------------> ------------> ------------> Very High Risk ------------> Very Low Risk Low Risk Nominal Risk High Risk Neglible Risk ----> Coefficent of Variation (must be at least 5%): Average Life Cycle Cost (Avg LCC) = Project Point Estimate (PE) = 40% $4,000 $3,500 0.8750 The project cost exhibits a High Risk The Average LCC is at the 50% confidence level Notes on the Coefficient of Variation (CV) CV is "spread" of +1 std dev above Ave LCC. One s towards right tail is at the 84% CL. PE / Avg LCC = The Project PE is equal to 87.5% of the project's Ave LCC. All CV entries > 50% are "Very High" Risks Refer to Lookup Table for additional guidance Unallocated Funding Expenditure (UFE)= $849 "On average" funds needed for scenarios when the PE > $3,500 Project's Point Estimate (PE) = $3,500 $4,349 The Project PE is at the 36.6 % confidence level. The (PE + UFE) is at the 58.5 % confidence level. The likelihood of exceeding the Project's PE = 63.4% The likelihood of exceeding the (PE + UFE) = 41.5% The (PE + UFE) is 349.3 $M over the Ave LCC of 4000 $M PE + UFE = If the project cost exceeds its $3.50B PE (i.e., exceeds 37% CL), then assigning the PE with $849M in reserve results in an effective PE of $4.35B with an effective 59% CL NASA Strategic Investment Division (SID) 15
Example Inputs and Outputs (2 of 3) You can view the Project s PE with & without reserves in 2 ways (CDF & PDF): 100% 90% PE = $3500M (36.6% CL) 80% Cumulative Confidence Level Avg LCC = $4000M (50% CL) 70% $4349.3M = PE + $849.3M UFE (58.5% CL) 60% 50% 40% 0.8% 30% PE = $3500M (36.6% CL) 0.7% 20% Avg LCC = $4000M (50% CL) 10% 0.6% 0% $4349.3M = PE + $849.3M UFE (58.5% CL) 0.5% $0 $5,000 $10,000 $15,000 $20,000 $25,000 $30,000 0.4% 0.3% 0.2% 0.1% 0.0% $0 $5,000 $10,000 $15,000 $20,000 $25,000 $30,000 NASA Strategic Investment Division (SID) 16
Example Inputs and Outputs (3 of 3) During formulation, Project PE and its Cost Risk are uncertain Sensitivity analysis helps address such uncertainties by: varying the project s PE ($3.5B) to cover the range from $3.15 - $3.85B varying the project s cost risk (CV = 40%) to go from 30% to 50% Nominal to High Risk High to Very High Risk 2-way Sensitivity Analysis Nominal Risk High Risk Very High Risk 30% 35% 40% 45% 50% UFE Estimate = $849 $3,150 $3,238 $3,325 $3,413 $965 $902 $841 $782 $727 $674 $624 $577 $533 $1,013 $953 $896 $841 $788 $738 $691 $646 $603 $1,063 $1,006 $951 $899 $849 $802 $756 $713 $672 $1,113 $1,059 $1,007 $957 $909 $864 $820 $778 $738 $1,163 $1,111 $1,061 $1,014 $968 $924 $881 $841 $802 Cost < PE Current PE = $3,500 $3,588 $3,675 $3,763 $3,850 Cost > PE If, for example, the PE stays at $3.50B PE, but tech scope causes cost risk to go to Nominal-to-High , the UFE would go from $849M to $788M, a savings in $61M. The effective PE (now reduced by $61M) goes to $4.29B with an effective 58% CL NASA Strategic Investment Division (SID) 17
DEMONSTRATION NASA Strategic Investment Division (SID) 18
Look-up Curves to Estimate UFE Calculated multiple UFE values to create reserve curves Varied PE values from 77.5% to 99.5% of Avg LCC Varied Cost Risk values where CVs went from 10% to 50% Example: You are asked to estimate the reserves for a project with a $3.5B Point Estimate (PE). (UFE / Point Estimate) versus (Point Estimate / Average Life Cycle Cost) 48% Based upon the given ICE s S- curve, this PE is at the 33% CL. The Average LCC (on the same S-curve) is $4.0B. 44% UFE relative to the Point Estimate (PE) 40% 36% All SMEs concur that this project has a High Cost Risk (CV ~ 40%). 32% Cost Risk: 28% Very High 24% High Per the High curve, the x-axis (PE / Avg LCC) = ($3.5 / $4.0) = 0.875 or 87.5% 20% Nominal 16% Low The corresponding value on the Y-axis is 27% = (UFE / PE) 12% Very Low 8% Therefore: UFE = 0.27 x $3.5B UFE = $945M 4% 76% 78% 80% 82% 84% 86% 88% 90% 92% 94% 96% 98% 100% Point Estimate (PE) relative to Average Life Cycle Cost (Avg LCC) NASA Strategic Investment Division (SID) 19
Look-up Curves & Single UFE Equation Reserve curves may help frame more specific UFE guidance (UFE / Point Estimate) versus (Point Estimate / Average Life Cycle Cost) 48% Low Cost Risk: 8% - 22% UFE Nominal Cost Risk: 12% - 25% UFE High Cost Risk: 17% - 30% UFE Very High Cost Risk: 22% - 35% UFE 44% UFE relative to the Point Estimate (PE) 40% 36% 32% Cost Risk: 28% Very High 24% High 20% Nominal 16% Low 12% Very Low 8% 4% 76% 78% 80% 82% 84% 86% 88% 90% 92% 94% 96% 98% 100% Point Estimate (PE) relative to Average Life Cycle Cost (Avg LCC) A single UFE equation may be a useful add-on to existing cost models a UFE = PE * [ R^2 (-96.2500 x^4 + 129.5100 x^3 - 58.6310 x^2 + 8.5494 x + 2.0751 + R (159.1800 x^4 - 210.5100 x^3 + 91.2570 x^2 - 11.4480 x - 5.0895 - 64.0580 x^4 + 82.6120 x^3 - 33.4060 x^2 + 3.4852 x + 3.0009 Where: UFE = Unallocated Future Expense (UFE) in RY$M. R = (Point Estimate / Avg Life Cycle Cost). x = Project s Cost Risk (measured as a Coefficient of Variation). PEis the Project s Point Estimate, RY$M. (a) Single UFE equation shown is for Point Estimates that are less than the Avg Life Cycle Cost. NASA Strategic Investment Division (SID) 20
Look-up Curves & Single UFE Equation Notional example on how the model calculates UFE using 3 inputs The model provides an estimate of UFE based upon 3 Inputs: - Point estimate (PE). Typically the project s estimated life cycle cost. PE = $3,500M - Average life cycle cost (Ave LCC). Typically based upon similar missions. Ave LCC = $4,000M - Life cycle cost spread (Cost Risk). Driven by known and unknown risks. Nominal/High Cost Risk For Notional Example: 100% Example: UFE estimate for a $3.5B mission (UFE / Point Estimate) versus (Point Estimate / Average Life Cycle Cost) 50% 90% Cost Risk Levels: 45% UFE relative to the Point Estimate 80% V_Low Low Nom Hi V-Hi 40% How much UFE would cover the project if, at a later time, the cost exceeds it s current PE of $3,500M? 70% 35% Confidence Level PE is at a 33% Confidence Level. 60% Ave LCC = $4,000M 30% 50% 25% UFE / PE = 25% (Derived) 40% 20% so there s a 67% chance that the project s life cycle cost is greater than $3,500M. PE = $3,500M 15% 30% 10% 20% 5% 10% PE / Ave LCC = 87.5% (Given) 0% 0% 75% 80% Point Estimate relative to Average Life Cycle Cost (LCC) 85% 90% 95% 100% $0 $1,000 $2,000 $3,000 $4,000 $5,000 $6,000 $7,000 $8,000 $9,000 Life Cycle Cost in RY$M Referring to the graph above: Cost Risk: Low/Nominal------> Nominal ------------------------> High ------->Very High $3,150 $1,105 $1,177 $3,238 $997 $1,071 $3,325 $897 $972 $3,413 $803 $880 $718 $795 $3,588 $639 $717 $3,675 $568 $646 $3,763 $504 $582 $3,850 $448 $525 $1,257 $1,152 $1,053 $962 $877 $799 $727 $662 $604 $1,341 $1,236 $1,138 $1,046 $961 $882 $809 $743 $684 $1,429 $1,324 $1,225 $1,133 $1,046 $966 $893 $826 $765 Project's Point A single equation * gives a more precise UFE estimate; it also enables seeing how UFE varies with changes in the Program s PE and Cost Risk. UFE = 25% x PE Estimate ($M): UFE = 25% x $3,500M UFE = $875M Current PE = $3,500 Total = PE + UFE = $4,375M (at the 62% confidence level) 21 NASA Strategic Investment Division (SID) 21
BACKUP SLIDES NASA Strategic Investment Division (SID) 22
Baseline Policy- Management Agreement (MA) The parameters and authorities over which the program or project manager has management control. The PM is accountable for compliance with the terms of their Management Agreement and has the authority to manage within the agreement. View as a contract between the Agency and the PM. A significant divergence from the Management Agreement must be accompanied by an amendment to the Decision Memorandum *. 23 NASA Strategic Investment Division (SID) 23
Baseline Policy- Agency Baseline Commitment For all projects and Tightly Coupled Programs, the life cycle cost estimate (and other parameters) at KDP C is the Agency s Baseline Commitment (ABC) for that Project or Program. The ABC is documented in the Decision Memorandum. The NASA AA approves the ABC for all projects with a life cycle cost estimate > $250 million. The ABC is the baseline against which the Agency s performance is measured during Implementation. NASA Strategic Investment Division (SID) 24
NPR 7120.5E Project Life Cycle Cost Agreements and Commitments Agency Baseline Commitment (ABC) 70% JCL External Commitments High Estimate LCC Range Life cycle cost estimate UFE managed above the project UFE managed by Project >50% JCL At KDP C and subsequent Agency Baseline Commitment rebaselines the ABC and the life cycle cost estimate are equal. Low Estimate Management Agreements Authorized Formulation Cost UFE Actual Formulation Costs During Implementation During Formulation KDP-C Five year budget run out and schedule estimates are reported to Congress. If a project signs a contract > $50 M, LCC range is reported to OMB. For selected projects, LCC and schedule ranges are reported to GAO Notional and Not to Scale From this point, Congress, OMB and GAO get detailed cost and schedule information. All changes are tracked back to the ABC. NASA Strategic Investment Division (SID) 25
Example Questions for Decision-makers Related to life cycle cost How many projects should we select this year? Is it possible to estimate a reasonable UFE for each project? For the portfolio? Will these estimates account for dollar savings due to potential de-scopes? Can this analysis be performed to support the PPBE process? To support decisions at reviews? If de-scopes already exercised, is the project s UFE at Decision Gate reasonable? Is project s cost risk going down over time? If so, how does this impact its UFE? How could a change in a given project s cost risk impact Directorate s portfolio? How could adding a project into the portfolio impact the portfolio s overall cost risk? What s the cost risk exposure? How does UFE allocation reduce such exposure? How can we choose among mutually exclusive projects? What should be a reasonable UFE be as a %-age of the sum of project costs? To what extent can de-scopes reduce our overall UFE? What are preferred ways to allocate UFE across projects? What s a good confidence level for a project Pt Estimate? For the whole Portfolio? What data is needed to run the reserves calculator? How do we obtain this data? NASA Strategic Investment Division (SID) 26
Terminology & Definitions Conditional Project Cost, Average Cost Overrun & Expected Average Cost Overrun Conditional Project Cost (CPC) - Under condition that an overrun occurs, the average of project costs that exceed the project's Point Estimate (PE). Estimated using Monte Carlo simulation. Average Cost Overrun (ACO) - The difference between the Conditional Project Cost (CPC) and the project's PE Expected Average Cost Overrun (EACO) - Likelihood of a Cost Overrun x Average Cost Overrun PE = $50M Average Cost Overrun: $70M - $50M = $20M Conditional Project Cost: $70M Ave = $70M Likelihood of Cost Overrun: Confidence Level (CL) = 25% Likelihood of Cost Overrun: 1 CL = 75% 25% 75% Expected Average Cost Overrun: $20M x 75% = $15M LCC ($ x 10,000) NASA Strategic Investment Division (SID) 27
Overview of Required Model Inputs Note: Technical de-scope inputs not covered in this slide The Point Estimate (PE) is the Project s Life Cycle Cost (LCC) estimate - Example: PM puts forth $50M estimate as the basis for her funding request The LCC Coefficient of Variation (CV) measures relative cost dispersion - A CV is the LCC s Standard Deviation divided by the Average LCC - Example: The Project has an average LCC of $69M with a standard deviation of $12M CoV = Standard Deviation = $12M / $69M = 17.75% Average LCC The Confidence Level (CL) is the likelihood of the LCC being at/below the PE - Example: Given a PE of $50M at the 25% Confidence Level There s a 25% chance that the project LCC will end up at or below $50M PE = $50M 100% Confidence Level (CL) = 25% 90% Ave LCC = $69M 80% Confidence Level (CL) 70% 60% Ave LCC = $69M 50% 40% PE = $50M 25% 30% 20% 10% 0% $20 $30 $40 $50 Life Cycle Cost in RY $M $60 $70 $80 $90 $100 $110 Life Cycle Cost (LCC) $ x 10,000 NASA Strategic Investment Division (SID) 28
