Risk Adjusted Estimating Techniques
•
3 j'aime•412 vues
Presented by Donald Shannon
Recommandé
Recommandé
Contenu connexe
Tendances
Tendances (20)
Similaire à Risk Adjusted Estimating Techniques
Similaire à Risk Adjusted Estimating Techniques (20)
Plus de Government Contract Pricing Summit
Plus de Government Contract Pricing Summit (14)
Dernier
Dernier (20)
Risk Adjusted Estimating Techniques
- 1. JUNE 19 – 21, Bethesda, MD
- 2. RISK ADJUSTED (UNCERTAIN) ESTIMATING TECHNIQUES Quantifying and Accounting for Uncertainty in Project Cost Estimates © 2018 – Donald E. Shannon dba The Contract Coach. All Rights reserved
- 3. Introduction Let’s start the discussion with a common frame of reference
- 4. The “Point Estimating” Process Cost estimates – especially for Government contracts are built on the assumptions of the Truthful Negotiations Act (aka “TINA”) “Current Complete and Accurate” cost data Bottom-up estimates using actual quotes, labor rates, indirect rates, and detailed BOEs Result is a detailed set of verifiable cost data used to determine contract price. Single point estimate of a complex problem That isn’t good enough!
- 5. Your cost estimate one value in an infinite range of possible outcomes Estimates are defined by a Probability Distribution Function (PDF) …NOT a single point Why? Because Estimates are “Uncertain” Your “Most Likely “Single Point” Estimate What Usually Happens
- 6. Risk or Uncertainty? Uncertainty is the indefiniteness about the outcome of a situation. It is assessed in cost estimate models to estimate the risk (or probability) that a specific funding level will be exceeded. Risk is the chance of loss or injury. In a situation that includes favorable and unfavorable events, risk is the probability that an unfavorable event will occur. © 2000 From Probability Methods for Cost Uncertainty Analysis by Paul Garvey.
- 7. What is Uncertainty? Uncertainty is a property of estimates that causes them to assume a range of values rather than a single, precise, value. Uncertainty is the total effect of: Aleatory Risk – the outcome of a probabilistic event is a random variable described by a frequency distribution. (e.g., the result of rolling a die is a uniform distribution between 1 and 6) Systemic Risk – The impact that organizational and environmental factors have on the outcome of a project. Project-specific risks – events that are quantifiable with respect to likelihood and impact that directly effect the project. Project Specific Risks Systemic Risks Aleatory Risks Uncertainty Project Specific Risks Systemic Risks Aleatory Risks
- 8. Aleatory Risk Naturally occurring process The outcome of an uncertain event is expressed as a random variable Aleatory Risk is always present in any uncertain event – p(100) Cost, performance time etc. forecasts can only be stated probabilistically Often times this risk appears as variability How long to drive to work? When we plan for uncertain events we typically plan for the “most likely” outcome “On average” it costs “x” or takes “y” days Sometimes we do better than we estimated Sometimes we do worse
- 9. Epistemic Risks Apply to the business or market as a whole Many times are unseen/unheard Often related to business practices or environment Can and should be considered and, where possible mitigated or eliminated. Often expressed as a single risk factor even though made up of numerous elements Are unique to the project at hand Often can be foreseen but not completely eliminated Usually technical or people oriented (safety, bad decisions) Also include uncontrollable events (weather, accidents, etc.) Can often be mitigated or transferred Systemic (Global) Project (Local)
- 10. Systemic Risk Project Specific Risks Systemic Risks Aleatory Risks Uncertainty Project Specific Risks Systemic Risks Aleatory Risks • Systemic Risks tend to be the largest driver of project (or contract) uncertainty • Systemic Risks are 100% likely to occur • Economic conditions such as inflation • Political environment such as pro- defense vs pro-social programs (guns or butter) • Organizational factors such as management systems, project controls, attitudes towards cost overruns etc. Note: Even if we do a great job on the Project Specific risks we are only addressing a small portion of the overall uncertainty.
- 11. Project-Specific Risks These are risks that MAY occur during your project Generally speaking program risks are “contingent” events Will it happen? If it does occur what’s the impact? This is the “standard” or frequently performed segment of risk management Calculate Risk Score Risk Score = Probability x Impact Record result in Risk Register Plot results on Risk Matrix (optional)
- 12. Current Cost Estimating Expected Cost Range (Usually Aleatory) Actual Cost Range Includes Total Risk (Aleatory + Systemic+ Project-Specific) -5% 0% +5% +20% +50% +100% +150% • Does Not Consider or Significantly Understates Uncertainty • Uncertainty = Risk • Estimates without risk tend to be optimistic Single Point Project Estimate -% +%
- 13. Include uncertainty in total cost Direct Cost Indirect Cost Uncertainty Profit Price Direct Cost Indirect Cost Uncertainty Profit Contract type determines which party carries the burden for uncertainty: - In Fixed Price contract the seller is at risk and includes contingencies and allowances in their estimate. - In Cost type contracts the buyer is at risk and must identify and retain an adequate program reserve.
- 14. Uncertainty Dimensions • Project Size • Complexity • Business Environment • Business Systems • Technology • Scope Definition / Project Stage
- 15. Uncertainty and Project Size Project Size 1. Small, routine project estimates Tend to be inflated – The Scotty from Star Trek syndrome Actual costs come in either on budget or under budget Especially true in organizations that “punish” overruns 2. Larger projects tend to have optimistic estimates (Most large Defense Contracts) 3. Mega Projects tend to be optimistic with significant risk of “Run-Away” (The Healthcare.gov type project) Cost Uncertainty 1. -3% to + 8% of estimated costs 2. -5% to + 10, 15, or 25% of Estimated Costs 1 depending on design maturity 3. -5% to as much as + 300% 1. These percentages are industry specific and should be determined by a Multiple Regression Analysis
- 16. Uncertainty and Project Complexity 1. Structural Project size or Value Number of WBS Elements Number of participants/organizations Interdependencies 2. Social Organization Structure Contract or subcontract types/terms Project Manager Engineering Electrical Mechanical Controls Software Logistics Training Documentation Spares Quality Safety Production Subassembly Final Assembly Admin
- 17. Uncertainty and the Environment Organizational structure Risk tolerance Shareholders Communications National or International politics
- 18. Uncertainty and Business Systems Formalized Business systems with third-party review/acceptance significantly reduce Systemic Risk ISO 9XXX quality system Acceptable Accounting System Change Management System Approved Estimating System ANSI/EIA 748 Earned Value Management System Six-Sigma Program Management Office/PMP Certifications Capability Maturity Model Level
- 19. Uncertainty and Technological Maturity • The technology used in a project is a significant contributor to Uncertainty. • This is a concern in both Hardware and software projects • Matrices such as the one shown are helpful in evaluating the technology component of Uncertainty which is then incorporated in the overall Uncertainty model
- 20. Project Uncertainty vs. Scope Definition and/or Design Maturity Most R&D Contracts are here Most Software and System Acquisition Contracts are here Most Design-Bid- Build Construction Contracts are here 12345 NACE Estimate Class Here’s another View of the same Information
- 21. Quantifying Uncertainty It’s one thing to know there is a problem .. It’s quite another to know how to solve it.
- 23. Typical Cost Estimate • Typical cost estimates are built up of several smaller estimates – the so called “Bottom-up approach. • Each component estimate is a ‘single point’ estimate based on various factors including labor, materials, etc. • Estimates are typically summed at a WBS or system summary level to arrive at total cost • Costs are supported by documentation such as • Quotes or purchase order data • Detailed Basis of Estimates (BOE) • Historical data • Cost Estimating Relationship / Parametric Model.
- 24. “Resource Loaded” Schedule Most Project Managers use a scheduling tool like MS Project or Primavera P6 to generate a top level project schedule. What you want is for them to assign resources to that schedule and provide a BOE supporting the resource estimate.
- 25. Import into the Pricing Tool Here the Project Schedule has been imported into a pricing tool (PROPRICER in this case) to create a standard single point estimate. Labor Hours by Task (Single Point Estimate)
- 26. PROPRICER Three-Point Estimate • Those of you who use PROPRICER to do your cost estimates will find that the software easily accommodates three-point estimates • To use it you must first enable three point estimating from the General Proposal Properties Tab • The simplest level of implementation uses a default set of best and worst case values. • Settings of -5% (.95) as best case and + 12% (1.12) are a reasonable starting place lacking any better data. Use these values to set +/- “Uncertainty” values like -5 to +12%
- 27. PROPRICER Three-Point Estimate Our next task will be to add a plus or minus range representing uncertainty to the estimate The existing single point estimate is the “most likely” value. To that we subtract x for the “Best Case” and add y for the “Worst Case” Three-point estimates are a more advanced estimating technique that although recommended by GAO is not commonly used. Use these values to set uncertainty ranges for different resources
- 28. How much “Uncertainty” Should I add? It Depends. The amount of “Uncertainty” included in an estimate depends on the reliability of your data. Ideally, it should be based on a detailed analysis of historical data via a Qualitative Analysis followed by a Quantitative Analysis. Sometimes we lack sufficient data and must rely on “expert judgements” expressed as – Best Case, Most Likely, and Worst Case Such estimates tend to be optimistic
- 29. PROPRICER Three-Point Estimate • If you wish (and I suggest you do) you may assign different risk factors to various categories and elements of cost. • The best solution would be to use the costs and +/- percentages from the historical (quantitive) cost model for your 3-point estimate. • You could assign different values for uncertain costs such as travel, materials, subcontracts, etc. Use these values to set uncertainty ranges for different resources
- 30. Risk Analysis
- 31. Qualitative Analysis Credit: Glen B. Alleman, Herding Cats: Why 3 Point Estimates Create False Optimism
- 32. Quantitative Analysis Assign Values to Uncertainty based on Historical Data Preferred method Extract impact data from historical data Cost Schedule delay Litigation Injury Difficult to do and oftentimes essential data does not exist
- 33. Multiple Regression Analysis Use statistical tools to determine line (or plane) of best fit for historical data points Multiple regression can not be depicted in two- dimensional plot We construct a linear equation of the form: Y = a + b1*X1 + b2*X2 + ... + bp*Xp We iterate to find the equation that offers the best fit to our data (lowest total of “error”)
- 34. Risk Modeling
- 35. Systemic Risk Model Based On Historical Data & Project Attributes Sample Model from John K. Hollman Inputs Scope Definition (Class 3,4,or 5) Project Complexity (L, M, H) Level of Technological Sophistication (L, M,H) Adjustments made for various factors Results are then used to define +/- range to define p10 (Best Case) and p90 (Worst Case) values.
- 36. Historical Cost Model Adjusted for Risk 1. We start with the data from your cost estimating tool. 2. Select the “Uncertainty Factors” from our model 3. The model calculates the probabilistic sum of the 3-point values * * To be explained on a later slide where we “total the uncertainty” …. Uncertainty Factors 3-Point Estimate Parameters CLIN or WBS Description Traditional Single Point Estimate Factor 1: Design Maturity Factor 2: Project Complexity Factor 3: Technology Maturity Risk Best Most Likely Worst 1.1 Preliminary Design $ 79,656.00 Conceptual High Medium $ 50,183.28 $ 109,128.72 $ 168,074.16 1.2 Detail Design $ 229,352.00 Conceptual Medium Medium $ 167,426.96 $ 291,277.04 $ 415,127.12 2 Prototype Build & Test $ 41,686.00 Budgetary Low Low $ 37,517.40 $ 45,854.60 $ 54,191.80 3 Project Management $ 57,023.00 Budgetary Medium Low $ 49,610.01 $ 64,435.99 $ 79,261.97 Total $ 407,717.00 Sum $ 304,737.65 $ 510,696.35 $ 716,655.05 $ 337,331.28 $ 510,585.72 $ 692,044.58
- 37. Why such a wide range of results? This estimate is VERY risky because: Engineering and prototyping tasks were estimated on “Conceptual” level data Complexity level of the prototype build What do we do if we want to improve the estimate? Obtain more data Better define the scope of work Preliminary engineering study Revise our strategy to reduce project complexity
- 38. Project Specific Risks Meanwhile, back on the ranch … Adding the “what if” factor of projects
- 40. Totaling the Uncertainty Adding apples and oranges …..
- 42. Don’t Simply add the individual elements! “It is inaccurate to add up the most likely WBS elements to derive a program cost estimate, since their sum is not usually the most likely estimate for the total program, even if they are estimated without bias. Yet summing costs estimated at the detailed level to derive a point estimate is the most common approach to estimating a total program. Simulation of program risks is a better way to estimate total program cost, …1.” 1. GAO Cost Estimating and Assessment Guide GAO-09-3SP Pg.. 153
- 43. Can You Add Probability Distributions? Two commonly used methods Which you choose to use depends on your expertise and the tools available Method of Moments is accurate but requires some math background Monte Carlo Simulation is easier but requires software tools
- 44. 1. Method of Moments Method of Moments Technique Analytical technique Used to calculate the “moments” of the combined distribution The resultant distribution from adding two triangular distributions is a lognormal distribution.1 The Moments of that are: Mean = μ = μ1 + μ2 … μn Variance = σ2 = it depends2 Skewedness1 = ϑ = Kurtosis = κ = 12/5 = 2.387 The math needed to calculate these is outside the scope of this presentation 1. Analytic Method for Cost and Schedule Risk Analysis, Raymond P. Covert, NASA, 5 April, 2013, pp 34 - 37 2. Calculating variance for the sum of two distributions is complicated when the two distributions are correlated. Formula shown is for correlated data For Rocket Scientists Only
- 45. 2. Simulation Monte Carlo Technique Multiple trials where unknown quantities are generated by random numbers Each value is added to the total to arrive at an overall sum The process is repeated numerous (several hundred or thousand) times Each trial becomes a possible outcome and is tallied in a histogram Overall statistics such as mean etc. are then calculated from collected data The resulting data may then be interpreted for select values for given probabilities p(x) Simulation output of 3,460 iterations © Intavar Institute Risky Project 6.0
- 46. Realistic Budgeting “One way to determine whether a program is realistically budgeted is to perform an uncertainty analysis, so that the probability associated with achieving its point estimate can be determined. A cumulative probability distribution, more commonly known as an S curve—usually derived from a simulation such as Monte Carlo—can be particularly useful in portraying the uncertainty implications of various cost estimates.” “The amount of contingency reserve should be based on the level of confidence with which management chooses to fund a program, based on the probabilities reported in the S curve.” GAO Cost Estimating and Assessment Guide GAO-09-3SP Pg. 157
- 47. The “S-Curve” Output The output from the simulation is presented in both a Probability Distribution Function (PDF) and a Cumulative Distribution Function (CDF) also called an “S” curve The value associated for an “acceptable” level of risk is taken directly from the plot (example 80% chance of completing at or below a cost is achieved at $555,000) RISKYPROJECT
- 48. “What’s my Take-Away?” 1. The initial estimate of $424,000 is outside the risk adjusted results. If you were to use that estimate you would almost certainly be wrong. 2. Risk and Uncertainty add on average $100,000 or nearly 25% to the estimate. 3. Depending on your risk tolerance you should be looking at a total project cost between p(50) = $528,000and p(80) = $555,000 4. If you are awarded a CPFF contract at a value for less than the p(50) amount you may get additional funding to complete the project but will have a lower average fee rate (typically no fee on overruns) and possibly earn a reputation for overrunning costs 5. If you accept a FFP contract for less than p(50) you won’t stay in business very long. 1 2 3
- 49. Everything we just said about cost … you can also say about schedule. Oh, by the way …
- 50. Conclusion
- 51. Words of Wisdom* “Because cost estimates predict future program costs, uncertainty is always associated with them. … Moreover, a cost estimate is usually composed of many lower-level WBS elements, each of which comes with its own source of error. Once these elements are added together, the resulting cost estimate can contain a great deal of uncertainty. Quantifying risk and uncertainty is a cost estimating best practice addressed in many guides and references. Quantitative risk and uncertainty analysis provide a way to assess the variability in the point estimate. … Having a range of costs around a point estimate is more useful to decision makers, because it conveys the level of confidence in achieving the most likely cost and also informs them on cost, schedule, and technical risks. “ * from the GAO Cost Estimating Guide
- 52. Key Take-Aways Uncertainty is quantifiable using the techniques presented Range estimates based on probability are superior to point estimates generated by conventional means. Many existing software products contain capabilities to implement these techniques
- 53. Don Shannon – The Contract Coach don@Contract-coach.com http://www.contract-coach.com (505) – 259-8485 Consulting Partner
Notes de l'éditeur
- Hello and welcome to “Risk Adjusted – or Uncertain – Estimating Techniques” In the next ___ minutes we will explore several best practices recommended by – among others – the Government Accountability Office, The Program Management Institute, and the American Association of Cost Engineers.
- When we are taught to estimate we are taught that there is a “correct” answer and that answer can easily be reproduced by following the methods and using the data available to us in constructing that estimate. The emphasis is on avoiding any inference of “defective” data or conclusions under significant penalty. Our response is to then perform a detailed analysis of the work to be performed then construct our estimates for doing that work - all the time backed up by other estimates, quotes, and relevant data. Doing so gives us “certainty” that our estimate is correct. Our output from this process is a total value – comprised of the summation of other values - that pinpoints our estimate down to the last penny. But is it accurate? Practice shows we may have a problem.
- When we walk away from the conference room where we have debated and ultimately approved our project budget or our bid price and we actually begin work - things often start to drift off plan. Managers are beside themselves trying to explain or quantify variances to the budget and, at the end of the day, we often see we have missed the mark. If I were to collect the data for a number of programs from your files and plot out the results of how well the program met the estimated cost here is what we would find. Your “most likely” or single point estimate would be represented by the Mode. In this example, roughly 23% of the projects are delivered at or below that cost. The “average” project value (the mean) - is significantly higher than the mode – typically on the order of between 12% as a low and more like 15 to 18% in common practice. Bear in mind that statistically the mean represents a midpoint – that is 50/50 chance of coming in at or below that value - The maximum value is way off the edge of the chart. These are the “runaway” projects that come in at 300% or more of the estimate. These are the programs that get lots of publicity – for the wrong reasons. The root of the problem is two-fold: 1. We use a single point estimate for something that is arguably a random event and should be represented by a continuous probability distribution Our estimate does not include an allowance – or perhaps an “adequate” allowance for uncertainty. With that as a starting point let’s see what’s really happening to our estimate.
- The first thing we need to do is to define two terms – terms that tend to get used somewhat interchangeably however, they are anything but. The first is Uncertainty. Uncertainty implies a lack of knowledge about some future event and is inclusive of risk plus other factors The Second is risk. Risk is the likelihood and consequences of some future event. If there is no event – then there is no consequence –good or bad.
- I have taught a number of classes and seminars covering the Risk Management topic and inevitably people seem to get hung up on definitions. Perhaps it’s our culture but the concept is pretty simple. In our case we are going to talk about an overarching topic called “Uncertainty” easily remembered by a simple definition “I DON’T KNOW” So the follow-on to that is “What don’t we know”? And at this point someone wants to waltz down the Donald Rumsfeld pathway to Know Unknowns and Unknown Unknowns. While that may be a interesting philosophical discussion our focus today will be a bit more down to earth. As shown in the chart my “Keep it Simple” approach lumps three elements together to define total uncertainty – and this will be the central focus of this presentation. Each of these elements of uncertainty is separately addressed in the following. What I want you to notice in this chart is that the so-called “Project Specific” risks like ”While digging the foundation for our building we hit a layer of compacted clay” which means more work and more cost …. Those risks are project specific and we sit through countless hours of “what if” meeting and brainstorming sessions to compile them. While that’s a good exercise Hollmann – in Project Risk Quantification – asserts these risks actually comprise a minor portion of the overall uncertainty. He contends – as I have illustrated – that the largest element of uncertainty is Systemic Risk. Let’s look at each of the three individually.
- Projects are comprised of a number of individual tasks or steps. Generically these tasks tend to be the kind of things a business specializes in doing – be that digging a trench or building a laser. It something the company does over and over again. But here’s the rub … each time we do something like the above task the time it takes to do it and the cost of doing it are (statistically speaking) an independent random variable. So when we consider driving to work usually takes us 12 minutes what we’re really saying is ‘the median (most often occurring) time for me to dive to work is 12 minutes. Some days it’s 10 and some it’s 20 or more. But “Usually” it takes me 12 minutes. So when we plan the cost for doing a task in our project we are planning for the “usual” value and we should recognize that the actual value will be a little more or a little less … The more we do something and the better trained our people are to do it and the better our processes are optimized for that task the smaller this risk becomes. Conversely – if we don’t do something over and over again we tend to forget lessons learned and the “historic” value we are putting in our estimate may be optimistic
- The remaining uncertainty is called “Epistemic” risk and is comprised of the reaming two constituents - Systemic risk which is global in nature to all our projects and project specific risks. In risk management 101 these comprise the know unknowns The strategy for handling these risks often includes obtaining more data thus transforming them from ”unknowns” to “knowns” or at lest improving the degree to which we actually know them.
- Systemic risks covers a broad swath of things that can affect your business and the outcome of your project/contract. Some of the components of systemic risk are controllable and can be ‘bought down’ to reduce their impact. These are things like business systems – you know the ones listed in the DFARS like an approved estimating system, an acceptable accounting system, earned value management, an ISO quality system etc. Some of these factors are more difficult to quantify such as interest rates, exchange rates, the political environment, etc. But collectively these systemic risks are always present – they occur every time we do a project or a contract. While we can control some – others are just a “fact of life”
- The third element of uncertainty is the one we get the most training on – yet is actually the least prominent factor in the big picture. These are the so called program specific risks that are unique to your project. These risks are really “What If” risks. We ask ourselves during risk planning “what could go wrong?” (or what could go much better than we hoped for) and then make a list of these disasters accompanied by a likelihood the event will take place and then some estimate of the likely effects or costs. Many times we can only state these parameters qualitatively so we end up scoring them on a 1 to 5 or similar scale for likelihood and consequence and portray them is a risk matrix like the one here. As we get more sophisticated in the process we often have some actual data that will help us quantify the effects – sometimes historical other times a Basis of Estimate – so that we can make more informed decisions about how to contend with a specific risk .
- So the upshot of this discussion is: Because we do not properly identify and assess all element of uncertainty when tend to do one of two things: 1. We generate a “point estimate” that describes our program cost with a single value. From a statistical point of view the odds of hitting the actual number with a point estimate are zero. (Discussion) The probability of a random event is expressed as the area beneath the Probability Distribution Function. Typically this value is computed as a range such as between x and y or greater than x or less than y. However when we look at the probability of a single point that area is a rectangle with a width of zero. So multiplying the height of the curve at that value by the zero width returns an area (probability) of zero. 2. More experienced estimators include some plus or minus range to the estimate to improve it’s accuracy. Unfortunately, few of these ranges include much more than some aleatory risk (variability) and perhaps an analysis of project specific risks. Consequently they tend to significantly understate the likely range of actual costs because they ignored the largest single driver – Systemic risk!
- In the government contracting arena it’s important to remember who has the risk for controlling cost. Much of the cost risk is transferable from the government to the contractor of vice versa through the mechanism of contract type. In a Fixed Price contract the seller must accurately compute their cost, contingencies, and profit to set an achievable total price. If they understate costs or do not appropriately consider risk then they may overrun cost and achieve a smaller profit, no profit,, or even a loss. Consequently the government is somewhat more flexible concerning the profit being earned by a contractor on fixed price contracts provided the overall price is deemed fair and reasonable to the government. Where it can get tricky is in a negotiated Fixed Price arena where there is only one bidder. Then the government may take an active role in setting profit ceilings that may or may not be reasonable. In Cost Reimbursement contracts the contractor has less risk from submitting an optimistic estimate or understating costs. Often it is to their advantage to do so since the nature of the contract allows the government to add cost (but not necessarily fee) in the event of a cost overrun. Consequently the government offers lower fees for this type contract in recognition of the lower risk to the contractor. However the overarching principle here – and one that must be well understood – is that Cost is fact based and verifiable whereas Price is a business decision. Therefore we shall look at cost and leave the determination of price to the company management.
- Uncertainty isn't some nebulous blob … actually we can break it down into smaller chunks that we can then analyze and make allowances for.
- The degree of uncertainty in a project – and more importantly its impact on that project is a function of project size. Small routine projects tend to have a chance to underrun the estimate – but not so much because they are more easily estimated as it is that the estimate is usually inflated. Typically this is defensive behavior by the project manager who is punished for overrunning the budget so they simply boost the estimate and get praised for “on time – on budget” performance. This in turn leads managers to automatically cut x percent from future budgets as a “management challenge” … it’s a game. Larger projects tend to have optimistic estimates. The reason is the PM wants the project to go forward and knows that it won’t be approved by senior management above a certain threshold. The ones to watch out for are the so-called mega projects. These tend to suffer from all of the above problems with a distinct possibility of a cost run-away. The percentages on the right are typically the range (plus and minus) to be added to the “best case” estimate to arrive at a three point estimate of what the cots should be. The origin of these percentages is a study of historical data that accounts for all “uncertain” factors.
- Project complexity is a second factor in determining the “uncertainty” The nature of the project itself is key. The larger your team and the more elements in your Work Breakdown Structure the more likely it is that something has been overlooked in your estimate or the more difficult it will be to execute your project. Finally our organization itself is a source of uncertainty – often we have to get approvals or funding from senior managers and that is subject to change if there is a reorganization. Also the contract type fixed price or cost plus fixed fee establishes certain risk thresholds.
- Environmentally we need to understand things like the stability of currency rates, the political environment and the assumptions we make about these things. For instance if we assume that Congress will plus up the funding for our project in years 3 through 5 (which would be a new budget and a new Congress) how much can we rely on that assumption? Do you want to bet the company? We recently saw announcements of proposed tariffs on imported steel and aluminum. If you are bidding a contract to build tanks for the Army – you may want to consider the stability of steel prices over the run of the contract.
- Often we tend to be our own worst enemy. If we have not implemented management best practices then even if we make a good prediction of cost or schedule our ability to perform to that level is endangered by immature management systems. If some of these factors look familiar its because they are – look at the weighted guidelines and you will see these are factors the Government is looking at when calculating fee. The better contractors control these elements the less uncertain their estimate and performance will be and consequently the Government has less risk and is in a position to reward the contractor with a higher (or lower) fee,
- Finally the “elephant in the room” - how well are contract requirements defined? There is a vast difference between an elevator speech where the PM tells the CEO how much some new – never done before project will cost and an estimate prepared for a new highway bridge based on a detailed design from a civil engineering company. As shown here, the better the definition of the project requirements the greater the accuracy of the estimate. This concept is tremendously important since how well the project is defined is a determinant of not only uncertainty and the accuracy of the estimate – but it also ties in with the appropriate contract type. Here is a discussion question: Is it in the Government’s interest to fund an exploratory contract to develop or define requirements before issuing a solicitation for a major project? If so, what would be the value (how much would they pay) for that contract? (i.e. what is the expect value of (more) perfect information?
- Now that we have identified the components of uncertainty let’s look at how we can use that knowledge in a practical sense.
- Current practice is to build “bottom-up” estimates based on a Work Breakdown Schedule (WBS) and discrete estimates for the various elements in the WBS – usually supported by a Basis of Estimate (BOE) We then sum the estimates at higher and higher levels of the WBS to arrive at a project total estimate.
- Here we are looking at a detailed estimate for one WBS element imported from a Basis of Estimate. Specifically we see the start and end dates and we see the resource input from our BOEs as to labor category and hours. Finally the hours have been “spread” over the period of performance using a curve. This estimate was done in PROPRICER which will then apply the various factors such as wage escalation and indirect costs so as to arrive at a fully burdened estimate for this task, Our next task will be to use the advanced features of PROPRICER to capture uncertainty. We will do this by transforming the single point estimate we imported into a 3-point estimate that captures uncertainty. More on this later. .
- Now come the “New” stuff. We want to add some amount of Uncertainty to our estimate. You can either do so manually by adding a +x% -y% to the estimated single point or you can let a tool do the work for you. In PROPRICER you can select an option to use Three-point estimates and define the + and – percentages that will be applied for you. The amount of uncertainty varies with the factors we have previously discussed. Absent any other basis you can make a conservative estimate for a well defined project (say an RFP based on a fairly detailed SOW) of -5 to +12 percent. (See Hollman for citation)
- But you will be better off if you do an analysis of historical data and develop your own +/- percentages based on categories and circumstances and then apply those factors to the three point model.
- Going back to the question of where do the factors like -5 to +12 come from … The derivation of these values is through an analysis of historical data. The tool we use to create the predictive model is known as regression analysis. I’ll discuss this in more detail in a minute – but essentially what we do is we look at how we have have performed in similar circumstances and attempt to apply that knowledge to our model.
- This leads to different ranges of +/- for different types of expenses . For example I might use one range of values for materials cost and a second set of values for subcontractor costs. Different categories of expense and different expectations.
- When we assess project specific risks we do so from an ”if this .. then that” approach. Again the best method to use when analyzing these events is to use historical data. If we find that over the past 10 years we have had two days of bad weather in June, one in July and two in August then we should include an allowance for bad weather in our estimate. We can compute the likelihood from the data. Then we can compute the impact – number of lost work hours or days – again from the data and apply that information to our estimate. How that happens in a minute. In many cases we lack historical data so we have to rely on expert opinion. In that case a matrix like the one shown can help convert these opinions into values. One import thing here is that the scales used in such a tool are highly subjective and will be based on the business’s tolerance for risk. So take the values shown as being “representative” not authoritative.
- Qualitative analysis is more precise than the aforementioned qualitative analysis but requires a significant effort to perform This is the method we used for assessing systemic risk where we did the regression analysis. You can do a similar analysis on frequently occurring risks that you can then add to your project specific risks.
- In the regression analysis we look for correlation between two factors. For example power required to make the system work and the heating and air-condition cost would be expected to change in relationship to each other. As stated before we could examine the cost performance (underrun or overrun as a percent of cost) with program complexity, the specificity of the requirements, and various organizational factors. We put the data into a regression model and what falls out is a line of best fit (shown here in red) that defines one variable (e.g., cost) in terms of the other variables
- Ultimately your business should be able to generate a risk model such that given information concerning certain factors you can produce a reasonable estimate of the uncertainty inherent in your estimate.
- Here we see the results of a detailed analysis in the “process industries” such as oil refineries conducted by John Hollman and published in a paper submitted to “Chemical Engineering in 2014”. It conforms to the American Association of Cost Engineer’s recommended practice 18R-97. We call such a tool a Parametric Estimating Tool and its output is a Parametric Estimate. The shaded area highlights the “Most Likely” value or the median of the distribution for systemic risk or uncertainty based on how well the scope is defined. The Class 3, 4, or 5 refer to American Association of Cost Engineering definitions with a Class 3 being at a pre-solicitation 10 – 40% engineered and a Class 5 a much less refined – conceptual level estimate. So a level 3 is about where most of us would be when an RFP is issued. Next Mr. Hollman creates bands for the level of project complexity. This is a subjective rating based on project scope, number of WBS elements, number of sub-contractors, etc. Finally the model cuts across the complexity rating in additional bands of technological sophistication. The lowest risk is a well designed (Class 3) Low complexity and simple technological project with a Median value of 3. The worst case is a loosely defined project with high complexity and lots of new or unique technology. That has a median value of 42.
- Applying the model to our single point estimate provides a different three point estimate than we got earlier. What’s different? This model selects factors to apply to the three-point estimate based on predictive factors such as design maturity, Project Complexity, and Technical maturity to come up with tailored factors or weights applicable to that set of circumstances. These factors lead to an estimate that better considers the various uncertainty levels and generates a more reliable estimate. More on this in a minute.
- All the preceding has given us some quantification of the uncertainty in our estimate. But there is one other category yet to consider. That is project specific risk.
- Here we see a risk register. Risks in the register are quantified with respect to two factors Likelihood of occurrence Consequences if the risk event occurs. The consequence may be either a fixed outcome such as 2 days delay or 10% increase in cost or it may be a random variable such as a cost impact of a Most likely 10,000 dollars with a best case of 3,000 and a worst case of 100,000 dollars. One point to note here …. We have added a risk with 100% probability for the effects of our systemic uncertainty calculated from our model earlier. The Best case, most likely, and worst case values used are the difference between our single point estimate (without uncertainty) and the Best, Worst and Most likely of the model results. That is the dollar value of the Uncertainty that must be added to the single point estimate to compensate for uncertainty.
- Now we are ready to calculate the “risk adjusted” estimate that includes not only the uncertainty caused by our project type and environment – but also includes an estimate for these various project specific risks.
- We do this by combining the probability distributions for each random variable to define the probability distribution for their sum. What’s interesting is that as we do so the highly skewed distributions tend to balance out (thank you central limits theorem) and the resulting distribution becomes more mound shaped and “Normal”
- Often times we do an estimate where we develop independent estimates for each WBS then we simply total those independent estimates into an overall total. From an accounting perspective this works. From a statistical perspective it’s wrong – especially when dealing with uncertainty. Statistically speaking you are summing (somewhat) independent random variables not values in a ledger. Therefore each of the lesser estimates is not a single value but a range of values defined by various moments such as mean, variance (or Standard Deviation), skewness and kurtosis. To simplify matters we often represent these values as three point estimates and infer they are shaped like a triangle. The result (total estimate) is also a random variable and also can be described by its moments. So to properly add multiple distributions you should not add static values – you should add the probability distributions for each value.
- Doing the math can be tricky. A lot depends on your level of expertise in math or statistics. The Method of Moments is considered the most accurate approach but the most difficult to do The less difficult method is to do a Monte Carlo Simulation.
- In the method of moments we calculate the various “moments” or parameters of the resulting distribution. We assume that adding triangular distributions produces a lognormal distribution whose mean is the sum of the means of all the distributions being combined. The Variance – and hence the standard deviation – is a pretty complex calculation and depends on whether the various elements being added are somehow related (i.e., Correlated). The shape parameters of skewedness and Kurtosis are calculated as shown. This level of math is beyond our scope so I’ll simply acknowledge this method exists and leave it for the experts.
- Simulation provides a reasonable approximation of the result with far less math knowledge involved. To do the simulation we basically solve the problem using randomly generated values conforming to the 3-point estimate for each value and add them all up. The answer is one data point in a histogram chart similar to that shown. We repeat the solution using new random numbers over and over again. Usually several hundred or several thousand times. What we get as an output is a chart looking like the one here … usually mound shaped – often skewed to the right. Our “answer” is derived from interpreting the graph.
- The net result of the probabilistic summation of the various task or activity estimates is an analysis of the the estimate considering uncertainty – inclusive of risk events, systemic risk, and aleatory risk. Using the cumulative distribution function (CDF) produced by this analysis allows us to establish a confidence interval such that we can state the likely total cost or duration of a project with a stated degree of certainty.
- Unlike single point estimates that yield only a single answer – the probabilistic approach we are using here provides a range of answers = each with an associated probability of occurring. The usefulness of this approach is that one can select an acceptable level of risk and then use the chart to translate that risk into a corresponding dollar value. One should remember the probabilities being expressed are range values such that the p(80) value displayed is the probability the actual cost will be equal to or less than $555,000
- This is admittedly an extreme example – selected to illustrate the technique. In normal practice the single point estimate will lie much closer to the mode of the S-cure plot.