1. JCL Status of Implementations-IPAO
Programmatic Assessment Philosophy
Mr. Richard Greathouse, PAG
Independent Program Assessment Office
Seventh Annual NASA
Program Management Challenge 2010
February 10, 2010
Used with permission NASA Internal Use Only
Page 1

2. Introduction
• Last year’s presentation reviewed how Programs
and Projects were to address Joint Confidence
Level (JCL) requirements as required by NPD
1000.5.
• This presentation will report on the status of the
implementation of the JCL policy. It will cover
challenge of integrating the cost and schedule
analysis with risk assessments to provide a JCL
product.
NASA Internal Use Only
Page 2

3. NPD 1000.5
• (a) Programs are to be baselined or rebaselined and
budgeted at a confidence level of 70% or the level
approved by the decision authority of the responsible
Agency-level management council. For a 70%
confidence level, this is the point on the joint cost
and schedule probability distribution where there is a
70% probability that the project will be completed at
or lower than the estimated amount and at or before
the projected schedule. The basis for a confidence
level less than 70% is to be formally documented.
• (b) Projects are to be baselined or rebaselined and
budgeted at a confidence level consistent with the
program’s confidence level.
NASA Internal Use Only
Page 3

4. P/p and SRB Roles and
Responsibilities
• Both P/p and SRB will develop probabilistic cost
and schedule estimates
• Both P/p and SRB will perform cost and schedule
risk analysis
• Both P/p and SRB will develop JCL Assessment
• SRB will evaluate the P/p products
• P/p and SRB results will be shown at the
governing PMC
NASA Internal Use Only
Page 4

5. Program/project Products
• Probabilistic Cost Estimate
• Integrated Master Schedule Tied To Cost Estimate
• Schedule Risk Analysis
• JCL Assessment
NASA Internal Use Only
Page 5

6. SRB KDP Products
• Benchmark Cost Estimates (for KDP B & C)
• Benchmark P/p Schedule Assessment/Risk
Analysis
• Independent JCL Assessment
• Above “yardstick” enables the SRB to evaluate P/p
products:
• Cost Estimate
• Schedule Risk Analysis
• JCL
NASA Internal Use Only
Page 6

7. Programs/Projects
• Completed JCL’s to-date:
• LDCM
• MMS
• NuStar
• MSL
• In addition, an Integrated “Cost Given Schedule”
risk assessment was performed for GPM
NASA Internal Use Only
Page 7

8. General Feedback
• Insufficient resources for building the JCL Model
• Resource-loaded integrated master schedules were not available
(about ½ of total effort)
• Cost, Schedule and Risk were not linked at any level (about ¼ of
total effort)
• Cost and Risk not captured at the same level of
detail as the schedule
• Excursions took hours to run with MS Project and
@Risk
• Work-arounds are in progress
• Weak basis for uncertainty factors used at the
subsystem level
NASA Internal Use Only
Page 8

9. General Feedback (Cont.)
• SRB process for evaluating the JCL was not well
defined
• Role of benchmark cost and schedule estimates in helping to
evaluate the JCL is not clear
• Processes for vetting the JCL technical inputs through the SRB
is somewhat confused
• The JCL effort required 3-8 staff months to prepare
JCL is still going thru some rough growing pains.
NASA Internal Use Only
Page 9

10. JCL Implementation Results
• LDCM
• GPM (SRB Integrated Cost/Schedule Analysis)
• MMS
NASA Internal Use Only
Page 10

11. Time and Resources for JCL
• LDCM spent over 1 person-year to complete JCL and reach consensus
with SRB
– Split ~50/50 between technical/management and scheduling/JCL personnel
– Requires senior project official to lead/coordinate at average of 0.2 rate
– Requires the most critical & knowledgeable people on the project for risk analysis
– Routine JCLs would require additional scheduling support and ready access to JCL
experts
• Install tool and perform trial run took ~3 months
• Actual JCL took ~2 months
– Facilitated by risk analysis from trial run
• Refinement with SRB took ~1 week
• Next JCL will probably take ~3 months
– Including one month allowance for:
• Full risk analysis
• Re-familiarization with process/tools
NASA Internal Use Only
Page 11

12. LDCM JCL Lessons Learned
• Projects must have training and perform trial run JCL before running first real
JCL
• Implementation plans and schedules must be very healthy to run JCL
– PDR-level plans/schedules may not be sufficient due to open trades and
lack of schedule refinement
• JCL cannot be done without JCL expert support
– Projects do not have sufficient statistical analysis expertise
• LDCM JCL results driven mostly by uncertainty factors, not risks
– 10 of the 12 months of LRD movement and 80% of cost growth attributed to
uncertainty factors
• Probably due to relatively low risk likelihoods because we mitigate
known risks
• Basis of estimate for uncertainty factors is currently not well developed
– Active area of research in cost estimating community
– Do uncertainty parameters double-book the impact of identified risks?
• After 200 iterations, 70 percentile solution stabilized to approximately +/- $15M
and +/- 3 weeks
NASA Internal Use Only
Page 12

13. LDCM JCL Lessons Learned (cont.)
• Risk analysis
– Requires significantly more work and speculation than required for effective continuous
risk management
– Value of detailed risk analysis in question since risks had little impact on LRD or cost
• IMS model for JCL cannot be maintained, it must be recreated every time a JCL is needed
– New schedules delivered monthly by contractors
– Need to remove reserves, prune unnecessary detail, assess and link in risks, cost load,
etc.
• JCL is not a useful tool for assessing exposure to a specific risk or trade mitigation options
– Impact of any particular risk is dwarfed by uncertainty in risk assessment and impact of
general uncertainty factors
• JCL cannot model IMS logic changes and work-arounds that would actually occur when risks
are realized
– Must be estimated when risk impacts are assessed
• JCL tool used by LDCM has significant weaknesses
– Required significant degree of custom coding by JCL experts to implement JCL
– Macros and settings that run JCL are not saved uniquely for each run file
NASA Internal Use Only
Page 13

14. Other Observations
• Yin
– JCL process forces project to analyze schedule, risks, and costs to a much higher degree than would otherwise
be done
– Unlike parametric ICE, JCL better captures unique aspects of a project
– JCL under project control to ensure accuracy of inputs/assumptions, and is objectively evaluated by SRB
– 70% values probably are conservative cost and schedule estimates
• But are only “neighborhood” estimates, not a prediction
– JCL results can be credibly “justified”
• Yang
– JCL requires high degree of speculation on risks, impacts, workarounds, and uncertainties
• The risks that bite you are the ones you don’t expect, and are therefore only covered by nebulous
uncertainty factors
– JCL cannot anticipate disaster scenarios and unforeseen political impacts to cost and schedule
• No cost/schedule analysis tool can
• JCL cost/schedule estimates are no more accurate than any other educated guess
– JCL process offers an infinite number of nebulous knobs to turn
• Both the project and the SRB have significant latitude to justifiably tailor results with high degree of
impunity
Bottom Line: Despite its many shortcomings, a JCL developed by the project and reviewed by the
SRB is probably the best known way to methodically establish a reasonably-conservative justifiable
cost/schedule estimate that has the support of both the project and independent reviewers
NASA Internal Use Only
Page 14

15. LDCM JCL Observations
• It is important to have an IMS first. The IMS needs to be reviewed and pass all health checks before
any risk or uncertainties are added. If the schedule is not any good, then the JCL results are worthless.
• Uncertainties are driving the JCL, not the risks themselves. The uncertainties are based on DoD
classified data, and may or may not be relevant to NASA spacecraft. Some knowledgeable SRB
members feel that the DoD experience is not equivalent to the LDCM spacecraft.
• The IMS is not a traditional “resource loaded schedule”. The resources applied are not labor and
material costs by activity, but an allocation of the budget dollars. Reasonable assumptions have been
made to derive the fixed costs (mostly material costs and level of effort activities) and variable costs
(labor) for the budget. Only TIRS and OLI allocate the budget to the next level of the WBS (e.g., FPA,
Structure, etc.) The remaining budget is allocated at the higher WBS Level (e.g. spacecraft, launch
vehicle, EPO, etc.)
• The budget is a “given” and assumed to be a good “cost estimate.” Therefore, any 70th percentile JCL
is grounded in acceptance of the budget.
• JCL must be maintained. This requires effort, skilled people and senior project official to lead and
coordinate JCL.
• Most Project Offices will have to purchase additional tools and training, and hire expert support for
statistical analysis.
• JCL process does force Projects to analyze schedule, risks, and costs to a much higher degree than
would otherwise be done. [Jenstrom, LDCM JCL Observations]
• LDCM Project Office bottom line was as follows: “Despite its many shortcomings, a JCL developed by
the project and reviewed by the SRB is probably the best known way to methodically establish a
reasonably-conservative justifiable cost/schedule estimate that has the support of both the project and
independent reviewers.” [Jenstrom, LDCM JCL Observations]
NASA Internal Use Only
Page 15

16. JCL Implementation Results
• LDCM
• GPM (SRB Integrated Cost/Schedule Analysis)
• MMS
NASA Internal Use Only
Page 16

17. GPM Conditional Cost Distribution
• Two plots of P (Cost < x | Schedule < y) the symbol, | , means “given”
• The blue curve is the original Cost S-curve, i.e., P (Cost < x | Schedule < ∞)
• The pink curve is the modified Cost S-curve, given that we know that the launch
will occur before 01 Oct 13.
P
TY$M
• Cost S-curve becomes steeper with increased certainty of the project’s
duration.
NASA Internal Use Only
Page 17

18. GPM Observations
• Overall cost and schedule risk is reduced by
observing the interaction between cost and
schedule.
– Conditional Probability of Cost given Schedule
– Joint Cost and Schedule Probability
• Correlation between cost and schedule can be
modeled in different ways:
– Parametric model vs. Resource-loaded schedule model
Page 18

19. GPM Observations
• Parametric Cost/Schedule Integration Model
– Calculations of Conditional S-curves done using the NASA Cost-
Schedule Integration Spreadsheet (MCR, Inc.)
• Requires five parameters:
– Mean and standard deviation from traditional Cost S-curve
– Mean and standard deviation from traditional Schedule S-curve
– Cost and Schedule correlation coefficient of 0.8 based on
analysis by Aerospace Corp.
Page 19

20. JCL Implementation Results
• LDCM
• GPM (SRB Integrated Cost/Schedule Analysis)
• MMS
NASA Internal Use Only
Page 20

21. MMS JCL Lessons Learned (1)
• The MMS Project JCL model implementation was too complex in
its construction, inefficient in its execution and will not satisfy
future Project needs for rapid, insightful “what if” analysis and
reporting.
• The PRSL model developed by the SRB using MS-Project with
@Risk with an Excel JCL add-on performed well.
– Simple model construction ( 165 total tasks for cost,
schedule and discrete risks).
– Validated the Project baseline (matched current plan with
reserves removed)
– Documented basis for uncertainty ranges selected and
discrete risk branching (likelihood, cost & schedule impacts)
– Rapid execution (complete JCL “what if” update cycle of 40
minutes)
NASA Internal Use Only
Page 21

22. MMS JCL Lessons Learned (1)
• The results of the JCL are highly dependent on the
assumptions made about the risk distributions to be used and
what types of discrete risks should be included in the analysis.
In order for future JCLs to provide “apple to apple”
comparisons across projects, process standardization
(normalization) should be provided by PA&E in the form of a
published guideline.
– MMS Project used narrow, symmetric triangular distributions for modeling
risk and uncertainty. SRB used broader, right skewed triangular
distributions that are more representative of aerospace industry norms.
– MMS Project included discrete risks that were already covered by
uncertainty (i.e., double accounting of risk). This did compensate
somewhat for their narrower probability distributions.
– MMS Project included discrete risks that were external in origin and
outside their span of control, though likely to have a negative impact on
performance. The SRB chair regarded these risks as “out of scope” for
SRB deliberations. A policy decision is needed on the handling of external
project risks in the JCL.
NASA Internal Use Only
Page 22

23. Overall Recommendations
JCL is a good product, yet to come of age.
– Continue development of good cost and schedule
tools.
• Schedules to-date often do not pass basic health checks.
– Use native data formats.
– Develop a smooth transition to a workable
integrated methodology.
NASA Internal Use Only
Page 23