1. Estimating Project Management and Systems Engineering Costs in a Changing NASA Environment Presenters: Steve Shinn Larry Wolfarth Co-Authors: Meagan Hahn Sally Whitley 2011 NASA Project Management Challenge 02/08/2011 Used with permission

2. Agenda Definitions Trends in Project Management (PM) and Systems Engineering (SE) Trends in PM/SE Costs How Well Do Our Current Cost Methods Capture Trends in PM/SE Costs? Can We Develop More Robust PM/SE Cost Estimating Relationships? For the Near Term: What Do We Do?

4. [PM] includes project reviews and documentation, non-project owned facilities, and project reserves.

5. [PM] excludes costs associated with technical planning and management and costs associated with delivering specific engineering, hardware, and software products.Project management can be traced back to core ideas developed by Frederick Taylor in the 1880s. Management research, development, and career advancement have been around for decades, with newer theories and tools replacing older ones. The introduction of Earned Value Management (EVM) is one more recent example of an innovation in project management.

6. Systems Engineering in the Space Context NASA NPR 7120.5D defines project-level systems engineering (SE) as: The technical and management efforts of directing and controlling an integrated engineering effort for the project. [SE] includes the efforts to define the project’s space flight vehicle(s) and ground system, conducting trade studies, the integrated planning and control of the technical program efforts of design engineering, software engineering, specialty engineering, system architecture development and integrated test planning, system requirements writing, configuration control, technical oversight, control and monitoring of the technical program, and risk management activities. Documentation products include requirements documents, interface control documents, a Risk Management Plan, and a master verification and validation plan.

7. Conceptual Linkage of SE and PM Systems Engineering provides the “bridge” between a project’s technical and management elements: “Systems engineering is an inherent part of project management―that part that is concerned with guiding the engineering effort itself―setting its objectives, guiding its execution, evaluating its results, and prescribing necessary corrective actions to keep it on course” −Alexander Kossiakoff &William Sweet, Systems Engineering: Principles and Practice

8. SE/PM Connection in the Modern Aerospace Context The disciplines of SE and PM are tied together now more than ever before. Example: 2007 Revision D of NASA NPR 7120.5, an engineering document, prescribes the following: Processes for overall management, risk management, as well as engineering of NASA space missions A range of products for each discipline New processes and products + increased technical complexity = Greater rigor Additional processes for analysis and reporting More resources Time

9. The “Growth” of Modern PM (1 of 2) Since its first edition in 1996, the Project Management Institute’s Project Management Body of Knowledge (PMBOK®) has increased in size and detail with each release:

10. The “Growth” of Modern PM (2 of 2) The last two editions of the Defense Acquisition Guidebook (DAG) have increased its coverage of both PM and SE:

11. What Drives the Modern Expansion of Systems Engineering Modern SE concepts and capabilities are a response to three distinct trends in system development: Increasing complexity – As systems become more complex and inclusive of advanced technologies, new SE approaches have been needed to provide a holistic system view. Increasing scale – As systems become larger or based on systems of systems, trade-offs have become a necessary component of any major project. Specialization – Specialists are assigned to build certain components of the larger system. Therefore, a new discipline focused on management of these component interfaces is needed to ensure compatibility and interoperability among components and overall system performance. * Alexander Kossiakoff and William Sweet, Systems Engineering: Principles and Practice, New York: Wiley

12. Increasing Complexity of Space Robotic Missions: Discovery Cost Cap Trend Trend in U.S. space robotic missions over time is to accomplish more than prior missions. Accordingly, the NASA cost cap for Discovery and other competed missions has increased faster than inflation. At the same time, the number and scope of science objectives has increased significantly from the 1990s missions to those being launched in the 21st century. NEAR: First mission to orbit & land on an asteroid DAWN: First mission to orbit an asteroid, then travel to a 2nd asteroid (requires ion propulsion)

14. When SE effort is less than 8% of total project cost, actual project cost can exceed planned costs by as much as 100%.

15. Conversely, for the few examples where SE effort exceeds 16% of total project cost, cost overruns are minimal.

17. The Trend: Integration of Cost Tools Across the Project Life-Cycle “newer” initiatives

18. PM/SE Cost Estimating Challenge Cost analysts rely on historical data to estimate future requirements for financial resources. To answer the questions posed by our management, we examined post-1995 cost histories of: NASA-funded APL space missions NASA-funded missions led by NASA centers and JPL Costs of remote sensing instruments developed for NASA space missions We considered both actual and projected costs as well as cost trends over time.

19. APL Trends in PM/SE Costs for Robotic Space Missions Since 1995 PM/SE costs for APL robotic space missions launched since 1995 have increased in absolute dollars and as a portion of mission cost. The trend is clearer for SE costs, as shown below. Trends in APL PM/SE costs as a percentage of spacecraft cost for NASA-funded APL missions launched from 1995 to 2002 Trends in APL PM/SE costs as a percentage of spacecraft cost for NASA-funded APL missions launched from 2002 forward, including planned and conceptual

21. Roughly half of the variation in PM can be explained by launch year.

22. The increase in PM as a percentage of spacecraft costs over time is statistically significant:

23. P(F-statistic) = 0.0139

24. 95% confidence interval around the slope = [0.0009,0.0063].R2 = 0.5073 Source: APL Mission Data History

26. 80% of the variation in SE can be explained by launch year.

27. That increase over time is statistically significant:

28. P(F-statistic)=0.0001

29. 95% confidence interval around the slope = [0.0043,0.0090].R2 = 0.8181 Source: APL Mission Data History

31. Although r2 is low (0.1585), the increase in PM as a percentage of spacecraft costs over time was statistically significant:

32. P(F-statistic) = 0.0162

33. 95% confidence interval around the slope =[0.0006,0.0059].R2 = 0.1585 Source: CADRe Database

35. About 40% of the variation in relative SE cost can be explained by launch year.

36. That result is statistically significant:

37. p(F-statistic) = 0.0000

38. 95% confidence interval around the slope = [0.0029,0.0072].R2 = 0.4080 Source: CADRe Database

39. Trends in Instrument Management and SE Costs, 1988−2009 The trends in instrument management and SE costs since 2002 seem to counter the trend of increasing management & engineering costs for space missions overall… …unless one notes that the delivered costs of space instruments launched since 2002 have in fact declined! Data Source: NASA Instrument Cost Model Dataset (JPL)

40. Drivers of Recent and Future PM/SE Cost Growth in APL Space Missions (1 of 2) APL’s Space Department was certified in 2009 as an AS-9100 vendor. This standard dictates: Quality improvement Variation control or management for key product characteristics Production and service provisions unique to the aerospace industry, such as part accountability, foreign object detection, production documentation, part identification, and part traceability Process/tooling change control and management Supply chain quality control, which covers the control of purchasing and acceptance processes Design and development control Product configuration control management Product quality, reliability, and safety control Continual improvement Requirement to respond to these new standards ensures that PM/SE costs will continue to increase at least in the near term.

41. Drivers of Recent and Future PM/SE Cost Growth for APL Space Missions (2 of 2) Other Drivers More technically challenging missions NPR 7120.5D Earned value management Safety requirements New SE documents Other drivers suggest that PM/SE costs will continue to increase near term.

43. a is a coefficient that depends on the selected analogy

44. HW DDTE Cost is the cost of design, development, test, and evaluation of spacecraft hardware in millions of FY06 dollars

45. b is a CER-specific value* A notable exception is The Aerospace Corporation’s SSCM (Small Spacecraft Cost Model), which predicts PM/SE costs for planetary missions, many of which must be completed in time to fit very tight launch windows, as a function of schedule duration. An example CER from NAFCOM 07

46. Problems with Using Mission/Hardware Costs as Predictors of PM/SE Costs Many cost and budget estimators rely on cost-to-cost factors for initial estimates of non-hardware program costs. Reliance on [predicted] hardware cost as the single predictor variable of PM/SE costs creates several problems for cost analysts and planners: Ignores the fact that similarly priced missions and hardware can vary widely in their non-hardware implementation costs because of unforeseen technical and programmatic uncertainties. When the initial prediction of hardware costs are low, the consequence is that final PM/SE costs will be underestimated (see next slide).

48. After CDR, however, growth in SE costs has tended to outstrip growth in spacecraft costs.*

49. With the exception of MESSENGER, the SE cost increases exceed those experienced by spacecraft bus development.*In other words, SE cost estimates based on spacecraft bus cost predictions at PDR would have been underestimated.

50. PM/SE CERs for Space Missions At least one spacecraft cost model owner has expressed an interest in developing multivariate CERs to predict PM/SE costs for space missions. NASA CADRe objective is to provide cost data on all NASA missions according to a consistent, clearly defined work breakdown (WBS) structure. CADRe data are being collected at major milestones. When sufficient data are available, PM/SE CERs can be developed. Also possible: comparison of PM/SE EACs across program milestones.

51. Findings (1 of 2) Management of space missions requires better means to estimate PM/SE effort and costs from project start. Both PM and SE contribute to project success. External and internal pressures for new PM/SE processes and products are resulting in higher costs over time. Current cost estimation methods rely on predicted hardware costs to predict PM/SE costs. Problems with this approach include resource underestimation and no ability to adjust estimates for complexity or increasing PM/SE requirements.

52. Findings (2 of 2) Cost data are becoming available to develop more robust, multivariate CERs. Example provided for instrument PM/SE cost prediction. May be too early to show effect of recent demands for additional PM/SE products on PM/SE effort. Statistical evidence suggests that what drives PM and SE effort may be slightly different: PM effort correlates more to the overall size and complexity of the mission—e.g., total cost, number of instruments, number of organizational participants. SE effort correlates more to technical complexity as measured by, e.g., spacecraft cost. In APL’s experience, in addition to dealing with technical and schedule issues that result in overruns within the spacecraft subsystem, SE has also been more profoundly and quantitatively impacted by increased requirements from sponsors and internal quality assurance improvements SE is being affected over time more directly and concretely than the management discipline, the major difference being attributed to rapidly increasing requirements and documentation This idea gains credence in light of relatively stable management costs.

53. Estimating PM/SE Cost in an Evolving Environment To account for the changing management/engineering environment, we currently estimate PM/SE costs of our space missions as follows: Start with a parametric estimate from whatever hardware estimating model is used Apply in-house cost-to-cost factors based on a moving average of last two or three space missions Smoothes out effects of implementation costs Adjusts for exceptions, anomalies Request senior managers and lead engineers generate bottom-up labor profile estimates along with bases of estimates (provides a “sanity check” against model-derived results) Compare the results to identify inconsistencies and to confirm the reasonableness of the labor profiles

54. PM/SE Resources in a Dynamic, Increasingly Complex Environment To the question posed by our management, “How much is enough?”, our answers are: More resources than 10 years ago. More (SE) resources than might be predicted by the hardware budget. Resources more like those of the most recent successfully completed mission, adjusted for Sufficient resources to meet requirements for all deliverables and products Resources needed to implement new processes Levels of resources consistent with the technical, organizational, schedule complexity of the mission Hopefully, as EVM and other processes are institutionalized, the process of answering the question can become more mechanical. On-going data collection and analysis will yield more precise and verifiable methods for estimating and managing SE/PM effort and anticipating future trends.

55. Questions?

56. Thank You