Strassner retherford

Spacecraft Software Engineering Branch/ER6

The CMMI Models
in
Oversight of Space Flight Software
Development

Liz Strassner
Lead, Software Process & Process Management Group
David Retherford
Sr. Software and Systems Engineer/ERC
Spacecraft Software Engineering Branch
NASA/JSC

Used with permission

Agenda

 Introduction
 Selection of CMMI Models for Spacecraft Software Engineering Branch
 NASA Surveillance Strategies
 Project Orion Surveillance Strategies
 Orion Software Oversight
 CMMI Maturity Level 2 Rating on Oversight
 Who are we
 What we did to earn CMMI Level 2
 Trials & Tribulations of Software Oversight
 What we are doing now

2/22/2010
2

Spacecraft Software Engineering & CMMI Models

 We evaluated the three CMMI models (CMMI-ACQ, CMMI-SVC, & CMMI-
DEV) for applicability to our work
 CMMI-SVC was discovered to be completely non-applicable
 IT support help desk; Pizza delivery service
 Oversight of a contract doesn’t meet the intent of “service” in this model
 CMMI-ACQ was reviewed for applicability in a SCAMPI B
 NASA/JSC organizational structure caused some goals to be completely out of
scope of the organization
 We were fully compliant with goals and practices that were in scope of the organization
 The model does not allow goals to be out of scope
 CMMI-DEV
 Software development work was obviously in scope
 Software oversight is also in scope, but needs to be looked at from a systems
engineering perspective
 Software oversight on a 30 year project is never-ending so needs to be broken
into smaller projects

2/22/2010
3

Selection of CMMI Model & Projects

 Based on our experience with the SCAMPI B, we chose to go for a
CMMI Maturity Level 2 rating under the CMMI-DEV model
 Selecting projects was an interesting experience
 Software Development Projects
 Many active software development projects to select from
 A long history of flight software development projects from the former
Flight Software Branch that was merged into Spacecraft Software
Engineering Branch
 Software Oversight
 Role is significantly different than in the Shuttle or ISS Programs
 Lead appraiser insisted that we have at least one project in this area
because of the large focus in the organization on oversight
 Chose to create sub-projects out of milestone reviews
 Software Requirements Review for SCAMPI B
 “System PDR & Software Spiral Review" for SCAMPI A

2/22/2010
4

NASA Surveillance Strategies (From COTR Refresher Training 2-18-
2008)

 Basic strategies: insight, oversight, or hybrid

 Insight: relies on performance requirements & metrics. Use a
minimum set of product/process data to give adequate visibility into
the integrity of the product/process.
 Contractor assumes more responsibility & accountability
 Government steps back, evaluates deliverables & existing contractor
processes—not day-to-day close monitoring
 Appropriate strategy for contracts with little cost risk, clearly-defined
products/services, & confidence in proven contractor performance
 Oversight: rely on NASA-imposed product specification’s & process
controls, MIL standards, mandatory inspection points, etc. Intrusive
monitoring at contractor’s plant, & in-line NASA involvement in
contract work.
 Appropriate strategy when NASA is assuming the liability for
performance or quality; when we determine oversight is necessary to
mitigate performance risk; or when the contractor is unproven

2/22/2010
5

Project Orion Surveillance

 The Orion Contract as a whole uses the hybrid approach to
surveillance with pre-declared oversight in high risk areas
 Building a space vehicle is high risk so the contract is Cost Plus Award
Fee (CPAF) type
 Very extensive DRD requirements for insight
 Software was pre-declared as a major risk item by the Agency, so
oversight of software was planned and scheduled
 Joint NASA-Contractor Risk Board
 Insight for Software
 All of the NPR 7150.2 chapter 5 requirements exist as individual DRDs
 CMMI Maturity Level 3 was levied on all software development
organizations

2/22/2010
6

Software Oversight of Orion

 Oversight for Software
 NASA co-chairs all boards and panels
 Every Contractor functional area lead has a NASA counterpart
 NASA participates in ALL software working groups
 NASA has signature authority on all major software deliveries including
the Software Development Plan
 NASA has an internal software test environment
 NPR 7150.2 NASA Software Engineering levied on contract in its
entirety
 NASA Standard 7009(I) Models & Simulations Standard levied on
contract in its entirety
 NASA Software Assurance and Safety Standards levied on contract in a
“cut and paste” method into the contract (DRD CEV-S-001)
 Constellation Software documents also levied by the “cut and paste”
method or by reference to specific sections or paragraphs of the CxP
document
2/22/2010
7


Spacecraft Software Engineering Branch & CMMI
Level 2

2/22/2010
8

Spacecraft Software Engineering Branch

Spacecraft Software Engineering
Branch

Orion Flight Software System
Manager
Orion Recon System Manager
Orion Vehicle Systems
Management Manager
Altair Lead

Process & Process Management
Test & Verification Group GFE Group Technology Group Systems Engineering Group
Group

2/22/2010
9

Orion Software Team Interfaces

Customer - Provider
Orion
Constellation Project
Office
Provider - Customer
Orion
Stakeholder – Provider Rep
Software
Team

Customer Rep - Provider - Customer
Provider

Prime
Contractor

Subcontractor
Subcontractor
Subcontractor

Subcontractor Subcontractor
Subcontractor

2/22/2010
10

NASA Orion Software Team Organization

 Orion Software Manger is part of the Orion Project Office
 Everyone below that level is matrixed out of Engineering organizations
at JSC, Ames, GRC, and LaRC
 Frank Delgado/ER6 is the Flight Software System Manager and leads
the multicenter flight software team
 Neil Townsend/ER6 is the Reconfiguration System Manager and leads
the multicenter software process and process management team
 The Spacecraft Software Engineering Branch and the Orion Software
Team are approximately the same size and have a large overlap.

Orion
Software Spacecraft
Team Software
Engineering
Branch

2/22/2010
11

Year One – Baby Steps

 Organization was very small
 Only function was Orion oversight
 Responsible for Orion Vehicle System Management (Class A)
 Responsible for in-house flight software development environment for
all Orion software oversight (Class D & E)
 Interviewed all employees to find out how they did their work
 Wrote processes based on the “how they did their work”
 Assumed no higher level organizational processes affected oversight
 At a conceptual level, we did not have any problem mapping the
CMMI Specific Goals and Practices to processes
 Late in the year, Orion Software Manager function moved here in
addition to Vehicle System Manager and the branch grew

2/22/2010
12

Year Two – Chaos Reigns

 Former Flight Software Branch merged into the branch as one of

many steps JSC Engineering took to focus software
 Brought in several GFE Flight Software Development projects, most in
maintenance and operations phase already CMMI Level 2 compliant
 Consolidated all JSC Orion software oversight personnel
 Brought in methodology for meeting Generic Goals and Practices useful
for all projects
 GFE Projects must follow extensive higher level organizational
processes
 New Branch Chief assigned
 Perfect storm
 Policies and processes for oversight crashed into GFE flight software
policies & processes
 Previous branch-only view had to be extended to reflect division and
directorate infrastructure
 New branch chief had new goals and expectations
2/22/2010
13

Year Two – Coming out of Chaos

 CMMI Consultation
 Bill Pierce provided advise on how to structure projects for oversight and
methods to apply CMMI practices to an oversight project
 Selected projects: 1) Class A development, 2) Class D
development/acquisition, 3) Class A-E oversight
 The former Intelligent Systems Branch was merged into Spacecraft
Software Engineering Branch
 Primarily Class E research and technology software
 SCAMPI B
 Newly acquired projects from the recent merger were excluded
 Uncovered a problem with PPQA
 Discovered that we still had disconnects in applicability of division and
directorate policies and processes
 Determined that our SwRR Oversight project was mis-scoped
 Appraiser direction was to do better for PDR

2/22/2010
14

Year 3 – Success!

 Solved PPQA problem temporarily by returning to original resources
 Orion PDR slipped significantly, so best we could do was “plan” for
PDR without any action past that
 Appraiser decided to merge our SwRR Project and our start of the PDR
Project into a single “Software Oversight” project for the purpose of the
SCAMPI A
 Allowed us to show that we learned from SwRR and incorporated those
lessons into the PDR planning activities
 Since all our development projects were new, we had to pull in an
old (CMMI L2 rated) project that was in maintenance to ensure end
of lifecycle coverage
 This end of lifecycle coverage would be a problem for any new
organization

2/22/2010
15


Trials & Tribulations of planning
Software Oversight

Orion Software Spiral Review – Overview

 Meet diverse set of goals & requirements
 Orion project
 SW Management Plan (CxP 72099)
 Need to plan component-level PDR activities for LM SW Spiral
delivery
 Follow general flow of Orion system PDR (CxP 72212)
 Spacecraft Software Engineering Branch
 Demonstrate an oversight project meeting CMMI ML 2
 Agency
 Software engineering requirements (NPR 7150.2)
 Systems/project requirements (NPR 7123.1)
 CMMI Dev 1.2
 Demonstrate that oversight project can meet ML 2 SPs & GPs

2/22/2010
17

Software Oversight – Changes in Attitude

 Requires an orthogonal change of view of
 Requirements
 Configuration Management
 Evaluation & assessment vs. technical development
 Requirements
 Technical requirements of contractor product (e.g. spacecraft) differ from
oversight
 As part of project tasks may influence or evaluate technical requirements
 Actual oversight project requirement is to review products (e.g. SRS) ,
generate comments
 Separation of concerns
 Technical requirement development vs. oversight activities

2/22/2010
18

SW PDR Project Plan Impact Sources

2/22/2010
19

Orion PDR Flow – High Level Overview

PDR Plan & Kickoff

• PDR Ground Rules & POD
• PDR Review Process
• PDR RID Process

Subsystem Design Reviews (SSDR)
• Subsystem Specifications & IRDs
• Subsystem Design Data Books
• Subsystem Specific Design & Requirements Review Presentations

System & Module Review
• Subsystem Rollups
• System Specifications and Design Review Presentations

System PDR
• RID Process
• RID Screening, Review, & Dispositions
• PDR Pre-board & Board

Software PDR
• Specification & Design Tech/Peer Review Process
• SW Artifact Release and RID Process
• SW RID Screening, Review, & Disposition
• SW PDR Pre-board & Board

2/22/2010
20

Oversight Project – CMMI Model

 CMMI Maturity Level 2 process areas for oversight
 7 PA’s – PP, PMC, REQM, CM, PPQA, MA, SAM
 CMMI Maturity Level 2 process area emphasis on managing project
 Planning project, process, and products
 Monitor & control project

2/22/2010
21

Spiral Review Project – CMMI Process Areas

 Project Planning (PP) compliance
 Develop project life cycle and  Evaluate team assessment
process products
 Document in project plan  Configuration Management
 Project Monitoring and (CM)
Control (PMC)  NASA ICE/Windchill area for
 Track team participation in document storage
contractor reviews  Measurement and Analysis
 Team assessment of reviews (MA)
 Requirements Management  Metrics for effort and effort per
(REQM) review/document
 Screen/review requests from  Supplier Agreement
Project Office or Branch Management (SAM)
Management  Contractors part of Orion SW
 Product & Process Quality team
Assurance (PPQA)  Use division process for
 Evaluate team process engineering service support
2/22/2010
22

Basic Software Oversight Requirements

 Not the same as the product technical requirements
 What ‘we’ must do to demonstrate
 Contractor development of software technical baseline
 Accomplished proper oversight of contractor
 Assure adequacy of contractor efforts and products
 As an oversight team we must
 Review contractor technical efforts (requirements & design)
 Generate comments/RIDs
 Attend technical and peer reviews
 Evaluate software product quality and maturity
 Within known (agreed to) constraints
 Basically – our requirements are:
 To provide a set of software engineering services to Orion project
 Engineering involvement and assessment

2/22/2010
23

Software PDR Oversight Lifecycle

 Not traditional SW lifecycle – no actual SW being produced
 Formulate and initiate the project/process (Inception)
 Execute the plan/process (Execution)
 Finish the project & baseline (Closeout)
 Product/artifact data
 Assessment/evaluation vs. SRS/IRD
 Configuration Management
 Oversight project artifacts, not contractor technical artifacts

2/22/2010
24

Oversight Phases

 Inception  Closeout
 Define project lifecycle and  Formal review release
organization  RID inputs
 Define review process(es) and  RID review/screening
boards
 RID dispositions
 Define NASA technical and
 SW Spiral Pre-board
management roles
 SW Spiral board
 Generate project plan
 Artifact update/re-release
 And away we go…
 Execution
 Monitor contractor software
development efforts
 Review artifacts, generate
comments, attend reviews,
attend disposition and solution
meetings

2/22/2010
25

Process Area Considerations

 PPQA  REQM
 No software artifacts (e.g. SRS, SDD,  Oversight requirements NOT the
source code, etc.) same as technical product
 Process and assessment report  Activity and evaluation/assessment
focused based
 Use outside organization – provide  “The software shall do this …..” vs.
independent assessment “The project shall participate in
technical reviews.”
 M&A
 Control/manage between Orion SW
 No technical measures (e.g. Mgr & Av&SW Project Office
SLOCs, etc.)
 PP
 Effort/product based
 Project “within a project”
 Hrs/review
 Predetermined schedule
 No. comments
 Orion project/contractor level
 Assessments
 Completion
 Quality

2/22/2010
26

SW Spiral Review – CMMI Appraisal Nuggets

 Double vision syndrome  KISS
 Easy to get cross-eyed over  Keep oversight plan, processes,
apparent identical areas between and tasks simple
LM and NASA oversight project  Simple data/configuration
 CMMI model vs. NASA management method
engineering focus  Only controlling documents
 Ex.: CM – whose CM? LM  Reasonable/simple plan and
Project Link vs. NASA ICE process description results in
 Ex.: Requirements management  Effective CMMI Maturity Level
– Oversight project requirements 2 compliance
vs. contractor project technical
requirements
 Project within a project
 Scope focus very tight for
oversight project – milestone
based
 Other work did not stand still
2/22/2010
27

Typical Review Process

Perform Tech
Review

Product Owner
SFM/SWE Perform Peer
Review Disposition
Product
Product Comments
Comments

 Initial attempt to use SE process to describe & define process
oversight
 Ex.: Use case diagram to represent project process for technical and peer
reviews
 Each use case may breakdown and be described in project plan for each
specific process
 Value of approach is still being evaluated (may refine for next version &
evaluate for improvement)
2/22/2010
28


2/22/2010
Next Steps

29

Next Steps

 New Project – Software Process Improvement Project
 Among other things, earn CMMI Maturity Level 3 by April 2011
 Requires major culture change to implement OT, OPF, & OPD
 Orion Software PDR Project
 Continues to evolve
 Creating checklists to further refine measures from reviews
 Expect to learn enough to provide guidance to development projects on
 Planning and scheduling peer reviews
 Planning and scheduling milestone reviews
 Expect confusion in implementing PI, VAL, VER, and TS because our
products are brain-power related rather than code

2/22/2010
30

Strassner retherford

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (20)

Similar to Strassner retherford

Similar to Strassner retherford (20)

More from NASAPMC

More from NASAPMC (20)

Recently uploaded

Recently uploaded (20)

Strassner retherford