The document discusses the Spacecraft Software Engineering Branch's use of the CMMI models to oversee space flight software development. It provides an overview of the branch's selection of the CMMI-DEV model and definition of projects for a CMMI Maturity Level 2 rating. It also describes the branch's software oversight role for the Orion project, trials in planning software oversight projects, and emphasis on key CMMI process areas like project planning, project monitoring and control, and requirements management for its oversight work.
Apidays New York 2024 - Accelerating FinTech Innovation by Vasa Krishnan, Fin...
Strassner retherford
1. Spacecraft Software Engineering Branch/ER6
The CMMI Models
in
Oversight of Space Flight Software
Development
Liz Strassner
Lead, Software Process & Process Management Group
David Retherford
Sr. Software and Systems Engineer/ERC
Spacecraft Software Engineering Branch
NASA/JSC
Used with permission
2. Agenda
Spacecraft Software Engineering Branch/ER6
Introduction
Selection of CMMI Models for Spacecraft Software Engineering Branch
NASA Surveillance Strategies
Project Orion Surveillance Strategies
Orion Software Oversight
CMMI Maturity Level 2 Rating on Oversight
Who are we
What we did to earn CMMI Level 2
Trials & Tribulations of Software Oversight
What we are doing now
2/22/2010
2
3. Spacecraft Software Engineering & CMMI Models
Spacecraft Software Engineering Branch/ER6
We evaluated the three CMMI models (CMMI-ACQ, CMMI-SVC, & CMMI-
DEV) for applicability to our work
CMMI-SVC was discovered to be completely non-applicable
IT support help desk; Pizza delivery service
Oversight of a contract doesn’t meet the intent of “service” in this model
CMMI-ACQ was reviewed for applicability in a SCAMPI B
NASA/JSC organizational structure caused some goals to be completely out of
scope of the organization
We were fully compliant with goals and practices that were in scope of the organization
The model does not allow goals to be out of scope
CMMI-DEV
Software development work was obviously in scope
Software oversight is also in scope, but needs to be looked at from a systems
engineering perspective
Software oversight on a 30 year project is never-ending so needs to be broken
into smaller projects
2/22/2010
3
4. Selection of CMMI Model & Projects
Spacecraft Software Engineering Branch/ER6
Based on our experience with the SCAMPI B, we chose to go for a
CMMI Maturity Level 2 rating under the CMMI-DEV model
Selecting projects was an interesting experience
Software Development Projects
Many active software development projects to select from
A long history of flight software development projects from the former
Flight Software Branch that was merged into Spacecraft Software
Engineering Branch
Software Oversight
Role is significantly different than in the Shuttle or ISS Programs
Lead appraiser insisted that we have at least one project in this area
because of the large focus in the organization on oversight
Chose to create sub-projects out of milestone reviews
Software Requirements Review for SCAMPI B
“System PDR & Software Spiral Review" for SCAMPI A
2/22/2010
4
5. NASA Surveillance Strategies (From COTR Refresher Training 2-18-
2008)
Basic strategies: insight, oversight, or hybrid
Spacecraft Software Engineering Branch/ER6
Insight: relies on performance requirements & metrics. Use a
minimum set of product/process data to give adequate visibility into
the integrity of the product/process.
Contractor assumes more responsibility & accountability
Government steps back, evaluates deliverables & existing contractor
processes—not day-to-day close monitoring
Appropriate strategy for contracts with little cost risk, clearly-defined
products/services, & confidence in proven contractor performance
Oversight: rely on NASA-imposed product specification’s & process
controls, MIL standards, mandatory inspection points, etc. Intrusive
monitoring at contractor’s plant, & in-line NASA involvement in
contract work.
Appropriate strategy when NASA is assuming the liability for
performance or quality; when we determine oversight is necessary to
mitigate performance risk; or when the contractor is unproven
2/22/2010
5
6. Project Orion Surveillance
Spacecraft Software Engineering Branch/ER6
The Orion Contract as a whole uses the hybrid approach to
surveillance with pre-declared oversight in high risk areas
Building a space vehicle is high risk so the contract is Cost Plus Award
Fee (CPAF) type
Very extensive DRD requirements for insight
Software was pre-declared as a major risk item by the Agency, so
oversight of software was planned and scheduled
Joint NASA-Contractor Risk Board
Insight for Software
All of the NPR 7150.2 chapter 5 requirements exist as individual DRDs
CMMI Maturity Level 3 was levied on all software development
organizations
2/22/2010
6
7. Software Oversight of Orion
Spacecraft Software Engineering Branch/ER6
Oversight for Software
NASA co-chairs all boards and panels
Every Contractor functional area lead has a NASA counterpart
NASA participates in ALL software working groups
NASA has signature authority on all major software deliveries including
the Software Development Plan
NASA has an internal software test environment
NPR 7150.2 NASA Software Engineering levied on contract in its
entirety
NASA Standard 7009(I) Models & Simulations Standard levied on
contract in its entirety
NASA Software Assurance and Safety Standards levied on contract in a
“cut and paste” method into the contract (DRD CEV-S-001)
Constellation Software documents also levied by the “cut and paste”
method or by reference to specific sections or paragraphs of the CxP
document
2/22/2010
7
9. Spacecraft Software Engineering Branch
Spacecraft Software Engineering Branch/ER6
Spacecraft Software Engineering
Branch
Orion Flight Software System
Manager
Orion Recon System Manager
Orion Vehicle Systems
Management Manager
Altair Lead
Process & Process Management
Test & Verification Group GFE Group Technology Group Systems Engineering Group
Group
2/22/2010
9
11. NASA Orion Software Team Organization
Spacecraft Software Engineering Branch/ER6
Orion Software Manger is part of the Orion Project Office
Everyone below that level is matrixed out of Engineering organizations
at JSC, Ames, GRC, and LaRC
Frank Delgado/ER6 is the Flight Software System Manager and leads
the multicenter flight software team
Neil Townsend/ER6 is the Reconfiguration System Manager and leads
the multicenter software process and process management team
The Spacecraft Software Engineering Branch and the Orion Software
Team are approximately the same size and have a large overlap.
Orion
Software Spacecraft
Team Software
Engineering
Branch
2/22/2010
11
12. Year One – Baby Steps
Spacecraft Software Engineering Branch/ER6
Organization was very small
Only function was Orion oversight
Responsible for Orion Vehicle System Management (Class A)
Responsible for in-house flight software development environment for
all Orion software oversight (Class D & E)
Interviewed all employees to find out how they did their work
Wrote processes based on the “how they did their work”
Assumed no higher level organizational processes affected oversight
At a conceptual level, we did not have any problem mapping the
CMMI Specific Goals and Practices to processes
Late in the year, Orion Software Manager function moved here in
addition to Vehicle System Manager and the branch grew
2/22/2010
12
13. Year Two – Chaos Reigns
Former Flight Software Branch merged into the branch as one of
Spacecraft Software Engineering Branch/ER6
many steps JSC Engineering took to focus software
Brought in several GFE Flight Software Development projects, most in
maintenance and operations phase already CMMI Level 2 compliant
Consolidated all JSC Orion software oversight personnel
Brought in methodology for meeting Generic Goals and Practices useful
for all projects
GFE Projects must follow extensive higher level organizational
processes
New Branch Chief assigned
Perfect storm
Policies and processes for oversight crashed into GFE flight software
policies & processes
Previous branch-only view had to be extended to reflect division and
directorate infrastructure
New branch chief had new goals and expectations
2/22/2010
13
14. Year Two – Coming out of Chaos
Spacecraft Software Engineering Branch/ER6
CMMI Consultation
Bill Pierce provided advise on how to structure projects for oversight and
methods to apply CMMI practices to an oversight project
Selected projects: 1) Class A development, 2) Class D
development/acquisition, 3) Class A-E oversight
The former Intelligent Systems Branch was merged into Spacecraft
Software Engineering Branch
Primarily Class E research and technology software
SCAMPI B
Newly acquired projects from the recent merger were excluded
Uncovered a problem with PPQA
Discovered that we still had disconnects in applicability of division and
directorate policies and processes
Determined that our SwRR Oversight project was mis-scoped
Appraiser direction was to do better for PDR
2/22/2010
14
15. Year 3 – Success!
Spacecraft Software Engineering Branch/ER6
Solved PPQA problem temporarily by returning to original resources
Orion PDR slipped significantly, so best we could do was “plan” for
PDR without any action past that
Appraiser decided to merge our SwRR Project and our start of the PDR
Project into a single “Software Oversight” project for the purpose of the
SCAMPI A
Allowed us to show that we learned from SwRR and incorporated those
lessons into the PDR planning activities
Since all our development projects were new, we had to pull in an
old (CMMI L2 rated) project that was in maintenance to ensure end
of lifecycle coverage
This end of lifecycle coverage would be a problem for any new
organization
2/22/2010
15
17. Orion Software Spiral Review – Overview
Spacecraft Software Engineering Branch/ER6
Meet diverse set of goals & requirements
Orion project
SW Management Plan (CxP 72099)
Need to plan component-level PDR activities for LM SW Spiral
delivery
Follow general flow of Orion system PDR (CxP 72212)
Spacecraft Software Engineering Branch
Demonstrate an oversight project meeting CMMI ML 2
Agency
Software engineering requirements (NPR 7150.2)
Systems/project requirements (NPR 7123.1)
CMMI Dev 1.2
Demonstrate that oversight project can meet ML 2 SPs & GPs
2/22/2010
17
18. Software Oversight – Changes in Attitude
Spacecraft Software Engineering Branch/ER6
Requires an orthogonal change of view of
Requirements
Configuration Management
Evaluation & assessment vs. technical development
Requirements
Technical requirements of contractor product (e.g. spacecraft) differ from
oversight
As part of project tasks may influence or evaluate technical requirements
Actual oversight project requirement is to review products (e.g. SRS) ,
generate comments
Separation of concerns
Technical requirement development vs. oversight activities
2/22/2010
18
20. Orion PDR Flow – High Level Overview
PDR Plan & Kickoff
Spacecraft Software Engineering Branch/ER6
• PDR Ground Rules & POD
• PDR Review Process
• PDR RID Process
Subsystem Design Reviews (SSDR)
• Subsystem Specifications & IRDs
• Subsystem Design Data Books
• Subsystem Specific Design & Requirements Review Presentations
System & Module Review
• Subsystem Rollups
• System Specifications and Design Review Presentations
System PDR
• RID Process
• RID Screening, Review, & Dispositions
• PDR Pre-board & Board
Software PDR
• Specification & Design Tech/Peer Review Process
• SW Artifact Release and RID Process
• SW RID Screening, Review, & Disposition
• SW PDR Pre-board & Board
2/22/2010
20
21. Oversight Project – CMMI Model
Spacecraft Software Engineering Branch/ER6
CMMI Maturity Level 2 process areas for oversight
7 PA’s – PP, PMC, REQM, CM, PPQA, MA, SAM
CMMI Maturity Level 2 process area emphasis on managing project
Planning project, process, and products
Monitor & control project
2/22/2010
21
22. Spiral Review Project – CMMI Process Areas
Spacecraft Software Engineering Branch/ER6
Project Planning (PP) compliance
Develop project life cycle and Evaluate team assessment
process products
Document in project plan Configuration Management
Project Monitoring and (CM)
Control (PMC) NASA ICE/Windchill area for
Track team participation in document storage
contractor reviews Measurement and Analysis
Team assessment of reviews (MA)
Requirements Management Metrics for effort and effort per
(REQM) review/document
Screen/review requests from Supplier Agreement
Project Office or Branch Management (SAM)
Management Contractors part of Orion SW
Product & Process Quality team
Assurance (PPQA) Use division process for
Evaluate team process engineering service support
2/22/2010
22
23. Basic Software Oversight Requirements
Spacecraft Software Engineering Branch/ER6
Not the same as the product technical requirements
What ‘we’ must do to demonstrate
Contractor development of software technical baseline
Accomplished proper oversight of contractor
Assure adequacy of contractor efforts and products
As an oversight team we must
Review contractor technical efforts (requirements & design)
Generate comments/RIDs
Attend technical and peer reviews
Evaluate software product quality and maturity
Within known (agreed to) constraints
Basically – our requirements are:
To provide a set of software engineering services to Orion project
Engineering involvement and assessment
2/22/2010
23
24. Software PDR Oversight Lifecycle
Spacecraft Software Engineering Branch/ER6
Not traditional SW lifecycle – no actual SW being produced
Formulate and initiate the project/process (Inception)
Execute the plan/process (Execution)
Finish the project & baseline (Closeout)
Product/artifact data
Assessment/evaluation vs. SRS/IRD
Configuration Management
Oversight project artifacts, not contractor technical artifacts
2/22/2010
24
25. Oversight Phases
Spacecraft Software Engineering Branch/ER6
Inception Closeout
Define project lifecycle and Formal review release
organization RID inputs
Define review process(es) and RID review/screening
boards
RID dispositions
Define NASA technical and
SW Spiral Pre-board
management roles
SW Spiral board
Generate project plan
Artifact update/re-release
And away we go…
Execution
Monitor contractor software
development efforts
Review artifacts, generate
comments, attend reviews,
attend disposition and solution
meetings
2/22/2010
25
26. Process Area Considerations
Spacecraft Software Engineering Branch/ER6
PPQA REQM
No software artifacts (e.g. SRS, SDD, Oversight requirements NOT the
source code, etc.) same as technical product
Process and assessment report Activity and evaluation/assessment
focused based
Use outside organization – provide “The software shall do this …..” vs.
independent assessment “The project shall participate in
technical reviews.”
M&A
Control/manage between Orion SW
No technical measures (e.g. Mgr & Av&SW Project Office
SLOCs, etc.)
PP
Effort/product based
Project “within a project”
Hrs/review
Predetermined schedule
No. comments
Orion project/contractor level
Assessments
Completion
Quality
2/22/2010
26
27. SW Spiral Review – CMMI Appraisal Nuggets
Spacecraft Software Engineering Branch/ER6
Double vision syndrome KISS
Easy to get cross-eyed over Keep oversight plan, processes,
apparent identical areas between and tasks simple
LM and NASA oversight project Simple data/configuration
CMMI model vs. NASA management method
engineering focus Only controlling documents
Ex.: CM – whose CM? LM Reasonable/simple plan and
Project Link vs. NASA ICE process description results in
Ex.: Requirements management Effective CMMI Maturity Level
– Oversight project requirements 2 compliance
vs. contractor project technical
requirements
Project within a project
Scope focus very tight for
oversight project – milestone
based
Other work did not stand still
2/22/2010
27
28. Typical Review Process
Spacecraft Software Engineering Branch/ER6
Perform Tech
Review
Product Owner
SFM/SWE Perform Peer
Review Disposition
Product
Product Comments
Comments
Initial attempt to use SE process to describe & define process
oversight
Ex.: Use case diagram to represent project process for technical and peer
reviews
Each use case may breakdown and be described in project plan for each
specific process
Value of approach is still being evaluated (may refine for next version &
evaluate for improvement)
2/22/2010
28
30. Next Steps
Spacecraft Software Engineering Branch/ER6
New Project – Software Process Improvement Project
Among other things, earn CMMI Maturity Level 3 by April 2011
Requires major culture change to implement OT, OPF, & OPD
Orion Software PDR Project
Continues to evolve
Creating checklists to further refine measures from reviews
Expect to learn enough to provide guidance to development projects on
Planning and scheduling peer reviews
Planning and scheduling milestone reviews
Expect confusion in implementing PI, VAL, VER, and TS because our
products are brain-power related rather than code
2/22/2010
30