The presentation shows how a test management solution has been established for a defence project in compliance with a set of applicable standards using an integrated IBM Rational tool chain consisting of Rational Quality Manager for test management, IBM Rational DOORS for requirement management, IBM Rational Team Concert for defect management, IBM Rational Publishing Engine for automatic generation of project deliverables and last - but not least - IBM Rational Insight for trend and status reporting.

1. © 2014 IBM Corporation
Deployment of a test
management solution for a
defence project using an
integrated tool chain
Code 1183
Dr. Einar Karlsen (IBM, Co-Speaker)
Thomas Schöning (Airbus DS)

2. 1
Introduction
Dr. Einar Karlsen (IBM)
-Technical Solution Architect
Thomas Schöning (Airbus Defence and Space)
- Integration and Test Manager
Thomas Südbröcker (IBM)
- Technical Sales and Solutions + Community of Practices
Leader Quality Management (Germany)

3. 2
• Subcontractor in a NATO Project with requirement based development,
test and formal verification of a complex system entity
• Find a proper tooling solution for test and verification process using the
IBM Rational tooling chain (Jazz Platform)
– Support for relevant standards ISO/IEE 15288, ISO/IEEE 12207,
ISO 9001, ISO/IEC 9126
– Support for relevant process: Waterfall (V-Model) with incremental
SW Development
– Integration with IBM Rational DOORS to accommodate for
requirement traceability
– Automatic document generation for requirements, design and test
artefacts (plans, procedures, reports)
– Insight into status and trend of the testing efforts (metrics)
Setting the Scene – Situation

4. 3
• Challenge
– Unknown effort to adequately install, integrate and customize the
whole tooling chain to perform modelling, implementation, and test
management
– Appropriate test management support (planning, execution,
reporting, document generation, etc) according to required
process?
• Improvements
– Decrease of costs, risks, inconsistencies and time-to-market
through automatic generation of project deliverables
– Collection of metrics to measure test design progress,
requirements coverage
– Online/Offline determination of test results while performing the test
Setting the Scene – Risk and Opportunity

5. 4
• Profound orientation for configuration and customization of IBM
Rational Tools for a tool-aided test management solution (planning,
execution, reporting, documentation) for similar test activities
1. Start early to implement test management prototyping to better
estimate the effort to adapt the tools to the project needs
2. Invite technical professionals to have guidance and avoid pitfalls
3. Perform team training
4. Look at the deliverables before customizing your tool chain
Setting the Scene – Position and Benefits

6. 5
• Introduction
– Programme Overview
– Test and Evaluation Strategy
• Test management
– Solution Overview
– Solution Architecture
– Document Generation
– Metrics and Dashboards
• Summary
Agenda

7. 6
• Confidential, multinational NATO C4ISR Project (C4ISR=Command,
Control, Communications, Computers, Intelligence, Surveillance, and
Reconnaissance)
– Unmanned Air, Command and Control Entities
– Transportable and Mobile Ground Stations for data exception,
processing, and archiving facilities
– Communication links to other NATO and interoperable national
ground stations, and to NATO and national Command and Control
systems
– Exchange of standardized data via different interfaces and
communication lines
• Airbus DS responsible for development, test and formal verification of a
system entity consisting of several Subsystems
Programme Overview

8. Test and Evaluation Strategy
Verification
Development
Testing
Entity
Subsystems
CSCIsHWCIs
ServicesComponents
Development
Testing
Development
TestingTest
Test
Test
Verification
Integration

9. 8
• Requirements of System Entity, Subsystems, Computer Software
Configuration Items (CSCIs), and Hardware Configuration Items
(HWCIs) are kept in DOORS database with traceable entries
• Testing/Verification is classified either to be
– Informal Tests
• conduct informal tests throughout the system development
– Subcontractor Development Tests
• For CSCI, HWCI, HW/CSCI groupings, Subsystem level,
Entities and Integration testing
– Formal Verification
• Verification of allocated customer requirements based on proof
of derived requirements (Entity, Subsystem, HW/CSCI
groupings)
Test and Evaluation Strategy

10. Test Management – Solution Overview
9

11. 10
Test Management – Solution Overview
• Test Plan
– consists of a number of Test Cases used to proof requirements
– is linked with one or more Requirements Modules (Test Plan views)
coverage and traceability (verification level)
• Test Case
– is linked to one or more particular requirements belonging to the
requirements view(s) proofing requirements on different levels
• Test Script
– keeps the Test Procedure with single steps for test execution
• Test Case Execution Record/Test Case Result
– container to keep the Test Case results depending on iteration
(integration, dry-run, acceptance) and test environment
• PTR (=Problem Trouble Report)
– keeps the defects occurred during Test Case execution

12. Test Management – Solution Architecture
11
IBM Rational
DOORS
IBM Rational Insight
IBM Rational Publishing Engine
REST, ETL REST, ETL
IBM Rational
Quality Manager
IBM Rational
Team Concert
REST, ETL
Dashboards, Status and trend reports
Test Plans, Procedures and Reports
(Project Deliverables)
RESTREST REST
OSLC
links
OSLC
links

13. 12
Test Management – Test Document Generation

14. Test Document Generation – Master Test List

15. Test Document Generation – Test Case Details

16. 15
• Metrics: What you can measure that can be done!
• Metric types: Project, Product, Process, and Personnel
• Definition of metrics for monitoring and control test planning, design,
development and execution
– test planning
• Percentage of requirements having a test case
– test design (analysis and concept)
• Percentage of test case with different status (draft, ready for
review, approved)
– test development
• Percentage of test scripts with different status (draft, ready for
review, approved)
– test execution
• Percentage of planned test cases being executed (passed,
passed with condition, failed, blocked, error)
Test Management – Metrics and Dashboards

17. 16
Metrics and Dashboards – Test Case Design

18. 17
Metrics and Dashboards – Test Case Execution

19. 18
• Tooling Solution (Jazz platform) for test management process
with combined DT&E strategy (Informal, Development Tests,
Formal Verification) applied for system entity (NATO Project)
• Test Management Solution (Overview and Architecture)
– Test Plans linked with requirements view
– Test Cases linked with requirements
– Test Execution Records keeping the results for each Iteration
– Definition of timelines, iterations to support test planning and
execution
• Test Document Generation
– Tooling adaption for artefact templates (Test Plan, Test Case)
– Examples (Master Test List, Test Case Details)
• Metrics and Dashboards
– metrics for monitoring and control test planning, design,
development and execution
Summary

20. Articles on developerWorks
• RQM Template authoring articles
on developer Works.
– Using IBM Rational Publishing
Engine to generate compliance
documents: Report generation
for Rational Quality Manager,
introducing REST service
URLs, importing schemas, and
basic template authoring
– Using IBM Rational Publishing
Engine to generate compliance
documents: Part 2. Report
generation for Rational Quality
Manager, showing how to
extract information from grid
sections and report on test
results and related defects

21. Thank You!
Your Feedback is Important!
Access the Innovate agenda tool to complete your
session surveys from your smartphone, laptop or
conference kiosk.

22. 21
• Test Plan:
– Artifact Templates (keep Test Cases and TCERs only)
– Artifact Categories (manufacturer, part number, type) to distinguish
between different plans (system, software,etc)
• Test Case:
– Artifact Template has been customized to the documentation
format needs (Test Plan, Test Procedure)
– Artifact Categories (test area, test method, test event, test type)
and RQM section (description, test objective, requirement links,
estimation) are used to appropriately keep Test Case data
• Timelines (Development and Production Phase)
• Iteration Types (Milestone, Software Test, System Test)
• Iteration
– directly corresponds with incremental software development
(integration tests, dry run, acceptance test)
– support generation of TCER
Test Management – Tooling adaption

23. 22
• Administration of project requirements and test cases as well as the
creation of bi-directional dependencies between them
• Planning test effort and milestones
• Defining test goals, acceptance and quality criteria
• Deriving test cases derived from requirement specifications
• Estimation of the effort of the test activities
• Definition of a time plan for each test iteration with finish dates of key
activities
• Collection of different test environments and the generation of test
configurations
• Creation and administration of test cases (manual and automated tests)
• Recording test results, creating defect and evaluating reports
• Support review and release process for the test plan and individual test
cases.
IBM Rational Tools – Usage