1. CESE
Automated Testing of Large Multi-
language SW Systems using Cloud
Computing
Executive Status Report
Principal Investigator (PI): Dr. Mikael Lindvall, CESE
NASA POC: Markland Benson, White Sands
Team members:
Dharma Ganesan, Dr. Chris Ackermann(CESE)
GMSEC, CFS, Space Network, MSL
© 2011 Fraunhofer USA, Inc. 1
Center for Experimental Software Engineering

2. CESE
Motivation
• Bugs can lead to deaths, injuries, or financial
loss
• Software testing is necessary to find the
presence of bugs
• Software testing consumes 50% - 75% of the
development effort
• Many NASA projects could benefit from
automated test generation
• In fact, several skeptical engineers noted that
the proposed work is illuminating!
© 2011 Fraunhofer USA, Inc.
Center for Experimental Software Engineering

3. CESE
Technical Context
• The system(s) under test (SUT) are complex
• Multiple programming languages
– Ada, C, C++, Fortran, Java
• Distributed legacy systems
– More than 20 yrs old
– Runs on multiple machines and involves various
OSes,e.g. OpenVMS
• GUI intensive
– Web, Java Swing, AWT, X Windows, Motif
© 2011 Fraunhofer USA, Inc.
3
Center for Experimental Software Engineering

4. CESE
Problems
• Test cases are developed manually
• Some test execution is automated (e.g., JUnit)
• Test cases miss valid “corner” cases
• Test cases are also programs: not easy for non-
technical stakeholders to understand
• Difficult to summarize what was tested
 Approach: Lightweight Model-based Test
Generation and Execution
© 2011 Fraunhofer USA, Inc.
4
Center for Experimental Software Engineering

5. CESE
Approach
• Develop a model of the system under test
• The model contains actions and expected results
• Automatically derive test cases from the model
• Execute the test cases
• Decide when to stop testing based on model
coverage!
© 2011 Fraunhofer USA, Inc.
Center for Experimental Software Engineering

6. CESE
An Example Model
APIs of the module under test
public interface IConnection
{
public Status Connect();
public Status Disconnect();
© 2011 Fraunhofer USA, Inc.
}
Center for Experimental Software Engineering

7. CESE
Currently Targeted Projects
• GMSEC – Reusable framework for ground systems
– Modeled the Core API and Applications
– Generated executable test cases from the model
– Confirmed defects/violations reported and fixed
• Core Flight Software – Reusable framework for
flight systems
– Modeled the OS abstraction layer (OSAL)
– Generated executable test cases from the model
– Confirmed defects/violations reported and fixed
– Test cases delivered to the CFS team
© 2011 Fraunhofer USA, Inc.
7
Center for Experimental Software Engineering

8. CESE
Currently Targeted Projects
• Space Network – White Sands
– Evaluating different test execution tools suitable
for Open VMS environment
– E.g. Image-based test execution tools: EggPlant
and Sikuli
– Challenges in this project:
• SUT not possible to run at Fraunhofer
• Must visit Goddard with an escort (secured building)
• Not possible to connect to Internet
© 2011 Fraunhofer USA, Inc.
8
Center for Experimental Software Engineering

9. CESE
Currently Targeted Projects
• Mars Science Laboratory (MSL) – JPL
– Can current test cases be modeled and used to
generate new test cases?
• New set cover scenarios not previously covered?
• New approach more cost effective?
– Challenges in this project:
• SUT not possible to run at Fraunhofer
• Software is sensitive (ITAR)
© 2011 Fraunhofer USA, Inc.
9
Center for Experimental Software Engineering

10. CESE
Current Results
• An end-to-end approach for test generation
– Successfully used on the GMSEC and the
CFS and detected bugs
• Approach works well on different levels:
– API (Module interface) level testing
– GUI testing
• Approach transferred to an intern at GSFC
– The intern found new bugs using the approach
– Demonstrates the approach is lightweight
© 2011 Fraunhofer USA, Inc.
10
Center for Experimental Software Engineering

11. CESE
Summary and Next Steps
• We’re building a new approach that
– Helps in test automation using lightweight
models
• Prototype approach successfully applied
to GMSEC and CFS/cFE
• Next steps are to apply to the Space
Network and the MSL projects
• Examples and technical details in
technical presentation!
© 2011 Fraunhofer USA, Inc.
11
Center for Experimental Software Engineering