The case study described in this presentation has taken place at a medical equipment manufacturer. The product developed was a medical x-ray device used during surgery operations. The system generates x-rays (called exposure) and a detector creates images of the patient based on the detected x-ray beams (called image acquisition). The image pipeline is real-time with several images per second, so the surgeon can e.g. see exactly where he is cutting the patient. The presentation describes the approach that has been taken to develop an automatic testing framework in order to execute reliability test cases and identify reliability issues. To achieve the control of the system under test, the existing hardware interfaces (physical buttons of the different keyboards, handswitches and footswitches) were used to inject the system with actions (with the use of LabVIEW). This has been done to minimize the so-called probe effect. The expected results of the test cases have been automatically retrieved from the log files generated by the system. This way the test framework could react on system failures immediately, without wasting valuable test time on scarce test systems. The log files were used to extract information about the performed actions and failures in order to measure the MTBF (Mean Time Between Failures) of different critical system functions (like start-up of the system, and image acquisition). The Crow-AMSAA model for reliability measurements has been chosen to report reliability metrics to the organization. A Return-On-Investment calculation has been performed to get buy-in from senior management who provided additional funding to further develop the testing framework, and to apply the same ideas to different products and projects. The presentation explains the points which were crucial for the success of this approach to automated reliability testing and briefly explains future plans and extensions (e.g. operational profiles).

1. Automated Reliability
Testing via hardware
interfaces
EuroSTAR 2011
Bryan Bakker
November, 2011

2. © Sioux Embedded Systems | Confidential | 2
Contents
 Sioux
 Intro – The need for action
 Increment 1 – First success
 Buy-In from management
 Increment 2 – A language for the testers
 Increment 3 – Logfile interpretation
 ROI and Crow-AMSAA
 Results – key success factors
2

3. © Sioux Embedded Systems | Confidential | 3
About Bryan Bakker
 Test Expert
 Certifications: ISTQB, TMap, Prince2
 Member of ISTQB Expert Level on Test Automation
 Accredited tutor of ISTQB Foundation
 Domains: medical systems, professional security
systems, semi-industry, electron microscopy
 Specialties: test automation, integration testing, design
for testability, reliability testing
3

4. About Sioux
© Sioux Embedded Systems | Confidential | 4
HERENTALS
MOSCOW
NEDERWEERT
EINDHOVEN
UTRECHT

5. © Sioux Embedded Systems | Confidential | 5
Intro – The need for action
Medical Surgery Device:
 X-ray exposure + acquisition during surgery activities
 Real-time image chain
 Mobile device (frequently off/on)
 Quality and testing considered
important in organization
Reliability was an issue:
 “Frequent” startup failures
 Aborted acquisitions
 Always safe… but not reliable!
5

6. © Sioux Embedded Systems | Confidential | 6
The need for action
 Reliability issues are nasty to analyse, solve and test
 Fixing defects in field (remember Boehm)
 Impact on other projects (development + test resources)
 High service costs
 Troublesome system test (up to 15 cycles!!)
6
Requirements Design Implementation Test Operation
Cost of defect fix (Barry Boehm)

7. © Sioux Embedded Systems | Confidential | 7
The need for action
 Start with automated reliability tests (simple, short)
 Quick and dirty at first
 No SW resources available
 To help with automation
 Implementing test interfaces
 Goal: show (quick) that reliability issues can be
reproduced
 Expectation: … then attention and funding would
increase
7

8.  Hardware interfaces used to invoke actions on SUT
 Buttons on different keyboards
 Handswitches
 Footswitches
 Different power-switches
 LabVIEW generates hardware signals
 Test cases defined in LabVIEW
 Only logfiles stored, no other verification performed
 No software changes needed for this approach
Increment 1 – First success
© Sioux Embedded Systems | Confidential | 8

9. Increment 1 – First success
© Sioux Embedded Systems | Confidential | 9

10. Increment 1 – First success
 Simple, but quick first results
 Multiple reliability issues found
 Work to do for the developers
© Sioux Embedded Systems | Confidential | 10
System Under Test
Hardware Abstraction Layer
(LabVIEW)
Input (Hardware)
Test cases
(LabVIEW)
Control
Output
Log file
Test Framework
1st
Increment

11.  Several defects found were already known:
 Customer issues
 Not reproducible -> no solution
 Now: developers could work on them
 And fix could be tested as well
 Several presentations given explaining the approach
 And… get clear what we are looking for!
 Primary functions should work reliable
Management buy-in
© Sioux Embedded Systems | Confidential | 11

12. Definition of Reliability Hit
© Sioux Embedded Systems | Confidential | 12
PF = Primary Function
Primary function: e.g. startup, acquisition
Non-primary function: e.g. printing, post-viewing

13.  LabVIEW is not that easy
 Provide general scripting language (Ruby)
 Ruby interfacing with LabVIEW via abstraction layer
 Development of test libraries was started
 Still only control, no verification
 Log file analysis after test (tools were used)
Increment 2
A language for the testers
© Sioux Embedded Systems | Confidential | 13

14. Increment 2
A language for the testers
© Sioux Embedded Systems | Confidential | 14
System Under Test
Hardware Abstraction Layer
(LabVIEW)
Input (Hardware)
Test cases + libraries
(Ruby)
Control
Output
Log file
Test Framework
2
nd
Increment

15.  Logfile scanned during test case execution
 Determine pass/fail criteria
 Detect system states and act upon:
 Hot generator  extensive acquisition not possible
 Execute other test cases (e.g. power-cycle), until
 Generator has cooled down
 Log file analysis after test was still performed
 Still no software changes in the SUT, but existing
interfaces were available now
Increment 3
Logfile interpretation
© Sioux Embedded Systems | Confidential | 15

16. Increment 3
Logfile interpretation
© Sioux Embedded Systems | Confidential | 16
System Under Test
Hardware Abstraction Layer
(LabVIEW)
Input (hardware &
software)
Test Execution Environment
incl. test cases and library
(Ruby)
Control
Repository
(Testcases+Results)
Test Framework
3rd
Increment
Output
Result
Test Scheduler
(Ruby)

17.  Best practise:
 Test actions by external interfaces
 Test verification by log file and internal state
information
 System statistics extracted from logfile:
 Number of startups (succeeded and failed)
 Number of acquisitions (succeeded and failed)
Increment 3
Logfile interpretation
© Sioux Embedded Systems | Confidential | 17

18.  Reliability hits could be identified from logfile (semi-
automatic)
 Pareto charts
 Performance measurements
(timing info in logfile)
 Crow-AMSAA
Statistics
© Sioux Embedded Systems | Confidential | 18

19. Crow-AMSAA Failure Plot
Example
© Sioux Embedded Systems | Confidential | 19
Cum. Number of
startups
Cum.
Number
of failed
startups
Individual
testruns
LogLog scale

20. Crow-AMSAA Failure Plot
Example
© Sioux Embedded Systems | Confidential | 20
100 extra startups
10 failures
Best fit

21. Crow-AMSAA MTBF Plot
Example
© Sioux Embedded Systems | Confidential | 21
Monitor trend
Make predictions
Mean
Time
Between
Failures

22.  >100 reliability hits identified
 Which ones would have slipped through other tests?
 Which ones would the customer complain about?
 “Independent” analysis of hits:
 8 would have been in system test, but not earlier
 7 would not have been found, but customer would
complain (and fix would be necessary)
ROI
© Sioux Embedded Systems | Confidential | 22

23.  ROI:
(8 x X1) + (7 x X2) – costs > 0
 Costs (man-hours + material) = 200K Euro
 X1: costs of defect found in system test: 10K Euro
 X2: costs of field defect: 200K Euro
 80K + 1.4M – 200K  1.2M Euro saved
 More money and time became available…
Implementing/executing more tests
More projects/products
ROI
© Sioux Embedded Systems | Confidential | 23

24. © Sioux Embedded Systems | Confidential | 24
Results
 Numerous reliability hits identified + solved
 MTBF measured and predicted
 Startup MTBF increased by factor 7.6
 Acquisition MTBF incr. by factor 18
 More testing hours on systems
 Customer satisfaction
 More projects wanted this approach
 Only 5 system test cycles remaining (was 15)
24

25. © Sioux Embedded Systems | Confidential | 25
Key success factors
 Choose right project at the right time
 Incremental development (early visible
benefit)
 Communication / ROI
 Clear and simple reporting (Crow-AMSAA)
 Hardware interfaces
Low probe effect (not a single false alarm)
Easy ported to different products
25

26. © Sioux Embedded Systems | Confidential | 26
Questions
26
 This case study is described in detail:
Dorothy Graham & Mark Fewster
Experiences of Test Automation
Case studies of software test automation
ISBN 0321754069

27. www.sioux.eu
bryan.bakker@sioux.eu
+31 (0)40 26 77 100