Front Loaded Test Approach Verhaert

Integrated hard- & software testing for first-time right development

FIRST-TIME RIGHT
www.mastersininnovation.com

DEVELOPMENT ...
A DIFFERENT TEST APPROACH

Koen Puimege
Koen.puimege@verhaert.com
Frederik Wouters
frederik.wouters@verhaert.com
www.verhaert.com
Commercially confidence – This presentation contains ideas and information which are proprietary of VERHAERT, Masters in Innovation®*, it is given in confidence. You are authorized to
open and view the electronic copy of this document and to print a single copy. Otherwise, the material may not in whole or in part be copied, stored electronically or communicated to third
parties without prior agreement of VERHAERT, Masters in Innovation®*.

* VERHAERT, Masters in Innovation is a registered trade name of Verhaert New Products & Services NV.

Slide 1


Agenda
Introduction

Software & Hardware in the loop

Front loaded test approach of Verhaert

Case study: Proba Satellite

Slide 2


Verhaert mission, strategy and activities
Activities:
Mission:

Innovative product development services Space & Security
•Instrumentation
•Small satellites
Strategy: •HALE UAV’s

Industry
• Act as system developer in key markets
•Marine
• Offer unique knowledge in industrial •Finance & Retail (ICT)

product development •Life Science & Care
•Automotive & Transport
•Materials & Machines

Slide 3


Introduction
Good development practices

Focus on early decisions & on specific design alternatives on a sound
basis

Analysis and specification freeze before initial physical prototyping

Feedback on design trade off and optimisations

Slide 4


Performance
Introduction Requirements
Testing

System Functional
Specifications Testing

Detail Low-level
Practices of testing Specifications Testing

Requirements based testing
D&D

Integrated development & test approach

Front loaded testing

Slide 5


Introduction
When can a large-scale test methodology be cost

effective ?

Cost driven:
Critical development schedule
High burden rate i.e. expensive equipment (crashtesting)
Usability - Man in the loop

Duration

Safety Critical

Slide 6


Introduction
Debugging tools don’t determine what

your bug is, but where it is.

Slide 7


SIL - Software in the loop
Virtual simulation of embedded system on host

Microprocessor
Application

Used for debugging and functional verification
Alow debugging before HW is available
Code validation on simulated hardware

Slide 8


SIL - Advantages
Provide feedback & selection for HW design

Enable testing/simulation of off-nominal behavior (error conditions,
fault recovery)

Concurrent HW & SW development
No HW resource conflict when developing/testing with >1 person

Cross validation of environment (interfaces, payloads) through
behavioral models

Slide 9


SIL - Disadvantages
Disadvantage: Accuracy of model

Need for code validation on real hardware (HIL)

No Possibility to test system in real-time
Verification SW timing and performance

Slide 10


HIL - Hardware in the loop
Highly realistic simulation of equipment in an operational real time

virtual environment.

Replacing plant interaction by RT simulations.

Slide 11


HIL - Advantages
Testing of extreme conditions upto the limits
Performance, Real time behaviour, Environmental

...

Anomaly testing & Isolate deficiencies

Testing with/without subsystems & environment

Reducing cost
Equipment & manpower
Time to market

Slide 12


HIL & Complex Systems
Multi-processor Platform Solar
Batteries

Multi-unit Array
Redundant system
Reaction Power
Payload#1
Many interfaces wheels Control
System
Magneto-
Payload#2
torquers
Board Payload
AOCS i/f
Magneto- Computer Computer
Payload#3
meters

Star
GPS Tracker Payload#4

Slide 13


HIL - Disadvantages
Difficulty to emulate REAL environment (crash, satellite launch, ...)

Doesn’t debug your embedded system,
No breakpoint or pause-on-command
No info of erroneously code
No info on bug location

Slide 14


Front loaded Test Approach

Requirements based testing

Software Validation Facility (SIL)

Integrated Test Environment (Extended HIL)

Slide 15


Requirements Based Testing
Identify Requirements related to eSW

Check verification method (Analysis / Design / Test)

Provide list of tests

Identify which tests to do in which project phase(s) and on
which level
Simulation / Emulation
Unit test / System test

Result is test matrix used for
Test progress
Traceability
«Test plan
must be established from the beginning together with the architectural design»

Slide 16

VALIDATION LEVELS
VHDL TEST BENCH AVIONICS TEST BENCH (SCOS)
0 1 2 3 4 5 6 7

EQM VHDL TEST RESULTS (PRIOR TO EQM BURN)

(test board functions at system level)

parallel execution of selected tests)
(functional system performance by
(test single functions of a board)

(verified by the designer)

ELECTRICAL TESTS

SUBSYSTEM TESTS
TEST-ID SUFFIX

SYSTEM TESTS

SYSTEM TESTS
MATURE CORE

VHDL MODULE

VHDL SYSTEM
CORE DESIGN

PASS/FAIL

PASS/FAIL

PASS/FAIL

PASS/FAIL

PASS/FAIL

PASS/FAIL

PASS/FAIL

PASS/FAIL
TEST-ID

TEST DEFINITION
PROFILE NAME TEST ITEMS
01 TME functional tests registers power-up defaults X PASS X PASS X (*)
TTM-50
register write/readback X PASS X PASS X (*)
02
VC7 generation of idle data / VC0 OBET
03
time source packet insertion X PASS X PASS X PASS
VC0 HKM data integrity and handshaking X PASS X PASS X PASS
04
VC3 VC4 External PacketWire data
05
initegrity of packetized data X PASS X PASS X PASS
VC3 VC4 External PacketWire data
06
initegrity of non-packetized data X PASS X PASS X (*)
VC1 Internal PacketWire data integrity of
07
non-packetized-data X PASS X PASS
VC2 Internal PacketWire data integrity of
08
non-packetized-data X PASS X PASS
VC1 VC2 Internal PacketWire data
09
integrity of non-packetized-data X PASS X PASS X (*)
VC1 VC2-Internal PacketWire data
10
integrity of packetized data X PASS X PASS X PASS X PASS
Time-source-packet X PASS X PASS X (*)
11
TX ringbuffer empty interrupt X PASS X (*)
12
Core reset via global reset register X PASS X PASS X PASS X
13
VC1 VC2 channel reset X PASS X PASS
14
FECW function X PASS X PASS X PASS
15
Reed Solomon function X PASS X PASS X (*)
16
Convolutional function X PASS X PASS X (*)
17
FECW, RS and CONV enabled X PASS X PASS X (*)
18
Mark-NRZ-M, SPLIT-PHASE, PSEUDO-
19
RANDOM functions X PASS X PASS X (*)
VC1 VC2-Internal-PacketWire data
20
integrity of packetized data (PD-flag=0) X PASS X PASS
CLCW-bit-16-17 option X PASS X PASS
21
VC bandwidth allocation (dynamic based) X PASS X PASS X FAIL
22
VC bandwidth allocation (priority based) X PASS X PASS X (*)
23

Slide 17


SVF - Key data
Software in the loop system

TSIM - Leon Sparc simulation model used from Gaisler Research

Behavioral models developed in-house in C

eSW
Written in C
SVF fully transparant to application
Binary code runs on TSIM

Slide 18


Software Validation Facility (SVF)

DUT Computer Model Environmental Model
Stimuli
I/F Behavioral model (C)
Processor simulation
(TSIM)
Response

eSW (C)

Slide 19


TEST APPLICATION
BINAIRE

CODE
IL DEVICE DEVICE DEVICE DEVICE
DRIVER DRIVER DRIVER DRIVER
SOFTWARE
SOFTWARE

IP IP IP IP
PROCESSOR
CORE CORE CORE CORE
PROCESSOR
FPGA
Synthesisable model (real)
HW HW HW HW
Executable code (binary)

Behavioural model (simulation)
MEMORY
BOARD BOARD
BOARD BOARD
No model Required

Slide 20


FUNCTIONAL APPLICATION
BINARY

CODE
DEVICE DEVICE DEVICE DEVICE
SOFTWARE
SOFTWARE

IP IP IP IP
PROCESSOR
CORE CORE CORE CORE
FPGA
HW HW HW HW

Behavioural model (simulation)
MEMORY
BOARD BOARD
BOARD BOARD
No model Required

Slide 21


FUNCTIONAL/TEST APPLICATION
BINARY

CODE
DEVICE DEVICE DEVICE DEVICE
SOFTWARE
SOFTWARE

IP IP IP IP
PROCESSOR
CORE CORE CORE CORE
Hardware
FPGA
HW HW HW HW

Behavioural model (simulatie)
MEMORY
BOARD BOARD
BOARD BOARD
No model Required

Slide 22


Integrated Test Environment (ITE)
DUT computer Test Bench (TB)

Stimuli Behavioral
I/F
model (C)
Leon Sparc
I/F 2x embedded
RTEMS Behavioral
cPCI PIII
I/F model (C)
Response
RT Linux
Debug Behavioral
eSW (C) DCA (C)
I/F model (C)
LAN

Test IF Bridge (TIFB) Monitoring & Control
PC-104 486 Pentium IV PC
RT Linux SuSe Linux

Slide 23


ITE - Differences with HIL systems?
ITE features integrated debugging functionalities

DUT , HIL & DUT – HIL interaction
I.e. Specific test software on DUT

Custom interface cards to validate off–nominal
behaviour;
Baud rate deviation, ...

Space specific HW cards
Custom HW
Custom Drivers for test software

Re-use of SVF SW

Slide 24


ITE – Key data
Functional and performance testing

ITE Components
Simulates HW environment of sub-system or whole
satellite
Same/similar behavioral models can be used through
hardware abstraction
DUT (Proba Satellite)
Debug interface
Real hardware (Leon Sparc,
Provides access to embedded system via Ethernet
I/F, PCB)
RTEMS
Monitoring & Control
Application software
User i/f
Device Control Application
Set of tools (compile, run, debug, test, log, scripting, ...)

Slide 25


ITE – Techniques and Tools
Device Control Application (DCA)

Debug interface

Scripting

Monitoring & Control

Transparency

Independent testing

Slide 26


ITE - Device Control Application (DCA)
Dedicated application

Separate from application SW
Aimed to test and validate all sub-units linked to Board
computer (payloads)
Implemented according to agreed ICD (Interface
Control Document)
Enables black-box testing of sub-units from within ITE
Supports integrated testing of whole system
Inherent cross-validation
Sub-unit made by sub-contractor/other team
DCA i/f made by system integrator

«The DCA facilitates integration testing on system level»

Slide 27


ITE - Debug Interface
Supports debugging and testing of the HW

Debug interface of Board computer allows
deep control of whole system,
not only processor

Two front-end applications used
GDB (open source)
Dedicated

«An adequate debug interface is imperative to test the eSW application»

Slide 28


ITE - Scripting
Automatic/repeated running of tests

Management of ITE

Top-level

Prevents human errors

Same tests must be run in different project phases
(Engineering/Qualification/Flight Model)

Advantage of Tcl scripting language
Easy to read/understand
Interpreted -> flexible (no compilation)

Slide 29


ITE - Monitoring & Control
Visual representation of system state

Change system state

LAN based
No keyboard/display i/f on Board computer
Enables remote testing and debugging
Good support for TCP/IP in most operating systems
Reliable connection

Platform independent GUI (Qt - Trolltech)

Slide 30


ITE - Transparency
Layered SW model

Transparency towards
Application
HW i/f (Hardware abstraction)
POSIX Host platform (x86, Leon Sparc)
Operating system
POSIX compliant OS + Device drivers

Advantages
Hardware
Re-use of SW
Test SW in appropriate environment
Early development/testing of SW

«Transparency leads to more effective eSW development and testing»

Slide 31


ITE - Independent Testing
eSW Developer

Focused on application
Highly involved, dedicated to implementation
Biasing

eSW Tester
Focused on requirements/functionality
Does not know implementation

Advantages
Bugs are found earlier
More bugs are found

«The developer should not validate his own code»

Slide 32


Conclusions
Integrated testing, an essential part of eSW
development

Good testplan, testmethodology & enhanced
testtools will save your time

Simulation enables early testing and provides
valuable input for the development

PROBA 1 mission successful
Launched 22.10.2001
Commissioned in space
Planned to operate for 2 years & still operational

Slide 33


PRoject for On-Board Autonomy
Platform for small satellites

Demonstration of
Operational autonomy
New technologies (batteries, positioning)
Payloads

Create wider path for use of small satellites
Autonomy (low cost of ownership)
Powerful on-board computer (advanced data
processing)
Advanced Attitude & Orbit Control System (AOCS)

Slide 34


PROBA – Mission
Main task

PROBA 1: Observation of the Earth
PROBA 2: Observation of the Sun

Schedule
PROBA 1: launched 22.10.2001
PROBA 2: scheduled for mid 2007
PROBA 3: on the roadmap

Low earth orbit (LEO) – 600km

Slide 35


PROBA – Key data
Micro satellite

120kg, 80cm x 60cm x 60cm
40W to 120W power

Modular & Flexible
Launcher interface
Accommodation of off-the-shelf payloads
Software modularity

Powerful processors: ERC32,LEON SPARC

OS : VxWorks, RTEMS

Slide 36


Slide 37


Verhaert New Products & Services nv
Hogenakkerhoekstraat 21
9150 Kruibeke
Belgium
Tel +32 (0)3 250 19 00
Fax +32 (0)3 254 10 08
www.verhaert.com
info@verhaert.com

Slide 38

Front Loaded Test Approach Verhaert

Recommandé

Recommandé

Contenu connexe

Tendances

Tendances (20)

En vedette

En vedette (19)

Similaire à Front Loaded Test Approach Verhaert

Similaire à Front Loaded Test Approach Verhaert (20)

Plus de Verhaert Masters in Innovation

Plus de Verhaert Masters in Innovation (20)

Dernier

Dernier (20)

Front Loaded Test Approach Verhaert