3. Page
3
Validas AG
Motivation for Tool Qualification
‣Development tools can have errors that
- Cause errors in the product
- Hide errors in the product
‣Both has to be avoided
‣Safety standards require to protect customer from them
- IEC 65108
- ISO 26262
- DO178 B/C
‣Product verification is required
‣Tool confidence is required
6. Page
6
Validas AG
ISO 26262 Requirements on Tools
Part 8, Chapter 11: Tool Qualification
‣ Analyze all used tools (the complete “Tool Chain”)
- Use cases
- Inputs/outputs
‣ Determine impact on safety of the product
- TI1: No impact => Tool Confidence Level (TCL 1)
- TI2: Impact: For all potential errors determine tool detection (TD) probability
in the applied process
• TD1: High => tool has TCL 1
• TD2: Medium => tool has TCL 2
• TD3: other => tool has TCL 3
‣ For a given ASIL and TCL select the
qualification methods: all “++” or an
equivalent combination
‣ Make a “Confirmation Review” of
- TCL classification
- Qualification methods of the tools
7. Page
7
Validas AG
Confidence from Use
Sufficient and adequate data for the use of the tool with
‣Tool version and configuration
‣Comparable use cases
‣Systematic error recording
‣details of the period of use and relevant data on its use
‣the safeguards, avoidance measures or work-arounds for the
known malfunctions, or detection measures for a corresponding
erroneous output, if applicable
The increased confidence from use argument shall only be valid for
the considered version of the software tool
In Eclipe there is a Usage
Data Collector that uploads
usage data
But where is the download?
And where are the
malfunctions and safeguards?
8. Page
8
Validas AG
Evaluation of the Development
Process
The development process applied for the development
of the software tool shall comply with an appropriate
standard (?!)
‣NOTE For open source developments some of the
standards used by those communities can also be
appropriate
‣This assessment covers the development of an
adequate and relevant subset of the features of the
software tool (Automotive SPICE, CMMI, ISO 15504,
etc.)
Where are the details, like tests?
9. Page
9
Validas AG
Validation
the validation measures shall demonstrate that the software tool
fulfils its specified requirements
‣Tests for functional and non-functional aspects
‣the malfunctions and their corresponding erroneous outputs of the
software tool occurring during validation shall be analysed together
with information on their possible consequences and with measures
to avoid or detect them
‣the reaction of the software tool to anomalous operating conditions
shall be examined
‣Validation suites can be build
11. Page
11
Validas AG
Tool Chain Analysis
‣Validas developed a method to determine the TCL automatically
‣Based on a simple but formal tool model with
- Tools, use cases, artefacts
- Data flow, control flow
‣Enriched by specification of
- Errors
- Detection and prevention
- Probabilities
‣Tool: Tool Chain Analyzer
13. Page
13
Validas AG
Results of a Simple Example
All tools have TCL 3
(unchecked errors)
Error Flow
Control Flow
Data Flow
Artefacts
Process
Tool / Use Case
make
dcc lcc
15. Page
15
Validas AG
Tool Qualification Lessons
‣New standards require to analyze all tools in the process for
“potential errors that affect the safety”
‣One tool can have different TCLs in different processes
‣Reduction of TCL in the process causes effort
‣A high TCL saves the effort for detecting the tool errors in development
‣Required information for TCL determination
- Application (Product development) process (from the user)
- Uses cases of tools (from user/supplier)
- Potential errors (from supplier)
- Error detection and prevention methods (from supplier)
‣Tool Chain Analysis automatically determines the TCL
18. Page
18
Validas AG
Example: Tool Chain Analyzer
‣ RCP Application
‣ Based on an ecore model
with EMF generator
‣ Plugin architecture
‣ Based on ISO 26262 and formal semantics
‣ Use cases
- TCL determination
- Generation of documentation / explanations
- Generation of graphical views
‣ Saftey Critical Errors:
- Wrong Classification (TCL)
‣ Not critical:
- Exceptions, Crashes, Dialogs, Persistency,..
19. Page
19
Validas AG
Bundles and Dependencies
‣Base model (Tools): determined
from the OSGI structures
‣TODO: potential errors and
possible checks
‣TCL has method to analyze it‘s
own dependecies and generates
the base models
‣Could be generated from every
eclipse tool architecture
‣Basis for error analysis ?
20. Page
20
Validas AG
TCA Classification Information
‣ The TCA provides the following use cases
- Textual export with potential errors
• Wrong TCL
❖ Ignoring reachable checks
❖ Using unreachable checks
• Wrong Conformance check of ASIL / Qualification
- Graphical export (for debugging) with irrelevant errors
- Determination of TCL within the tree view (for development) with irrelevant errors
‣ If textual export is reviewed (against the above errors) the TCA has TCL 1
‣ If the export is not reviewed it would have TCL 3
- We could build a validation suite for TCL 3 (ASIL D) with
• test automatization
• our tests models
• comparing the TCA results with its formal semantics and
• a coverage measurement (EMMA)
‣ Since „confirmation review“ is required in ISO 26262 the TCA has TCL 1
22. Page
22
Validas AG
Summary
‣ ISO 26262 requires to check all used tools for confidence
‣ Tool Confidence Level (TCL) depends on the application process
‣ Tool User
- has to classify tools
- can restrict to safety relevant functions
- check the result of tools (manually or by redundancy)
‣ Tool provider/developer (of each plugin) should
- provide information on use cases and tools
- provide information on potential errors and checks
- can NOT restrict to safety relevant functions
- provide help for qualification of tools
• Usage information on versions, configuration
• Development process
• Test cases / code coverage
‣ Tool Chain Analyzer has TCL 1, but requires manual review
‣ Validation with Coverage Measurement can reduce reviews
