DEV meet-up UiPath Document Understanding May 7 2024 Amsterdam
Tool Qualification v12.02
1. Tool Qualification
Minimize the risk
of systematic faults in the developed product
due to malfunctions of the software tool
(introduce or fail to detect errors)
Debug and Test Tools
2. ISO 26262
Part 8 Supported Processes, Chapter 11
Safety-Lifecycle Process
― Big portions of this procoss is covered by software tools today
This works only if
― The risk of systematic fault in the developed product due to
malfunctions of the software tool (introduce or fail to detect errors) is
minimized
― The software tool works accurate and correct within such a process
Detailed and up front analysis / risk assessment of the software tools used
within a safety project
Debug and Test Tools
3. Software Tools
Software Development Tools: May introduce an error in the final
product
Software Verification Tools: May fail to detect an error in the
final product
Other Software Tools
― Depending on the size of a company we are talking of a big number of
tools (including in-house tools, Excel, …)
Debug and Test Tools
4. Part 1 – Overview
The Blue Box and ISO26262 Use Cases
Debug and Test Tools
5. Reference V ISO 26262-6
ISO 26262-5, 10
Hardware
Debug and Test Tools
6. iSYSTEM Tools within a Safety Project I
Software Development (ISO 26262-6, 9)
― Not so critical
― Assembler (interrupt handler, time-critical algorithms, …)
Software Test (ISO 26262-6, 9)
― Unit Testing
― Integration Testing
Hardware-Software Integration and Testing (ISO 26262-5, 10/-4,8)
System Integration Testing (ISO 26262-4,8)
ISO 26262-6 mentions: „…The test environment … shall correspond as
closely as possible to the target environment …“
Debug and Test Tools
7. iSYSTEM Tools within a Safety Project II
- Unit Testing
isystem.connect
isystem.test
Debug and Test Tools
8. iSYSTEM Tools within a Safety Project II
- Unit Testing
isystem.connect
isystem.test
+
Trace &
Object Code
Level Coverage
Debug and Test Tools
9. iSYSTEM Tools within a Safety Project II
- Unit Testing
isystem.connect
isystem.test
Debug and Test Tools
10. iSYSTEM Tools within a Safety Project II
- Hardware-Software Integration Testing
isystem.connect
isystem.test
+ Trace & Profiler
isystem.connect
isystem.test
+ Trace &
I/O Module
Debug and Test Tools
11. Part 2 – How to get confidence
in the use of software tools
Detailed and up front analysis / risk assessment of the software tools
used within a safety project
Debug and Test Tools
13. e?
Confidence in the use of e nc
id
software tools ISO 26262-8-11
on
f
c
and
sk
How high is the risk that the malfunction of a tool leads to an error in the
ri
ss
final product or errors are not detected (Tool Impact, TI)?
se
as
ou
How much do you trust your own methods and procedures (your
y
processes) in order to uncover suchhmalfunction of a tool
(Tool Error Detection (TD)? hic
w
on
high se average low
ba Trust in process
minimal risk
ti on
high risk a
o rm
f
in
e
th
Beside the tool itself also the process plays a major role in getting
is
confidence
t
a
h TCL2, TCL3: Tool Qualification applies
W
Debug and Test Tools
14. How iSYSTEM helps you
getting/creating confidence?
Process
― SPICE, CMMI, … iSYSTEM: ISO9001:2008
― Output: Process documentation, external audits
Transparency es
ur
― Release and test process (http://www.isystem.com/downloads/winidea/release-policy)
eas new features), item (bugs, changes, new
― Output: Test documentation, release notes (major
features, …) tracking lists l M
o na
Documentation a ti
is be used in a safety project or to be qualified)
― Description of features/functions (to
an
rg
― Explanation how these features/functions should (not) be used
O
― Manuals: User Manual, Hardware Reference Manual, Getting Started Guides, PCB Desing
Guidelines, ...
― System Requirements in general or specific to a micro controller
― Description of well-known workarounds dedicated to a specific micro controller
― …
Pre-Qualification Environment: fitIDEA
― Regression test suite, reference hardware, reference application, reference test cases
Debug and Test Tools
15. Result of Step 1
Determine a „Tool Confidence Level“ could be
l?
minimal risk, high trust = t oo
e
no tool qualificationr needed
w
a
ft
so
a
y
lif
or you see a high risk and an average trust in
your processes andqua a tool qualification would
to apply
w
Ho
high average low
Trust in process
minimal risk
high risk
Debug and Test Tools
16. ISO 26262-8-11 Tool Qualification Methods
TCL2 A B C D A-D: Measure of the
Methods safety relevance of a
1a Increased confidence from use ++ ++ ++ + malfunction
(Automotive Safety
1b Evaluation of the tool dev process ++ ++ ++ +
Integrity Level (ASIL))
1c Validation of the software tool + + + ++
1d Dev. in accordance with a safety standard + + + ++
TCL3 A B C D
1a Increased confidence from use ++ ++ + +
1b Evaluation of the tool dev process ++ ++ + +
1c Validation of the software tool + + ++ ++
1d Dev. in accordance with a safety standard + + ++ ++
+ recommended
++ highly recommended
Qualifying a software tool means checking/discussing all 4 methods
Debug and Test Tools
17. What are we talking about?
The company iSYSTEM
Development and test tools
Standard debugging and IDE functions (memory read, write,
step, memory dump, download, flash programming, etc.)
Advanced debugging with trace and profiling (especially time
measurements)
Software test with code coverage and unit test
Debug and Test Tools
18. TCL2 A B C D
1a Increased confidence from use ++ ++ ++ +
1b Evaluation of the tool dev process ++ ++ ++ +
1c Validation of the software tool + + + ++
1d Dev. in accordance with a safety standard + + + ++
TCL3 A B C D
1a Increased confidence from use ++ ++ + +
1b Evaluation of the tool dev process ++ ++ + +
1c Validation of the software tool + + ++ ++
1d Dev. in accordance with a safety standard + + ++ ++
Debug and Test Tools
19. Increased Confidence From Use?
How is a software tool used today?
― Dedicated version and configuration
ll ?
ta
― For one or more specific micro controllers
a
― In conjunction with other software tools (e.g., compilers)
hod
et
New project: m
s
― The tool setup may definitely differ fromi the previous project
th
― Very new controllers will be used er
id
ns
co
Dynamic release policy of the software tool manufacturer
e
w
― Official and regression tested release
d
ul
ho
― Hotfixes (customer wishes)
S
― Fairly good tested software versions
Debug and Test Tools
20. YES! – Because …
In the market since 1986
Thousends of users, word-wide
• Similar projects successfully done in the past
Great cooperation between you and iSYSTEM
― Transparency ISO 262
62
― Openness
― Pragmatism
Debug and Test Tools
21. How iSYSTEM helps you
getting/creating confidence?
Process
― SPICE, CMMI, … iSYSTEM: ISO9001:2008
!
EM
― Output: Process documentation, external audits
ST
Transparency
i SY
=
― Release and test process (http://www.isystem.com/downloads/winidea/release-policy)
― Output: Test documentation, release notes se
(major new features), item (bugs, changes, new
features, …) tracking lists u
in
ce
Documentation en
fid
― Description of features/functions (to be used in a safety project or to be qualified)
on
― Explanation how these features/functions should (not) be used
c
d
― Manuals: User Manual, Hardware Reference Manual, Getting Started Guides, PCB Desing
se
ea in general or specific to a micro controller
Guidelines, ...
cr
― System Requirements
n
― DescriptionIof well-known workarounds dedicated to a specific micro controller
― …
Pre-Qualification Environment: fitIDEA
― Regression test suite, reference hardware, reference application, reference test cases
Debug and Test Tools
22. TCL2 A B C D
1a Increased confidence from use ++ ++ ++ +
1b Evaluation of the tool dev process ++ ++ ++ +
1c Validation of the software tool + + + ++
1d Dev. in accordance with a safety standard + + + ++
TCL3 A B C D
1a Increased confidence from use ++ ++ + +
1b Evaluation of the tool dev process ++ ++ + +
1c Validation of the software tool + + ++ ++
1d Dev. in accordance with a safety standard + + ++ ++
Debug and Test Tools
23. Evaluation of the tool dev process
Most of the software tool manufacturers do have defined and documented
their processes (and lives those, hopefully), ISO9001 or similar
We guess that non of them have a real process model in place (such as
CMMI, SPICE, …)
Audit/Assessment of these processes is possible (quality manual, process
descriptions, external audit, …)
Zertificates?!
It may be a big challenge to evaluate a dev process, again and again
Too many software tools are already in use, also Excel and so forth
Debug and Test Tools
24. How iSYSTEM helps you
getting/creating confidence?!
s
se
Process es
― SPICE, CMMI, … iSYSTEM: ISO9001:2008 oc
pr
st
― Output: Process documentation, external audits
te
Transparency and
ev
― Release and test process (http://www.isystem.com/downloads/winidea/release-policy)
― Output: Test documentation, release notes d
s (major new features), item (bugs, changes, new
features, …) tracking lists M
TE
YS
Documentation iS
e
at
― Description of features/functions (to be used in a safety project or to be qualified)
u
― Explanation how these lfeatures/functions should (not) be used
a
ev
― Manuals: User Manual, Hardware Reference Manual, Getting Started Guides, PCB Desing
Guidelines, ... to
e
bl
― System Requirements in general or specific to a micro controller
si
os
― Description of well-known workarounds dedicated to a specific micro controller
― … p
t is
I
Pre-Qualification Environment: fitIDEA
― Regression test suite, reference hardware, reference application, reference test cases
Debug and Test Tools
25. TCL2 A B C D
1a Increased confidence from use ++ ++ ++ +
1b Evaluation of the tool dev process ++ ++ ++ +
1c Validation of the software tool + + + ++
1d Dev. in accordance with a safety standard + + + ++
TCL3 A B C D
1a Increased confidence from use ++ ++ + +
1b Evaluation of the tool dev process ++ ++ + +
1c Validation of the software tool + + ++ ++
1d Dev. in accordance with a safety standard + + ++ ++
Debug and Test Tools
26. Validation of the software tool
It is also the responsibility of the tool manufacturer to provide means and
methods to enable a customer to perform a validation
Ideal: Availability of the manufacturer’s test tool suite for a customer.
Documentation, Test suite with reference hardware, reference applications
and appropriate test cases.
isystem.connect for Python wrapper
isystem.connect test
testIDEA
isystem.connect
iSYSTEM Development Environment & Debugger winIDEA
Reference Target
Reference Application
Debug and Test Tools
27. How iSYSTEM helps you
getting/creating confidence?
e!
Process
― SPICE, CMMI, … iSYSTEM: ISO9001:2008
os
― Output: Process documentation, external audits rp
pu
ct
Transparency
exa
― Release and test process (http://www.isystem.com/downloads/winidea/release-policy)
s
hi
― Output: Test documentation, release notes t(major new features), item (bugs, changes, new
features, …) tracking lists
f or
A
DE
itI (to be used in a safety project or to be qualified)
Documentation
f
― Description of features/functions
s
de
― Explanation how these features/functions should (not) be used
vi
― Manuals: User Manual, Hardware Reference Manual, Getting Started Guides, PCB Desing
o
Guidelines, ... pr
EM
― System Requirements in general or specific to a micro controller
― Description T well-known workarounds dedicated to a specific micro controller
Sof
― …
i SY
Pre-Qualification Environment: fitIDEA
― Regression test suite, reference hardware, reference application, reference test cases
Debug and Test Tools
28. fitIDEA - Tool Pre-Qualification Environment
Test GUI
Test Cases
Consulting &
Test Reports
Integration
Safety Manual &
Services
Documentation
isystem.connect for Python wrapper
isystem.connect test
testIDEA
isystem.connect
iSYSTEM Development Environment & Debugger winIDEA
Reference Target
Reference Application
Debug and Test Tools
29. Tool Pre-Qualification Environment
• If a specific function of an iSYSTEM tool has to get verified in the
customer’s use case context
• Consists of reference hardware plus test cases to verify several functions
of an on-chip debug and trace tool such as:
― Standard debugging and IDE functions, e.g. memory read, write, step, memory
dump, download, flash programming, etc.
― Advanced debugging with trace and profiling (especially time measurements)
― Software test with code coverage and unit test
―…
• Is an extract of iSYSTEM’s internally used regression test tool suite
• May be ported/adapted to a customer’s target system
Debug and Test Tools
30. TCL2 A B C D
1a Increased confidence from use ++ ++ ++ +
1b Evaluation of the tool dev process ++ ++ ++ +
1c Validation of the software tool + + + ++
1d Dev. in accordance with a safety standard + + + ++
TCL3 A B C D
1a Increased confidence from use ++ ++ + +
1b Evaluation of the tool dev process ++ ++ + +
1c Validation of the software tool + + ++ ++
1d Dev. in accordance with a safety standard + + ++ ++
Debug and Test Tools
31. Development in accordance with a
saftey standard
View and adapt
recommendations of
such standards
Transfer know-how to
software developers
(how others have to do
it)
Debug and Test Tools
32. How iSYSTEM helps you
getting/creating confidence?
Process
― SPICE, CMMI, … iSYSTEM: ISO9001:2008
― Output: Process documentation, external audits
Transparency n!
ar
le
― Release and test process (http://www.isystem.com/downloads/winidea/release-policy)
to
― Output: Test documentation, release notes (major new features), item (bugs, changes,
new features, …) tracking lists g n
i
i ll
w
Documentation
is
EM
― Description of features/functions (to be used in a safety project or to be qualified)
T
― Explanation how these features/functions should (not) be used
SHardware Reference Manual, Getting Started Guides, PCB
― Manuals: User Manual,Y
iS
Desing Guidelines, ...
― System Requirements in general or specific to a micro controller
― Description of well-known workarounds dedicated to a specific micro controller
― …
Pre-Qualification Environment: fitIDEA
― Regression test suite, reference hardware, reference application, reference test cases
Debug and Test Tools
33. 62
ISO 262
Summary
Detailed and up front analysis / risk assessment
of the software tools used within a safety project
Beside the tool itself also the process plays a
major role in getting confidence
It is also the responsibility of the tool manufacturer to provide means and
methods to enable a customer to perform a validation
Tool manufacturers will improve their development and test processes
Transparency first, qualification second
Debug and Test Tools