A presentation on a vision of innovative software testing paradigms for embedded systems such as automotive

1. Tomorrow’s Software Testing
for Embedded Systems
25th July 2019 (Tue)
Yasuharu NISHI
Department of Informatics
Graduate School of Informatics and Engineering
The University of Electro-Communications
@YasuharuNishi / Yasuharu.Nishi@uec.ac.jp
Also Read: https://www.slideshare.net/YasuharuNishi/line-developer-meetup-in-tokyo-39-presentation-modified

2. Profile
• Assistant professor:
– The University of Electro-Communications, Tokyo, Japan
• President:
– Association of Software Test Engineering, Japan - Nonprofit organization (ASTER)
• President:
– Japan Software Testing Qualifications Board (JSTQB)
• National delegate:
– ISO/IEC JTC1/SC7/WG26 Software testing (ISO/IEC(/IEEE) 29119, 33063, 20246)
• Founder:
– Japan Symposium on Software Testing (JaSST)
• Founder:
– Testing Engineers’ Forum (TEF: Japanese community on software testing)
• Judgement Panel Chair / member:
– Test Design Contest Japan, Test Design Competition Malaysia (TDC)
• Vice chair:
– Software Quality Committee of JUSE (SQiP)
• Vice chair:
– Society of Embedded Software Skill Acquisition for Managers and Engineers (SESSAME)
• Steering Committee Chair
– QA4AI Consortium
• Advisor:
– Software Testing Automation Research group (STAR)
© NISHI, Yasuharup.2

3. Today’s Software Testing
for Embedded Systems

4. © NISHI, Yasuharup.4
QA/Test Architecture
Test Viewpoint
TDD+NarrowCI
E2E Test Automation+KDT+MBT
Exp. Based Shift Left / Frontloading
Balanced Cloud-ization
Full MBD-ization and Agilitization
Full Model-based Testing and CI
AI supported Shift Left / Frontloading
Shift Right
Huge Technical Debt
No Concept of Technical Logistics
Bureaucratic Quality Management
Passive Engineers
Lack of Responsibility in Management
Product Variability
Mission Criticality
No Big Picture of Development Concepts
Innovator's Dilemma / Hesitating Investment
Integration with Serious Hardware Development
Current
Background
Innovative
Technologies
Management
Considerations
Development
Problems
Current
Technologies

5. © NISHI, Yasuharup.5
Volatility
Uncertainty
Complexity
Ambiguity
Scalability
&
DX
QA/Test Architecture
Test Viewpoint
TDD+NarrowCI
E2E Test Automation+KDT+MBT
Exp. Based Shift Left / Frontloading
Balanced Cloud-ization
Full MBD-ization and Agilitization
Full Model-based Testing and CI
AI supported Shift Left / Frontloading
Shift Right
Huge Technical Debt
No Concept of Technical Logistics
Bureaucratic Quality Management
Passive Engineers
Lack of Responsibility in Management
Product Variability
Mission Criticality
No Big Picture of Development Concepts
Innovator's Dilemma / Hesitating Investment
Integration with Serious Hardware Development

6. p.6
Tomorrow’s Software Testing
for Embedded Systems

7. Tomorrow’s Software Testing in Embedded Systems
Current Technologies
• QA/Test Architecture
• Test Viewpoint
• TDD+NarrowCI
• E2E Test Automation
+KDT+MBT
• Exp. Based
Shift Left / Frontloading
Innovative Technologies
• Balanced Cloud-ization
• Full MBD-ization and
Agilitization
• Full Model-based Testing
and CI
• AI supported
Shift Left / Frontloading
• Shift Right
© NISHI, Yasuharu
OTAOSS/COTS/ML

8. Innovation (1) – Cloud-ization and De-centralization
© NISHI, Yasuharup.8
ECU Virtualization Cloud-ization
and De-centralization
Current
Unit A Unit B Unit C Cloud
Platform
Application
Dev. Team

9. Innovation (2) – Full MBD-ization and Agilitization
© NISHI, Yasuharup.9
Edge-Cloud
Balance Optimization
Full MBD-ization Agilitization
and Asyncronization

10. Innovation (3) – Full Model-based Testing and Full CI
© NISHI, Yasuharup.10
Manual Testing
Full CI
Full Model-based Testing

11. Current Technologies (1) – QA Architecture Design
© NISHI, Yasuharup.11
Fragmented
Well-organized
“What quality
as a whole
are we assuring?”

12. Current Technologies (1) – QA Architecture Design
© NISHI, Yasuharup.12
Fragmented
Well-organized
“What quality
as a whole
are we assuring?”

13. Current Technologies (1) – Test Architecture Design
© NISHI, Yasuharup.13
Test Container
Test Viewpoint
Test architecture design in Daimler
Systematic Test Platform Selection
- Reducing Costs for Testing
Software-based Automotive E/E Systems
C. Schwarzl and J. Herrmann in ICST 2018 Industry session
・
Test Viewpoint
Test Container

14. Current Technologies (2) – Test Viewpoint
© NISHI, Yasuharup.14
Kind of OS
OS
Platform
Environment
- Win10
- Win8
- Win7
Test Values
Bottom
viewpoint
= Parameter
Viewpoint
Viewpoint
Viewpoint
Example of
a viewpoint diagram
on a mindmap format

15. Current Technologies (3) – TDD+Narrow CI
© NISHI, Yasuharup.15
“Test Yourself” by Takuto Wada @ JaSST Hokkaido 2014
https://cloud.google.com/solutions/continuous-integration/

16. Current Technologies (4) – E2E Test Automation+KDT+MDT
© NISHI, Yasuharup.16
ISO/IEC/IEEE 29119-5:2016
Quality Commander
https://www.jnovel.co.jp/service/qc/product/
“It is great that we automate our tests, but why are they so bad?”
by Jeff Offutt @ ICST 2018

17. Current Technologies (5) – Experienced-based Shift Left / Frontloading
© NISHI, Yasuharup.17
Reusable
Development
RecommendationDevelopment
Data
Non-reusable
Development
Data
Development Data
Experience-based
Frontloading
No Frontloading

18. Innovation (4) – AI/ML-supported Shift-Left / Frontloading
© NISHI, Yasuharup.18
Digitalized
Development Data
Development
Recommendation
by AI/ML
AI/ML supported Frontloading
Development
Data
Non-reusable
Development
Data
No Frontloading

19. Innovation (5) – Shift Right
© NISHI, Yasuharup.19
Phase-gate
(but actually unfinished)
Testing
Full CI Shift Right
(Test after Release)
OTAOSS/COTS/ML

20. Tomorrow’s Software Testing in Embedded Systems
Current Technologies
• QA/Test Architecture
• Test Viewpoint
• TDD+NarrowCI
• E2E Test Automation
+KDT+MBT
• Exp. Based
Shift Left / Frontloading
Innovative Technologies
• Balanced Cloud-ization
• Full MBD-ization and
Agilitization
• Full Model-based Testing
and CI
• AI supported
Shift Left / Frontloading
• Shift Right
© NISHI, Yasuharu
OTAOSS/COTS/ML

21. p.21
Tomorrow’s Software Testing
for Embedded Systems