1. Machine Learning-based Test
Selection for Simulation-based
Testing of Self-driving Cars Software
Journal-First Track
Empirical Software Engineering 28, 71 (2023)
Christian Birchler, Sajad Khatiri, Bill Bosshard, Alessio Gambi, Sebastiano Panichella

2. Motivation
392 incidents with ADAS vehicles, NHTSA 2022.
ASE'23 Journal-First 2
14.09.23

3. SDC Testing in Simulation
ASE'23 Journal-First 3
14.09.23

4. SDC-Scissor Approach
ASE'23 Journal-First 4
14.09.23
Test Generation
Test Execution
Labeled Dataset
Feature Extraction
ML Assessment
Test Generation
Feature Extraction
Test Outcome
Prediction

5. SDC Test Definition
ASE'23 Journal-First 5
14.09.23

6. Dataset Generation
ASE'23 Journal-First 6
14.09.23

7. Generic ML-based Workflow
Test Outcome Prediction
Road Features
ASE'23 Journal-First 7
14.09.23

8. Road Features
Direct Distance
Road Length
# Left/Right Turns
Turn Angles
ASE'23 Journal-First 8
14.09.23

9. Evaluation
RQ1: To what extent is it possible to identify safe and unsafe SDC test cases before executing them?
RQ2: Does SDC-Scissor improve the cost-effectiveness of simulation-based testing of SDCs?
RQ3: What is the actual upper bound on the precision and recall of ML techniques in identifying SDC
safe and unsafe test cases when using static SDC features?
ASE'23 Journal-First 9
14.09.23

10. RQ1: To what extent is it possible to identify safe and unsafe SDC test cases before executing them?
J48 Decision Tree
Naïve Bayes
Logistic
Random Forest
ASE'23 Journal-First 10
14.09.23

11. RQ2: Does SDC-Scissor improve the cost-effectiveness of simulation-based testing of SDCs?
# Test cases is fixed
Select N=10 test cases
ASE'23 Journal-First 11
14.09.23

12. RQ2: Does SDC-Scissor improve the cost-effectiveness of simulation-based testing of SDCs?
# Test cases selected until
N=10 failing test cases are
identified
SDC-Scissor spends ca. 50%
less time on executing passing
tests!
ASE'23 Journal-First 12
14.09.23

13. RQ3: What is the actual upper bound on the precision and recall of ML techniques in identifying SDC
safe and unsafe test cases when using static SDC features?
Improving the ML models
Hyperparameter optimization
Grid Search
ASE'23 Journal-First 13
14.09.23

14. Practical Relevance
ASE'23 Journal-First 14
14.09.23

15. COSMOS: DevOps for Complex Cyber-physical Systems
Sebastiano Panichella Sajad Khatiri
Christian Birchler
https://www.cosmos-devops.org/ https://twitter.com/COSMOS_DEVOPS
ASE'23 Journal-First 15
14.09.23

16. CAN Bus Use Case
Lighting
Brakes
Transmission
Engine
ASE'23 Journal-First 16
14.09.23

17. CAN Bus Use Case
CAN Bus
Lighting
Brakes
Transmission
Engine
Brakes
ASE'23 Journal-First 17
14.09.23

18. Problem
CAN Bus
Our test cases do not reflect
real world scenarios!
ASE'23 Journal-First 18
14.09.23

19. Our Intuition
CAN Bus
I simulate real world
scenarios!
ASE'23 Journal-First 19
14.09.23

20. System Overview
ASE'23 Journal-First 20
14.09.23

21. Use Case Scenario
Windows Server
Simulator
SDC-Scissor
CAN Driver
ECU (Raspberry Pi)
CAN Driver
Filter
Cloud App
CAN Bus
ASE'23 Journal-First 21
14.09.23

22. SDC-Scissor selects relevant test cases
Summary
SDC-Scissor uses ML models and road features
SDC-Scissor has a practical relevance for AICAS
ASE'23 Journal-First 22
14.09.23

23. Q&A
https://github.com/christianbirchler-org/sdc-scissor
ASE'23 Journal-First 23
14.09.23