At the EuroSTAR conference 2016 in Stockholm I presented about the testing of artificial intelligence and machine learning. But also about testing using intelligent machines.

1. Testing Machine Learning,
Learning Machine Testing
Rik Marselis, Sogeti Nederland
rik.marselis@sogeti.com

2. 1980
2007 2012 20122008
Workgroup
-member
20142009
Rik Marselis
@rikmarselis

3. Who uses a robot today?

4. You will use multiple within just a few years!!

5. And please realise: these are robots too!
Chatbot Self-driving mini-bus

6. WePod: self-driving bus
This minibus is
allowed on the
road now in
Wageningen (the
Netherlands) and
was tested by my
Sogeti-colleagues
Dragan, Hugo and
Remco.

7. Robots, ok, but in less than 10 years????
Well;
Did you understand this
trafficsign 10 years ago?
That’s how fast technology
progresses!!

8. What is Robotics?
• Internet of Things (IoT)
• Big data analytics
• Cloud technology
• Machine learning
• Artificial Intelligence
Robotics
Definition: A robot is a machine that gathers information about its environment by input of sensors and based on this input changes
its behavior. Combined with machine learning and artificial intelligence the robot’s reactions over time get more and more adequate.
The use of Internet of Things, Bid Data Analytics and Cloud technology make a robot versatile. Robots may appear in a physical shape
(e.g. a tin man) or non-physical (e.g. chatbot).

9. Traditional: Robots for dull & dangerous work

10. New possibilities, just 2 examples
Exo-skeleton
Healthcare  disabled people
Transport  drone delivery

11. New risks
The bull doesn’t like
the manure robot and
constantly pushes it off
its track.
(to much frustration of the
farmer who needs to come
to rescue the robot ;-)

12. New angles of quality, aimed at robotics
Mechanical
Electrical
Information
Processing
Machine
Intelligence
Business
Impact
Social Impact
e.g. Ethics
e.g. Embodiment
e.g. Empathy

13. Part 1:
Testing machine learning

14. Machine Learning in the 80’s

15. Machine learning today: demonstrating

16. Machine learning today: demonstrating

17. Learning by creating rules
How can we find the
rules that self-driving
cars must use?
Ask you!
moralmachine.mit.edu

18. Some
results

19. Machine learning or machine training?
• A learning machine consists of a complex network
of algorithms
• The learning machine has very many “dials and
switches”
• Making sure the machine learns the right things is
just like ordinary training (well, in a different way…)

20. Testing Machine Intelligence
• Human: “Dear robot please get me a softdrink”
• Robot: “Dear human here is your cola”
• Is this the correct output?
• No!
In Sweden I want:

21. Testing Machine Intelligence
• Human: “Dear robot please get me a softdrink”
• Robot: “Dear human it’s Saturday
evening now, so here is your beer”
• Is this the correct output?
• After studying your behavior,
the robot knows your patterns

22. Testing traditional IT vs. machine intelligence
Traditional (example):
• 1 + 1 = ?
• There’s only one correct output
With Machine Intelligence (example):
• Prepare a meal
• When is the meal “good enough”?
• Is it always the same meal?
2

23. Two sides of testing machine intelligence
• Testing the output
This we know from traditional testing,
but it’s hard to predict outputs, the
result may be different every time.
• Testing the input
If the machine gets the right input
it will learn and do the right things.

24. To control the results of AI:
control the inputs…

25. Helpful chatbots?
• There’s a very thin line between a robot that is not helpful,
a robot that is helpful and a robot that’s too helpful

26. Helpful chatbots?
The uncanny valley:

27. What is the role of the tester?
• The tester must see whether the intelligent
machine will maintain the intended behavior in live
operation
• In a way the tester is the psychologist of the
intelligent machine
• A tester must ask even more questions than
before !!
Note: I picked up the idea of the tester being the psychologist of robots from James Bach.

28. It’s no longer about right or wrong
• Testing intelligent machines is not an easy check
whether the result is right or wrong
• It becomes an appraisal of a specific response in a
certain situation with a certain amount of
knowledge for a certain audience
• If these parameters change the right behavior will
also change

29. After finishing the test…
• The machine continues to learn and adapt…
• What about confidence?
• The test must result in the confidence that the
machine will learn and adapt in the right way…

30. Part 2:
Learning machine testing

31. Using a robot for dull or dangerous testing
• Mechanical tests  use robot-arm

32. Using a robot for intelligent testing
• Functional tests  use machine intelligence to
– create test cases and
– execute test cases
– and (important!!) to judge the results

33. Using a robot for testing physical objects

34. Using a robot for intelligent testing
• Non-functional tests  use machine intelligence
For example intelligent performance testing
(not just simple transactions but a normal variety)

35. People and Robots testing together
• Teamwork of person & robot
• Each does what it is best at
– Robot  repetitive work
– Human  creative work

36. Machine learning: unsupervised & supervised
Deep learning:
Robot learns at various
layers. Lower layers
are unsupervised.
Highest layer(s) is (are)
supervised.

37. Machine Intelligence generates test data
• Based on Big Data Analytics of live data
• This is so-called “weak intelligence”
The machine is only intelligent in a limited field

38. Machine Intelligence generates test cases
• Evolutionary algorithms
• Brute force testing
The machine is able to generate and execute
many many more test cases than a human tester

39. Conclusion

40. Testing machine learning
• Testing inputs is very important because the output
is not predictable
• The objective of testing is to gain confidence that
the machine will learn and develop itself in a right
way

41. Learning machine testing
• Test automation will go to the next level
• Use machine intelligence (e.g. big data analytics)
to create your optimal test data
• Use machine intelligence to create many more test
cases, execute them and analyze results

42. Conclusion…
As a tester better prepare yourself for:
• Testing of Robots
and
• Testing with Robots

43. Thanks for attending my talk.
Any questions ???????????
Rik Marselis, Sogeti Nederland
rik.marselis@sogeti.com