Model-based Testing: Today And Tomorrow

MODEL-BASED TESTING:
TODAY AND TOMORROW
Robert V. Binder

ETSI Model-Based Testing User Conference
Tallinn, Estonia
September 27, 2012
© 2012, Robert V. Binder. All Rights Reserved

2

Overview
• Is software eating the world?
• Bugs
• Black Swans
• Dragon Kings
• Multi-dimensional Testing
• Challenges

3

“Software is Eating the World”
-Marc Andreessen

Wall Street Journal, Aug 20 2011

4

Internet Trends. Mary Meeker, D10 Conference. May 2012

5


6


7

Embedded and Connected

2015 Connected Device Landscape: The Expanding Computing Continuum.
David Perlmutter, Intel. 2009

8

IT Disruption

Gartner Top Predictions for 2012: Control Slips Away, Daryl Plummer. March 2012

9


10

Is Software really eating the world?
Methods/Tools since Y2K
• 2nd gen IDE
• MBT, MDD
• Aspect OP
• Open Source
• UML 2.0
• Cont. Integration
• Web 2.0
• Agile
• ISO 12027
• LAMP Stack
• Many protocols
• HTML 5
• iOS, Android
• Big Data http://langpop.com/
• Software Def Netwks Relative rank from ~2M data points

11


Engineering Change to Accelerate Product and Systems Development.
Christopher Rommel, VDC Research. August 2012

12

Hardware is eating the world
Everything Computes & Connects Software Methods and Tools

• 10x to 100x • ?X Productivity
• ten years • ?X Quality
• 100x to 1000x • Last ten years?
• twenty years
• Last twenty years?

• Cheaper, better, faster • Cheaper, better, faster
• Disrupt or die • “Choose two”

13

Old bugs, new bugs
• No fundamental or • End of the uni-stack
transformative • Multi-stack, mobile,
improvements in connected
software • Inherently unstable

• Bugs still as likely, • New failure modes
more code

What do you think will happen?

14

Old Bugs
• Incorrect output • Bohr Bugs
• Missing feature  Predictable, repeatable

• Abnormal end • Mandel Bugs
 Obscure, complex trigger
• Missed deadline
• Heisen Bugs
 Disappears if you look

The Jargon File
http://www.catb.org/jargon

18

New Bugs
• Power Law Models
• Black Swans
• Dragon Kings

19

Power Law Models

• Many natural
phenomena

• “Fat Tails”

Dragon-Kings, Black Swans and the Prediction of Crises. Didier Sornette

http://www.youtube.com/watch?v=FlTSbzOvKZI

20

Reliability Growth

The Lognormal Distribution of Software Failure Rates: Application to
Software Reliability Growth Modeling. Robert E. Mullen

21

Black Swans
• Very rare
• Surprise
• Cannot predict
• Power law outliers

22

Software Black Swans
• Total eclipse
• Once every 360 years

• Multi-cycle race
conditions in hard
realtime systems

23

Dragon Kings
• Extreme Outlier
 Beyond fat tail
• Latent Positive Feedback
• “Phase Shift”
 Earthquakes
 Material Fractures
 Epileptic Seizures
 Market Crashes

24

Dragon Kings

Sornett calls these exceptional events dragon
kings “to stress that we deal with a completely
different kind of animal, beyond the normal,
whose extraordinary characteristics [have]
profound significance.”

25

Latent Positive Feedback = Dragon King

26

New Failure Modes:
Amazon Cloud Outage
• Amazon web service Elastic • ELB reaction: try to allocate
Compute Cloud crashes more larger servers.
(power failure)
• Backlog in the “control plane”
• The Elastic Load Balancing results.
(ELB) system frantically tries
to assign workloads to • Demand from customers in
available servers. unaffected availability zones
continues
• As Amazon’s cloud rebooted,
“a large number of ELBs • System swamped and
came up in a state which crashes again.
triggered a bug we hadn’t
seen before”

27

What kind of failure was this?

• Why didn’t test find it?

• What would we have to do to find
these kind of bugs?

29

Here be Dragons
 Latent positive feedback
 External disruptors
 Malicious attack target
 Partial degradation
 Provoked panic
 Low testability
 Multi-stack
 Field-infeasible
 Uncontrollable inputs

30

Looking under the Model-based Lampost?
Logic errors and incorrect
assumptions are inversely
proportional to a path’s
execution probability.

Boris Beizer
• 9 in 10 MBT testers do
functionality only
• Climbs a tree, not the twigs
• Typically no load test

31

Multi-Dimensional Testing
Test suite interleaves three kinds of variation
Behavior Interfaces
• Realistic variation • Drive all interfaces
• Markov event/data • Disrupt resources
• Include
• Fat finger Load
• Abuse cases • Vary input rate wrt time
• Sneak paths
• Panics

Not possible without full automation

32

Interleave Behavior, Load, Interfaces
3500.000
10000000
3000.000
1000000
2500.000
100000

X
2000.000
10000

Events Per Second
1500.000
1000
1000.000
100
500.000
10
0.000
1 -5000 0 5000 10000 15000 20000 25000
Use Case A B C D E F G H I J K L -500.000
Time (seconds)

Simulator

Oracle

X =
Adapter
Splainer
Adapter

Adapter Comparator

Adapter Test Verdict
Run Reports
SUT

33

Hunting the Dragon King
Common Black Dragon
Bugs Swans Kings
Fragile 1 0 0
Massively Manual 2 0 0
Test Driven Dev 3 1 0
Exploratory 3 1 0
Crowdsourced 2 0 0
Functional MBT 5 3 1
Multi-dimensional MBT 5 4 3

• 0 You must be joking • 3 One Sigma
• 1 Write if you get work • 4 Michael Jordan free throw (82%)
• 2 Even a broken clock is right… • 5 Sargent Preston

34

What have we learned?
• Information Technology is eating the world

• Common bug rates will not decline

• More new systems will have dragon kings

• Functional/model coverage MBT will not find
dragon kings

• Multi-dimensional MBT can find dragon kings

• Opportunity for test engineering to lead

35

May you live in interesting times …

Challenges for the MBT Community

Tool Providers MBT Test Engineers
• Start shipping multi- • Learn to lead
dimensional • Start hunting your
solutions dragon kings

36

Thank you !

rvbinder@gmail.com

www.robertvbinder.com

37

Content and Image Sources
First Generation of Re-Imagination Reliability Growth
Android Phone Adoption has ramped even faster R. E. Mullen, “The Lognormal Distribution of Software Failure Rates:
Despite Tremendous Ramp so Far, Smartphone User Adoption… Application to Software Reliability Growth Modeling,” International
Magnitude of Upcoming Change Will be Stunning… Symposium on Software Reliability Engineering, 1998, p 134.
Internet Trends. Mary Meeker, D10 Conference May 2012
http://kpcb.com/insights/2012-internet-trends Black Swans
The Black Swan: the Impact of the Highly Improbable. Nassim
Embedded and Connected Taleb.
2015 Connected Device Landscape: The Expanding http://www.publicdomainpictures.net/view-
Computing Continuum. David Perlmutter, Intel. 2009 image.php?image=8248&picture=black-swan&large=1
http://www.intel.com/pressroom/kits/events/idffall_2009/video.htm
Software Black Swans
IT Disruption http://apod.nasa.gov/apod/ap060404.html
Gartner Top Predictions for 2012: Control Slips Away, Daryl
Plummer. March 2012 Dragon Kings
http://wallpaper4me.com/wallpaper/Dragon-King-of-The-Land/
Engineering Change to Accelerate Product and Systems Latent Positive Feedback = Dragon King
Development. Christopher Rommel, VDC Research. August http://www.everythingselectric.com/forum/index.php?topic=244.0
2012
Amazon Outage
Mandel Bug Hidden bugs that made Amazon Web Service outage worse. Cnet
http://www.incamouflage.com/2009/05/bug-camouflage/ News, July 3, 2012.
http://news.cnet.com/8301-1009_3-57465761-83/hidden-bugs-that-
Heisen Bug made-amazon-web-service-outage-worse/
http://www.adriennehedger.com/2009/07/june-bug-situation.html
http://openclipart.org/detail/145735/cartoon-fly-by-printerkiller Hic Sunt Dracones
http://tomlytle.com/
Power Law Models
Dragon-Kings, Black Swans and the Prediction of Crises. Looking under the Model-based Lampost?
Didier Sornette. International Journal of Terraspace Science and http://stconsultant.blogspot.com/2010/11/old-saying.html
Engineering 2009
http://www.youtube.com/watch?v=FlTSbzOvKZI

Model-based Testing: Today And Tomorrow

Recommended

Recommended

More Related Content

Similar to Model-based Testing: Today And Tomorrow

Similar to Model-based Testing: Today And Tomorrow (20)

More from Bob Binder

More from Bob Binder (20)

Recently uploaded

Recently uploaded (20)

Model-based Testing: Today And Tomorrow