SlideShare a Scribd company logo
1 of 113
Download to read offline
Automated	
  Failure	
  Analysis	
  in	
  
Absence	
  of	
  Specifica7on	
  
Leonardo	
  Mariani	
  
University	
  of	
  Milano	
  Bicocca	
  
mariani@disco.unimib.it	
  
Analysis	
  of	
  So?ware	
  Behaviors	
  
Analysis	
  of	
  So?ware	
  Failures	
  
(semi-­‐)automa7cally,	
  when	
  
no	
  specifica7on	
  is	
  available	
  
Automated	
  Debugging	
  
Fault	
  localiza7on:	
  search	
  for	
  the	
  fault	
  loca7on	
  	
  
	
   	
  (e.g.,	
  search	
  for	
  faulty	
  program	
  statements	
  
	
   	
  in	
  the	
  program	
  source	
  code)	
  
Failure	
  analysis	
  (aka	
  anomaly	
  detec7on):	
  search	
  
	
   	
  for	
  failure	
  causes	
  	
  
	
   	
  (e.g.,	
  search	
  for	
  erroneous	
  events	
  in	
  the	
   	
  
	
   	
  execu7on	
  space)	
  
Example	
  
•  known	
  issue	
  in	
  Tomcat	
  6.0.0	
  (to	
  6.0.9)	
  
8	
  
Fault	
  Localiza7on	
  (output	
  obtained	
  with	
  Tarantula)	
  
public	
  class	
  ChipsListener	
  implements	
  ServletContextListener	
  {	
  
	
  	
  	
  	
  public	
  ChipsListener()	
  	
  	
  {	
  }	
  
	
  	
  	
  	
  
	
  	
  	
  	
  public	
  void	
  contextIni7alized(ServletContextEvent	
  evt)	
  {	
  	
  
	
  	
  	
  	
   	
  ServletContext	
  context	
  =	
  evt.getServletContext();	
  	
  
	
  	
  	
  	
   	
  JspApplica7onContext	
  jspContext	
  =	
  
	
  JspFactory.getDefaultFactory().getJspApplica7onContext(context);	
  	
  
	
  	
  	
  	
  	
  	
  	
  	
  	
  jspContext.addELResolver(new	
  ChipsELResolver());	
  	
  
	
  	
  	
  	
  }	
  	
  
	
  
n  even	
  if	
  related	
  to	
  the	
  bug,	
  the	
  bug	
  is	
  not	
  there!	
  
n  this	
  is	
  not	
  the	
  best	
  ranked	
  piece	
  of	
  code	
  
n  why	
  is	
  this	
  fragment	
  of	
  code	
  relevant?	
  
9	
  
[Jones	
  et	
  al.	
  Visualiza7on	
  of	
  Test	
  Informa7on	
  to	
  Assist	
  Fault	
  Localiza7on.	
  ICSE,	
  2002.]	
  
Failure	
  Analysis	
  (output	
  obtained	
  with	
  BCT)	
  
•  javax.servlet.jsp.JspFactory.getDefaultFactory()	
  returned	
  null	
  
•  then	
  
org.apache.tomcat.u7l.modeler.Registry.unregisterComponent(javax.managem
ent.ObjectName)	
  invoked	
  
org.apache.catalina.session.ManagerBase.postDeregister()	
  	
  
•  and	
  
org.apache.tomcat.u7l.modeler.Registry.unregisterComponent(javax.mana
gement.ObjectName)	
  	
  invoked	
  
org.apache.catalina.loader.WebappLoader.postDeregister()	
  	
  
11	
  
[Mariani,	
  Pastore,	
  Pezzè.	
  Dynamic	
  Analysis	
  for	
  Diagnosing	
  Integra7on	
  Faults.	
  TSE,	
  2011.]	
  
Tomcat	
  failed	
  because	
  
How	
  to	
  iden7fy	
  the	
  
events	
  responsible	
  for	
  
a	
  failure	
  when	
  no	
  spec	
  
is	
  available?	
  
Specifica?on	
  Mining:	
  
Learn	
  specifica?ons	
  
from	
  actual	
  execu?ons	
  
Specifica7on	
  mining	
  for	
  
failure	
  analysis:	
  learn	
  
the	
  regular	
  behavior	
  
(specifica(on)…	
  to	
  
detect	
  anomalous	
  
events	
  
Behavioral	
  Anomalies	
  
1 2
a
3
c
4 5
d f
eb
Applica7on	
  
Traces	
  
Model	
  
Analysis	
  
What	
  models	
  can	
  we	
  mine?	
  
What	
  traces	
  do	
  we	
  need?	
  
How	
  can	
  we	
  analyze	
  a	
  failure?	
  
Trace	
  
Failure	
  
What	
  models	
  can	
  we	
  mine?	
  
generate	
  a	
  
model	
  that	
  
represents	
  the	
  
actual	
  behavior	
  
from	
  samples	
  
X=-­‐2	
  
X=-­‐9	
  
X=0	
  
X=5	
  
X=7	
  
…	
  
Actual	
  Behavior	
  
-­‐10<X<10	
  
Mined	
  behavior	
  
-­‐10<X<10	
  
Specifica7on	
  Mining	
  
Model	
  genera7on	
  is	
  imprecise…	
  
Over-­‐Generaliza7on	
   Over-­‐Restric7on	
  
Over-­‐Generaliza7on	
  
and	
  
Over-­‐Restric7on	
  
Actual	
  Behavior	
  
-­‐10<X<10	
  
Mined	
  specifica7on	
  
X	
  >	
  -­‐100	
  
Mined	
  specifica7on	
  
-­‐5	
  <	
  X	
  <	
  5	
  
Mined	
  specifica7on	
  
-­‐100	
  <	
  X	
  <	
  5	
  
Real	
  Specifica7on	
  
-­‐10<X<10	
  
Mined	
  Specifica7on	
  
-­‐100<X<5	
  
Specifica7on	
  Mining:	
  Models	
  Used	
  as	
  Specifica7ons	
  
X	
  =	
  100	
  
Correctly	
  rejected	
  
behavior	
  
X	
  =	
  1	
  
Correctly	
  accepted	
  
behavior	
  
X	
  =	
  7	
  
Erroneously	
  rejected	
  
behavior	
  
X	
  =	
  -­‐50	
  
Erroneously	
  accepted	
  
behavior	
  
Models	
  
1 2
a
3
c
4 5
d f
eb
x > 0
Full	
  Ordering	
  of	
  Events	
   Data	
  Values	
  
Ordering	
  of	
  Events	
  +	
  
Data	
  Values	
  
Par7al	
  Ordering	
  of	
  
Events	
  
open	
  =>	
  close	
  
MODELS	
  THAT	
  REPRESENT	
  THE	
  
FULL	
  ORDERING	
  OF	
  EVENTS	
  
Mining	
  of	
  Finite	
  State	
  Models	
  
•  Trace-­‐based	
  mining	
  
– State-­‐based	
  merging	
  
– Behavior-­‐based	
  merging	
  
•  State-­‐based	
  mining	
  
Total	
  =	
  0	
  
Elem	
  =	
  0	
  
Total	
  =	
  3	
  
Elem	
  =	
  1	
  
Total	
  =	
  5	
  
Elem	
  =	
  2	
  
Total	
  =	
  0	
  
Elem	
  =	
  0	
  
onLoad	
   add	
   add	
   empty	
  
onLoad	
   add	
   add	
   empty	
  
kTail	
  (state-­‐based	
  merging)	
  
TRACES	
  
PTA	
  
FSA	
  
[Biermann	
  and	
  Feldman.	
  On	
  the	
  synthesis	
  of	
  
finite	
  state	
  machines	
  from	
  samples	
  of	
  their	
  
behavior.	
  IEEE	
  ToC,	
  1972.	
  ]	
  
a	
   a	
   a	
   b	
   c	
  
a	
   b	
   c	
  
a	
   a	
   b	
   c	
  
a	
   a	
   a	
   a	
   a	
   c	
  
Build	
  the	
  PTA	
  
TRACES	
  
PTA	
  
a	
   a	
   a	
   b	
   c	
  
a	
   b	
   c	
  
a	
   a	
   b	
   c	
  
a	
   a	
   a	
   a	
   a	
   c	
  
k=2	
  
2-­‐future(2)	
  =	
  {aa,ab,bc}	
  
2-­‐future(5)	
  =	
  {aa,	
  bc}	
  
2-­‐future(11)	
  =	
  {}	
  
2-­‐future(8)	
  =	
  {c}	
  
…	
  
2	
  FUTURES	
  
2-­‐future(8)	
  =	
  {c}	
   2-­‐future(12)	
  =	
  {c}	
  
TAROT2013 Testing School - Leonardo Mariani presentation
2-­‐future(11)	
  =	
  {}	
   2-­‐future(13)	
  =	
  {}	
  
TAROT2013 Testing School - Leonardo Mariani presentation
2-­‐future(2)	
  =	
  {aa,	
  ab,	
  bc}	
   2-­‐future(3)	
  =	
  {aa,	
  ab,	
  bc}	
  
TAROT2013 Testing School - Leonardo Mariani presentation
…	
  
TAROT2013 Testing School - Leonardo Mariani presentation
K	
  
Over-­‐
restric?on	
  
Over-­‐
generaliza?on	
  
Small	
  K	
   Big	
  K	
  
The	
  Parameter	
  K	
  
kBehavior	
  (behavior-­‐based	
  merging)	
  
•  incremental	
  
Traces	
  
login	
   home	
   checkMsg	
   logout	
  
[Mariani,	
  Pastore,	
  Pezzè.	
  Dynamic	
  Analysis	
  for	
  Diagnosing	
  Integra7on	
  Faults.	
  TSE,	
  2011.]	
  
kBehavior	
  (behavior-­‐based	
  merging)	
  
Traces	
  
login	
   home	
   checkMsg	
   logout	
  
login	
   home	
   checkMsg	
   7meout	
  
kBehavior	
  (behavior-­‐based	
  merging)	
  
Traces	
  
login	
   home	
   checkMsg	
   logout	
  
login	
   home	
   checkMsg	
   7meout	
  
kBehavior	
  (behavior-­‐based	
  merging)	
  
Traces	
  
login	
   home	
   checkMsg	
   logout	
  
login	
   home	
   checkMsg	
   7meout	
  
login	
   home	
   checkMsg	
  watchVideo	
  home	
   checkMsg	
   logout	
  
K	
  =	
  min	
  length	
  of	
  
matched	
  behavior	
  
kBehavior	
  (behavior-­‐based	
  merging)	
  
Traces	
  
login	
   home	
   checkMsg	
   logout	
  
login	
   home	
   checkMsg	
   7meout	
  
login	
   home	
   checkMsg	
  watchVideo	
  home	
   checkMsg	
   logout	
  
kBehavior	
  (behavior-­‐based	
  merging)	
  
Traces	
  
login	
   home	
   checkMsg	
   logout	
  
login	
   home	
   checkMsg	
   7meout	
  
login	
   home	
   checkMsg	
  watchVideo	
  home	
   checkMsg	
   logout	
  
login	
   home	
   checkMsg	
   read	
   home	
   checkMsg	
   logout	
  reply	
  
kBehavior	
  (behavior-­‐based	
  merging)	
  
Traces	
  
login	
   home	
   checkMsg	
   logout	
  
login	
   home	
   checkMsg	
   7meout	
  
login	
   home	
   checkMsg	
  watchVideo	
  home	
   checkMsg	
   logout	
  
login	
   home	
   checkMsg	
   read	
   home	
   checkMsg	
   logout	
  reply	
  
The	
  Parameter	
  K	
  
•  K	
  determines	
  the	
  degree	
  of	
  generaliza7on	
  
•  Empirically,	
  behavior-­‐based	
  merging	
  
generates	
  models	
  that	
  are	
  more	
  general	
  than	
  
state-­‐based	
  merging	
  [Lo	
  et	
  al.,	
  JSS,	
  2012]	
  
State-­‐based	
  merging	
  
behavior-­‐based	
  
merging	
  
State-­‐Based	
  Inference	
  of	
  FSM	
  Models	
  
Total	
  =	
  0	
  
Elem	
  =	
  0	
  
Total	
  =	
  3	
  
Elem	
  =	
  1	
  
Total	
  =	
  5	
  
Elem	
  =	
  2	
  
Total	
  =	
  0	
  
Elem	
  =	
  0	
  
onLoad	
   add	
   add	
   empty	
  
<0	
  
==0	
  
>0	
  
Abstrac7on	
  
func7on	
  
<0	
  
=0	
  
>0	
  
<0	
  
=0	
  
>0	
  
<0	
  
=0	
  
>0	
  
<init>	
  
Total	
  ==	
  0	
  
Elem==0	
  
Total	
  >	
  0	
  
Elem	
  >	
  0	
  
onLoad	
   add	
  
add	
  
empty	
  
[Dallmeier,	
  Lindig,	
  Wasylkowski,	
  Zeller:	
  Mining	
  Object	
  Behavior	
  with	
  ADABU.	
  WODA	
  2006]	
  
[Marcheso,	
  Tonella,	
  Ricca:	
  State-­‐Based	
  Tes7ng	
  of	
  Ajax	
  Web	
  Applica7ons.	
  ICST	
  2008]	
  
[Mariani,	
  Marcheso,	
  Nguyen,	
  Tonella.	
  Revolu7on:	
  Automa7c	
  evolu7on	
  of	
  mined	
  
specifica7ons.	
  ISSRE.	
  2012]	
  
The	
  Abstrac7on	
  Func7on	
  
•  Quality	
  of	
  the	
  final	
  model	
  influenced	
  by	
  
– Completeness	
  of	
  the	
  state	
  informa7on	
  that	
  is	
  
traced	
  
– The	
  kind	
  of	
  abstrac7on	
  implemented	
  by	
  the	
  
abstrac7on	
  func7on	
  	
  
	
  
numElements	
  
numDis7nctElements	
  
numElements	
  VS	
  
<0	
  	
  	
  	
  =0	
  	
  	
  	
  >0	
   <-­‐1	
  	
  	
  	
  =-­‐1	
  	
  	
  	
  =0	
  	
  	
  	
  =1	
  	
  	
  >1	
  VS	
  
MODELS	
  THAT	
  REPRESENT	
  THE	
  
VALUES	
  OF	
  VARIABLES	
  
Program	
  for	
  eLectronic	
  Commerce	
  
Execu7ons	
  
Trace	
  with	
  
variable	
  values	
  
totalCost	
  unitCost	
  
4	
  
3
1
7
…
	
  
8	
  
3
8
14
…
	
  
totalCost	
  unitCost	
  
4	
  
3
1
7
…
	
  
8	
  
3
8
14
…
	
  
_	
  +	
  _	
  =_	
   _	
  <	
  _	
  
_=_	
  
_	
  >	
  0	
  
unitCost	
  =	
  totalCost	
  
unitCost	
  <	
  totalCost	
  
unitCost	
  <=	
  totalCost	
  
unitCost	
  +	
  totalCost	
  >	
  unitCost	
  
…	
  
preserve	
  expressions	
  with	
  
perfect	
  confidence	
  
unitCost	
  <=	
  totalCost	
  
unitCost	
  +	
  totalCost	
  >	
  unitCost	
  
totalCost	
  >	
  0	
  
	
  
1	
  >	
  _	
  
remove	
  redundant	
  proper(es	
  
unitCost	
  <=	
  totalCost	
  
totalCost	
  >	
  	
  0	
  
Daikon	
  in	
  a	
  
nutshell	
  
Traces	
   Template	
  Expressions	
  
Candidate	
  Expressions	
  
[Ernst,	
  Cockrell,	
  
Griswold,	
  Notkin.	
  
Dynamically	
  
Discovering	
  Likely	
  
Program	
  Invariants	
  
to	
  Support	
  Program	
  
Evolu7on.	
  IEEE	
  TSE	
  
2001]	
  
The	
  Set	
  of	
  Template	
  Expressions	
  
•  Expressiveness	
  depends	
  on	
  the	
  template	
  expressions	
  
•  More	
  template	
  expr	
  =>	
  more	
  candidate	
  expressions	
  
=>	
  higher	
  computa7onl	
  cost	
  
•  Recently	
  defined	
  an	
  approach	
  to	
  deal	
  with	
  
polynomial	
  and	
  array	
  expressions	
  [Nguyen	
  et	
  al.	
  ICSE	
  2012]	
  
_	
  +	
  _	
  =_	
   _	
  <	
  _	
  
_=_	
  
_	
  >	
  0	
  1	
  >	
  _	
  
MODELS	
  THAT	
  REPRESENT	
  PARTIAL	
  
ORDERING	
  OF	
  EVENTS	
  
Mine	
  Temporal	
  Rules	
  
Traces	
  
Template	
  Rules	
  
<pre>	
   <post>	
  
CONFIDENCE	
  AND	
  SUPPORT	
  THRESHOLDS	
  
Temporal	
  Rules	
  
[Lo,	
  Khoo,	
  Liu.	
  Mining	
  temporal	
  
rules	
  for	
  so?ware	
  maintenance.	
  
JSME,	
  2008]	
  
[Yang,	
  Evans,	
  Bhardwaj,	
  Bhat,	
  Das.	
  
Perracosa:	
  mining	
  temporal	
  API	
  
Rules	
  from	
  Imperfect	
  Traces.	
  ICSE.	
  
2006]	
  
…	
  
start	
   open	
   close	
   stop	
  
start	
   load	
   stop	
  
start	
   open	
   close	
   stop	
  
begin	
   end	
  
Mine	
  Temporal	
  Rules	
  
start	
   open	
   close	
   stop	
  
start	
   load	
   stop	
  
start	
   open	
   close	
   stop	
  
Traces	
  
Template	
  Rules	
  
<pre>	
   <post>	
  
CONFIDENCE	
  AND	
  SUPPORT	
  THRESHOLDS	
  
Temporal	
  Rules	
  
CONFIDENCE	
  OF	
  A	
  RULE	
  
#	
  traces	
  rule	
  holds	
  
…	
  
#traces	
  	
  pre	
  applies	
  
start	
   open	
  
has	
  67%	
  confidence	
  
begin	
   end	
  
Mine	
  Temporal	
  Rules	
  
Traces	
  
Template	
  Rules	
  
<pre>	
   <post>	
  
CONFIDENCE	
  AND	
  SUPPORT	
  THRESHOLDS	
  
Temporal	
  Rules	
  
Conf	
  =	
  100%	
  
Supp	
  =	
  20%	
  
start	
   stop	
  
close	
  open	
  
…	
  
…	
  
SUPPORT	
  OF	
  A	
  RULE	
  
#	
  traces	
  rule	
  holds	
  
#traces	
  
start	
   open	
  
has	
  50%	
  support	
  
start	
   open	
   close	
   stop	
  
start	
   load	
   stop	
  
start	
   open	
   close	
   stop	
  
begin	
   end	
  
Template	
  Rules	
  
•  Expressiveness	
  depends	
  on	
  the	
  template	
  rules	
  
•  Confidence	
  and	
  Support	
  for	
  tuning	
  the	
  
technique	
  wrt	
  imperfect	
  traces	
  
Steering	
  FSA	
  Models	
  with	
  Temporal	
  Rules	
  
kTail	
  with	
  k=2	
  
Overgeneraliza7on	
  problem:	
  
-­‐  locally,	
  it	
  sounds	
  to	
  be	
  a	
  
good	
  decision	
  
-­‐	
  globally,	
  it	
  generates	
  
anomalous	
  behaviors	
  
Idea:	
  mine	
  global	
  proper7es,	
  exploit	
  
them	
  when	
  taking	
  decisions	
  locally	
  
Traces	
  
Mine	
  Temporal	
  Rules	
  
Build	
  PTA	
  
openFile	
   closeFile	
  
closeConn	
  connDB	
  
Apply	
  kTail	
  (e.g.	
  
with	
  k=2)	
  
BUT	
  
prevent	
  state	
  
merges	
  that	
  violate	
  
temporal	
  rules	
  
(LOCAL	
  DECISIONS)	
  
[Lo,	
  Mariani,	
  Pezzè.	
  Automa7c	
  Steering	
  of	
  Behavioral	
  Model	
  Inference.	
  ESEC/FSE	
  2009]	
  
GLOBAL	
  PROPERTIES	
  
EXTENDED	
  MODELS	
  
FSA	
  With	
  Annota7ons	
  to	
  Represent	
  
•  Constraints	
  on	
  parameter	
  values	
  	
  
Traces	
  With	
  Parameter	
  Values	
  
addItem	
   addItem	
   buy	
  
qt=1	
  
unitCost=1	
  
totalCost=1	
  
qt=2	
  
unitCost=3	
  
totalCost=6	
  
Traces	
  
…	
  
GKTail	
  
merging	
  similar	
  traces	
  
EFSM	
  
[Lorenzoli,	
  Mariani,	
  Pezzè.	
  
Automa7c	
  Genera7on	
  of	
  
So?ware	
  Behavioral	
  Models,	
  
ICSE,	
  2008]	
  
0 1 2 3 4 5 6
m1
0≤x≤15
m1
x=1
m2
x=0
y=0
x=y
m3
z={’IT’,’UK’}
m1
x=0
m2
x=0
0≤y≤20
8 9 10 11 12 13
m3
z=’UK’
m3
z=’UK’
m2
x=0
y=3
m3
z=’UK’
m1
x=0
m2
x=0
y=15
22 23 24 25 26 27
m1
x=0
m1
x=1
m2
x=0
y=0
x=y
m3
z=’IT’
m3
z=’IT’
m2
x=0
y=30
PTA	
  
deriving	
  guards	
  
X=0	
  
Y=0,	
  
X=0	
  
Y=15,	
  
…	
  
X=0,	
  	
  
0≤Y≤20	
  
DAIKON	
  
m2	
   m2	
  
GKTail	
  
merging	
  similar	
  traces	
   deriving	
  guards	
  
EFSM	
  
[Lorenzoli,	
  Mariani,	
  Pezzè.	
  
Automa7c	
  Genera7on	
  of	
  
So?ware	
  Behavioral	
  Models,	
  
ICSE,	
  2008]	
  
0 1 2 3 4 5 6
m1
0≤x≤15
m1
x=1
m2
x=0
y=0
x=y
m3
z={’IT’,’UK’}
m1
x=0
m2
x=0
0≤y≤20
8 9 10 11 12 13
m3
z=’UK’
m3
z=’UK’
m2
x=0
y=3
m3
z=’UK’
m1
x=0
m2
x=0
y=15
22 23 24 25 26 27
m1
x=0
m1
x=1
m2
x=0
y=0
x=y
m3
z=’IT’
m3
z=’IT’
m2
x=0
y=30
PTA	
  
X=0	
  
Y=0,	
  
X=0	
  
Y=15,	
  
…	
  
X=0,	
  	
  
0≤Y≤20	
  
DAIKON	
  
m2	
   m2	
  
Mining	
  Specifica7ons:	
  Different	
  
Models	
  for	
  Different	
  Aspects	
  
Different	
  models	
  can	
  capture	
  different	
  types	
  of	
  anomalous	
  behaviors	
  
Specifica7on	
  Mining	
  Tools	
  
•  Synop7c	
  
– hsp://code.google.com/p/synop7c/	
  
•  Perracosa	
  
– hsp://www.cs.virginia.edu/perracosa/	
  
•  Adabu	
  
– hsp://www.st.cs.uni-­‐saarland.de/models/
adabu.php3	
  
•  KLFA	
  
– hsp://www.lta.disco.unimib.it/tools/klfa/	
  
(When)	
  Are	
  Mined	
  Models	
  
Precise	
  Enough?	
  
Empirical	
  Studies	
  
-­‐	
  complexity	
  -­‐	
  
Length	
  of	
  traces/Noise/Number	
  	
  
of	
  different	
  events	
  in	
  the	
  traces	
  
	
  
Mining	
  simple	
  FSA	
  Mining	
  extended	
  FSA	
  
Mining	
  temporal	
  rules	
  
Mining	
  constraints	
  
[Lo,	
  Mariani,	
  Santoro,	
  Learning	
  extended	
  FSA	
  from	
  So?ware:	
  An	
  Empirical	
  Assessment.	
  	
  
JSS,	
  2012]	
  
[Yang,	
  Evans,	
  Bhardwaj,	
  Bhat,	
  Das.	
  Perracosa:	
  mining	
  temporal	
  API	
  Rules	
  from	
  Imperfect	
  
Traces.	
  ICSE.	
  2006]	
  
[Nugyen,	
  Marcheso,	
  Tonella.	
  Automated	
  Oracles:	
  An	
  Empirical	
  Study	
  on	
  Cost	
  and	
  
Effec7veness,	
  ESEC/FSE,	
  2013]	
  
Empirical	
  Studies	
  
-­‐	
  sensi7vity	
  -­‐	
  
Capture	
  small	
  differences	
  
Mining	
  simple	
  FSA	
  Mining	
  extended	
  FSA	
  
Mining	
  temporal	
  rules	
  
Mining	
  constraints	
  
Capture	
  major	
  differences	
  
FSA	
  good	
  to	
  analy7cally	
  
capture	
  the	
  behavior	
  of	
  small	
  
units	
  (e.g.,	
  components)	
  
Temporal	
  rules	
  and	
  constraints	
  
good	
  to	
  capture	
  some	
  behaviors	
  
in	
  rela7vely	
  big	
  applica7ons	
  
Quality	
  of	
  Models	
  vs	
  Number	
  of	
  Traces	
  
Component/
API/Method	
   Traces	
  
Ideal	
  Model	
  
-­‐	
  Transi7on	
  coverage	
  
enough	
  for	
  mining	
  good	
  
FSAs	
  [Lo,	
  JSS,	
  2012]	
  
-­‐	
  Addi7onal	
  tests	
  can	
  be	
  
generated	
  to	
  improve	
  
models	
  [Dallmeier,	
  TSE,	
  
2012]	
  
	
  
Quality	
  of	
  Models	
  vs	
  Number	
  of	
  Traces	
  
Applica7on	
   Traces	
  
Ideal	
  Model	
  
-­‐	
  Good	
  FSAs	
  hard	
  to	
  
mine	
  	
  
-­‐	
  Other	
  models:	
  several	
  
traces	
  necessary	
  for	
  
par7cularly	
  complex	
  
cases	
  [Nguyen,	
  ESEC/FSE,	
  
2013]	
  
Take	
  Home	
  About	
  Specifica7on	
  Mining	
  
•  Think	
  to	
  your	
  research	
  area	
  
– If	
  you	
  need	
  models	
  and	
  
specifica7ons…	
  
– …and	
  you	
  do	
  not	
  have	
  any,	
  
– but	
  you	
  have	
  a	
  way	
  of	
  
execu7ng	
  your	
  so?ware	
  
– Specifica7on	
  Mining	
  could	
  
an	
  op7on!	
  
Failure	
  Analysis	
  
1 2
a
3
c
4 5
d f
eb
Applica7on	
  
Traces	
  
Model	
  
Analysis	
  
Trace	
  
Failure	
  
Failure	
  Analysis	
  Based	
  on	
  Specifica7on	
  
Mining	
  
•  Analysis	
  of	
  (Field	
  and	
  Regression)	
  Failures	
  
– BCT	
  
	
  [Mariani	
  et	
  al.	
  Dynamic	
  Analysis	
  for	
  Diagnosing	
  Integra7on	
  Faults.	
  TSE,	
  2011.]	
  
•  Analysis	
  of	
  Regression	
  Failures	
  
– Radar	
  
[Pastore	
  et	
  al.	
  Dynamic	
  Analysis	
  of	
  Upgrades	
  in	
  C/C++	
  So?ware.	
  ISSRE,	
  2012.]	
  
•  Produce	
  Descrip7ve	
  Reports	
  
– AVA	
  
[Babenko	
  et	
  al.	
  AVA:	
  automated	
  interpreta7on	
  of	
  dynamically	
  detected	
  
anomalies.	
  ISSTA,	
  2009.]	
  
BCT:	
  a	
  technique	
  for	
  automated	
  
iden7fica7on	
  of	
  func7onal	
  faults	
  
1.  Capturing	
  Behavioral	
  Data	
  
•  Monitoring	
  Component	
  Execu7ons	
  
•  Capturing	
  Run-­‐Time	
  Informa7on	
  
2.  Dis(lling	
  Behavioral	
  Models	
  
•  I/O	
  Models	
  
•  Interac7on	
  Models	
  
3.  Failure	
  Analysis	
  
•  Regression	
  failures	
  
•  Field	
  failure	
  analysis	
  
addItem(c)
c.getTotalCost>=c.getCost
c.getQuantity>0
...
getCart()
Cart
newCart()
addItem(c)
getCart()
...
Catalog
imageDB
Failure!!
unexpected	
  
interac?on!	
  
unexpected	
  
interac?on!	
  
unexpected	
  
interac?on!	
  
unexpected	
  
value!	
  
unexpected	
  
value!	
  
System
Capturing	
  IO	
  Data	
  
purchase	
  
…	
  
checkCreden7als(usr,	
  taskType)	
  
makeOrder(cart,	
  usr)	
  
checkCreden7als(User,	
  TaskType)	
  
usr.name	
  =	
  “Leonardo”	
  
usr.address.streetName	
  =	
  “viale	
  Certosa”	
  
…	
  
taskType.type	
  =	
  2	
  
…	
  
	
  
usr.name	
  =	
  “Carlo”	
  
usr.address.streetName	
  =	
  “viale	
  Manzoni”	
  
…	
  
taskType.type	
  =	
  1	
  
…	
  
execu7on	
  1	
  
execu7on	
  2	
  
makeOrder(cart,	
  usr)	
  
purchase(cart)	
  
Capturing	
  Interac7on	
  Data	
  
101	
  
purchase	
  
…	
  
checkCreden7als(usr,	
  taskType)	
  
makeOrder(cart,	
  usr)	
  
purchase()	
  
Auth.checkCreden7als(User,TaskType)	
  
Shop.makeOrder(Cart,	
  User)	
  
…	
  
	
  
Auth.checkCreden7als(User,	
  TaskType)	
  
Shop.makeOrder(Cart,	
  User)	
  
…	
  
	
  
Auth.checkCreden7als(User,	
  TaskType)	
  
Logger.logPermissionDenied(User,	
  TaskType)	
  
…	
  
execu7on	
  1	
  
execu7on	
  2	
  
execu7on	
  3	
  
Dis7lling	
  Behavioural	
  Models	
  
I/O	
  Data	
  
Interac7on	
  Data	
  
Daikon	
  
kBehavior	
  
I/O	
  Model	
  
Interac7on	
  Model	
  
x != null
method1	
  
method2	
  
method3	
  
method4	
  
I/O	
  and	
  Interac7on	
  
Models	
  
I/O	
  and	
  Interac7on	
  
Models	
  
I/O	
  and	
  Interac7on	
  
Models	
  
I/O	
  and	
  Interac7on	
  
Models	
  
Run-­‐Time	
  Verifica(on	
  and	
  Failure	
  
Analysis	
  
System Failure!!
unexpected	
  
interac?on!	
  
unexpected	
  
interac?on!	
  
unexpected	
  
interac?on!	
  
unexpected	
  
value!	
  
unexpected	
  
value!	
  
103	
  
Filtering	
  
•  Re-­‐execute	
  tests	
  and	
  remove	
  anomalies	
  detected	
  in	
  both	
  passing	
  
and	
  failing	
  tests	
  
	
  
Regression	
  tes7ng	
  
•  Country==US violated	
  by	
  passing	
  regression	
  tests	
  because	
  the	
  new	
  
version	
  of	
  the	
  applica7on	
  is	
  available	
  	
  outside	
  US	
  
•  Viola7ons	
  of	
  this	
  property	
  can	
  be	
  ignored	
  
	
  
Field	
  failures	
  
•  date==20/3/2013 spurious property violated	
  by	
  passing	
  
regression	
  tests	
  
•  Viola7ons	
  of	
  this	
  property	
  can	
  be	
  ignored	
  
Remaining	
  anomalies	
  are	
  re-­‐arranged	
  according	
  to	
  likely	
  cause-­‐
effects	
  
Run-­‐Time	
  Verifica(on	
  and	
  Failure	
  Analysis:	
  
Rela(ng	
  Anomalies	
  
105	
  
start	
  
ini?alize	
  
getValue	
  
I	
  ini7alize	
  the	
  
next	
  component	
  
I	
  need	
  a	
  proper	
  
value	
  for	
  
ini7aliza7on	
  
I	
  do	
  not	
  know	
  
the	
  value!!!	
  I	
  
return	
  null
null	
  is	
  not	
  a	
  
proper	
  value!	
  I	
  
return	
  an	
  excep7on	
  
we	
  have	
  an	
  
excep7on!!!	
  Let’s	
  try	
  to	
  
terminate	
  safely
Run-­‐Time	
  Verifica(on	
  and	
  Failure	
  Analysis:	
  
Rela(ng	
  Failures	
  	
  
106	
  
start	
  
ini?alize	
  
getValue	
  
undo	
  
log	
   closeConnec?on	
  
log	
  the	
  event	
  
and	
  close	
  the	
  
connec7on
Run-­‐Time	
  Verifica(on	
  and	
  Failure	
  Analysis:	
  
Rela(ng	
  Failures	
  	
  
107	
  
start	
  
ini?alize	
  
getValue	
  
undo	
  
log	
   closeConnec?on	
  
one	
  anomaly	
  is	
  the	
  cause	
  of	
  many	
  others!	
  
return	
  null	
  value	
  
throw	
  excep?on	
  
call	
  undo	
  
early	
  close	
  the	
  
connec?on	
  
Capturing	
  
Clusters	
  
Dynamic	
  call	
  tree	
  for	
  the	
  Tomcat	
  case	
  study	
  
ini?al	
  anomaly	
  graph	
  
109	
  
Capturing	
  
Clusters	
  
Dynamic	
  call	
  tree	
  for	
  the	
  Tomcat	
  case	
  study	
  
ini?al	
  anomaly	
  graph	
  
110	
  
Capturing	
  
Clusters	
  
Dynamic	
  call	
  tree	
  for	
  the	
  Tomcat	
  case	
  study	
  
ini?al	
  anomaly	
  graph	
  
111	
  
Output	
  Obtained	
  with	
  BCT	
  for	
  the	
  
Tomcat	
  Failure	
  
ON	
  EXIT	
  from	
  javax.servlet.jsp.JspFactory.getDefaultFactory()	
  
MODEL	
  VIOLATED	
  returnValue	
  !=	
  null	
  =	
  false	
  
FROM	
  
org.apache.tomcat.u7l.modeler.Registry.unregisterComponent(javax.man
agement.ObjectName)	
  
	
  
UNEXPECTED	
  CALL	
  TO	
  
org.apache.catalina.session.ManagerBase.postDeregister()	
  	
  
FROM	
  	
  
org.apache.tomcat.u7l.modeler.Registry.unregisterComponent(ja
vax.management.ObjectName)	
  	
  
	
  
UNEXPECTED	
  CALL	
  TO	
  	
  
org.apache.catalina.loader.WebappLoader.postDeregister()	
  	
  
112	
  
Eclipse	
  3.3	
  Anomaly	
  Graph	
  
113	
  
Capturing	
  
Clusters	
  
Dynamic	
  call	
  tree	
  for	
  the	
  Tomcat	
  case	
  study	
  
ini?al	
  anomaly	
  graph	
  
114	
  
Eclipse	
  3.3	
  Anomaly	
  Graph	
  
115	
  
Stopping	
  Criterion	
  
edges	
  with	
  weights	
  greater	
  than	
  
this	
  value	
  are	
  removed	
  
cohesion(graph)	
  
116	
  
cohesion(graph)	
  =	
  avg(cohesion	
  (CCs))	
  
cohesion(CC)	
  =	
  avg(weight	
  edges)	
  
	
  
smaller	
  value	
  ==	
  beser	
  cohesion	
  
	
  
Resul7ng	
  Graph	
  
…	
  
• 	
  the	
  components	
  are	
  
inspected	
  from	
  the	
  biggest	
  to	
  
the	
  smallest	
  
	
  
• 	
  the	
  first	
  two	
  graphs	
  are	
  
enough	
  to	
  explain	
  the	
  
problem!	
  
117	
  
Improvements	
  
Make	
  the	
  analysis	
  
specific	
  to	
  the	
  
type	
  of	
  
considered	
  faults	
  
Radar:	
  failure	
  analysis	
  
of	
  regression	
  problems	
  
Produce	
  outputs	
  
that	
  beser	
  explain	
  
the	
  reason	
  of	
  the	
  
failure	
  
AVA:	
  automa7c	
  
analysis	
  of	
  anomalies	
  
V1	
   V2	
  
chainItems.size > 0
24	
  
25	
  
27	
  
28	
  
29	
  
31	
  
availableQty	
  
32	
  
34	
  
36	
  
37	
  
28	
  
28	
  
33	
  
Radar	
  in	
  a	
  Nutshell	
  
TEST	
  SUITE	
  TEST	
  SUITE	
  
TRACE	
  
Failed	
  because	
  
initItems()	
  has	
  not	
  
been	
  invoked	
  and	
  
chainItem.size	
  =0	
  
AVA:	
  PRODUCING	
  DESCRIPTIVE	
  
OUTPUTS	
  
File.open
File.write
File.close sortFile File.delete
File.open	
   …sortFile	
   File.delete	
  File.write	
  
Anomaly	
  Detec7on	
  with	
  FSA	
  
File.open
File.write
File.close sortFile File.delete
File.open …sortFile File.deleteFile.write
Anomaly	
  Detec7on	
  with	
  FSA	
  
File.open
File.write
File.close sortFile File.delete
File.open …sortFile File.deleteFile.write
Anomaly	
  Detec7on	
  
File.open
File.write
File.close sortFile File.delete
File.open …sortFile File.deleteFile.write
Anomaly	
  Detec7on	
  
Should the path of the file be
the problem?
Should the sorting be the
problem?
Should the content of the file
be the problem?
May be the file has not been
closed!!!
…
File.open
File.write
File.close sortFile File.delete
File.open …sortFile File.deleteFile.write
File.open
File.write
File.close sortFile File.delete
File.open …sortFile File.deleteFile.write
Anomaly interpretation:
Missing event: File.close
The	
  file	
  has	
  not	
  been	
  closed!	
  
AVA	
  =	
  Compare	
  actual	
  and	
  
expected	
  behaviors	
  (represented	
  
with	
  a	
  FSA)	
  to	
  produce	
  informa7ve	
  
outputs	
  
Automata	
  Viola7ons	
  Analysis	
  -­‐	
  AVA	
  
Iden7fy	
  
Model	
  
Viola7ons	
  
____	
  
_____	
  
_____	
  
_____	
  
_____	
  
_____	
  
_____	
  
	
  
	
   FSA	
  
Trace	
  
Branches
Tails
Final States
Automata	
  Viola7ons	
  Analysis	
  -­‐	
  AVA	
  
Iden7fy	
  
Model	
  
Viola7ons	
  
Iden7fy	
  
Basic	
  
Interpreta7ons	
  
____	
  
_____	
  
_____	
  
_____	
  
_____	
  
_____	
  
_____	
  
	
  
	
   FSA	
  
Trace	
  
Deletions
Insertions
Replacements
Terminations
Branches
Tails
Final States
Automata	
  Viola7ons	
  Analysis	
  -­‐	
  AVA	
  
Iden7fy	
  
Model	
  
Viola7ons	
  
Iden7fy	
  
Basic	
  
Interpreta7ons	
  
Iden7fy	
  
Composite	
  
Interpreta7ons	
  
____	
  
_____	
  
_____	
  
_____	
  
_____	
  
_____	
  
_____	
  
	
  
	
   FSA	
  
Trace	
  
Deletions
Insertions
Replacements
Terminations
Branches
Tails
Final States
Anticipations
Postponements
Swaps
File.open
File.write
File.close sortFile File.delete
File.open	
   File.close	
  sortFile	
   File.delete	
  File.write	
  
ε File.close
Iden7fy	
  Model	
  Viola7ons	
  	
  
with	
  FSA	
  Extensions	
  
fsa’	
  =	
  kBehavior(fsa,t)	
  
extensions	
  =	
  diff(fsa,fsa’)	
  
File.open
File.write
File.close sortFile File.delete
File.open File.closesortFile File.deleteFile.write
ε
Iden7fy	
  Basic	
  Interpreta7ons	
  
File.open File.write
File.open
sortFile File.delete
sortFile File.delete
File.close
File.close
File.write File.write sortFile File.deleteFile.close
File.open sortFile File.deleteFile.write
Iden7fy	
  Basic	
  Interpreta7ons	
  
Observed sequence
Expected sequences
Compare using alignment algorithms
File.open File.write
File.open
sortFile File.delete
sortFile File.delete
File.close
File.close
File.write File.write sortFile File.deleteFile.close
File.open File.write File.close sortFile File.delete
File.open File.write - sortFile File.delete
Expected	
  
Observed	
  
The	
  applica7on	
  failed	
  
because	
  File.close	
  has	
  not	
  
been	
  executed	
  	
  
sortFile	
  is	
  anomalous	
  BETTER	
  THAN	
  
Different	
  interpreta7ons	
  can	
  be	
  discovered	
  
with	
  different	
  alignment	
  strategies	
  
Deletion
EV1 EV2 EV3 EV4 EV5
Insertion
EV1 EV2 EV3 EV EV4 EV5
Replacements
EV1 EV2 EV3 EV EV5
Terminations
EV1 EV2 EV3
Anticipations
EV1 EV5 EV2 EV3 EV4
Postponements
EV1 EV3 EV4 EV5 EV2
Swap
EV1 EV5 EV3 EV4 EV2
SIMPLE	
  INTERPRETATIONS	
   COMPOSITE	
  INTERPRETATIONS	
  
Tools	
  
•  BCT	
  
– hsp://www.lta.disco.unimib.it/tools/bct/	
  
•  Radar	
  
– hsp://www.lta.disco.unimib.it/tools/radar/	
  
•  AVA	
  
– hsp://www.lta.disco.unimib.it/tools/ava/	
  
CONCLUDING	
  REMARKS	
  
Specifica?on	
  Mining	
  	
  
can	
  be	
  used	
  to	
  enable	
  
several	
  analyses	
  in	
  the	
  
frequent	
  case	
  no	
  
specifica?on	
  is	
  available	
  
One	
  interes7ng	
  
domain	
  is	
  failure	
  
analysis	
  
Specifica7on	
  mining	
  has	
  been	
  
experienced	
  in	
  several	
  other	
  contexts	
  
Other	
  Failure	
  analysis	
  approaches	
  
[Hangal,	
  Lam.	
  Tracking	
  down	
  so?ware	
  bugs	
  using	
  automa7c	
  anomaly	
  detec7on.	
  ICSE	
  
2002]	
  
[Yilmaz,	
  Paradkar,	
  Williams.	
  Time	
  will	
  tell.	
  ICSE.	
  2010]	
  
	
  
Regression	
  tes?ng	
  
•  Behavioral	
  Regression	
  Tes7ng	
  
	
  [Jin,	
  Orso,	
  Xie.	
  Automated	
  Behavioral	
  Regression	
  Tes7ng.	
  ICST.	
  2010.]	
  
•  Compa7bility	
  Tes7ng	
  
	
  [Mariani,	
  Papagiannakis,	
  Pezzè.	
  Compa7bility	
  and	
  regression	
  tes7ng	
  of	
  COTS-­‐
Component-­‐based	
  so?ware.	
  ICSE.	
  2007.]	
  
	
  
Combined	
  with	
  Sta?c	
  analysis	
  
[Pradel,	
  Gross.	
  Leveraging	
  test	
  genera7on	
  and	
  specifica7on	
  mining	
  for	
  automated	
  bug	
  
detec7on	
  without	
  false	
  posi7ves.	
  ICSE	
  2012]	
  
[Dallmeier,	
  Zeller,	
  Meyer.	
  Genera7ng	
  fixes	
  from	
  object	
  behavior	
  anomalies.	
  ASE.	
  2009]	
  
But	
  be	
  careful	
  with	
  “posi7ve”	
  
behavioral	
  anomalies	
  
References	
  
•  Babenko,	
  Mariani,	
  Pastore.	
  AVA:	
  automated	
  interpreta7on	
  of	
  dynamically	
  detected	
  anomalies.	
  
ISSTA,	
  2009	
  
•  Biermann	
  and	
  Feldman.	
  On	
  the	
  synthesis	
  of	
  finite	
  state	
  machines	
  from	
  samples	
  of	
  their	
  behavior.	
  
IEEE	
  ToC,	
  1972.	
  
•  Dallmeier,	
  Knopp,	
  Mallon,	
  Fraser,	
  Hack,	
  Zeller.	
  Automa7cally	
  Genera7ng	
  Test	
  Cases	
  for	
  
Specifica7on	
  Mining.	
  TSE,	
  2012	
  
•  Dallmeier,	
  Lindig,	
  Wasylkowski,	
  Zeller:	
  Mining	
  Object	
  Behavior	
  with	
  ADABU.	
  WODA	
  2006	
  
•  Dallmeier,	
  Zeller,	
  Meyer.	
  Genera7ng	
  fixes	
  from	
  object	
  behavior	
  anomalies.	
  ASE.	
  2009	
  
•  Ernst,	
  Cockrell,	
  Griswold,	
  Notkin.	
  Dynamically	
  Discovering	
  Likely	
  Program	
  Invariants	
  to	
  Support	
  
Program	
  Evolu7on.	
  IEEE	
  TSE	
  2001	
  
•  Gabel,	
  Su.	
  Tes7ng	
  Mined	
  Specifica7ons.	
  ESEC/FSE.	
  2012.	
  
•  Hangal,	
  Lam.	
  Tracking	
  down	
  so?ware	
  bugs	
  using	
  automa7c	
  anomaly	
  detec7on.	
  ICSE	
  2002	
  
•  Jin,	
  Orso,	
  Xie.	
  Automated	
  Behavioral	
  Regression	
  Tes7ng.	
  ICST.	
  2010.	
  
•  Jones,	
  Harrold,	
  Stasko.	
  Visualiza7on	
  of	
  Test	
  Informa7on	
  to	
  Assist	
  Fault	
  Localiza7on.	
  ICSE,	
  2002.	
  
•  Lo,	
  Khoo,	
  Liu.	
  Mining	
  temporal	
  rules	
  for	
  so?ware	
  maintenance.	
  JSME,	
  2008	
  
•  Lo,	
  Mariani,	
  Pezzè.	
  Automa7c	
  Steering	
  of	
  Behavioral	
  Model	
  Inference.	
  ESEC/FSE	
  2009	
  
•  Lo,	
  Mariani,	
  Santoro,	
  Learning	
  extended	
  FSA	
  from	
  So?ware:	
  An	
  Empirical	
  Assessment.	
  	
  JSS,	
  2012	
  
•  Lorenzoli,	
  Mariani,	
  Pezzè.	
  Automa7c	
  Genera7on	
  of	
  So?ware	
  Behavioral	
  Models,	
  ICSE,	
  2008	
  
	
  
References	
  
•  Marcheso,	
  Tonella,	
  Ricca:	
  State-­‐Based	
  Tes7ng	
  of	
  Ajax	
  Web	
  Applica7ons.	
  ICST	
  2008	
  
•  Mariani,	
  Marcheso,	
  Nguyen,	
  Tonella.	
  Revolu7on:	
  Automa7c	
  evolu7on	
  of	
  mined	
  specifica7ons.	
  
ISSRE.	
  2012	
  
•  Mariani,	
  Papagiannakis,	
  Pezzè.	
  Compa7bility	
  and	
  regression	
  tes7ng	
  of	
  COTS-­‐Component-­‐based	
  
so?ware.	
  ICSE.	
  2007.	
  
•  Mariani,	
  Pastore.	
  Automa7c	
  Iden7fica7on	
  of	
  Failure	
  Causes	
  in	
  System	
  Logs,	
  ISSRE,	
  2008	
  
•  Mariani,	
  Pastore,	
  Pezzè.	
  Dynamic	
  Analysis	
  for	
  Diagnosing	
  Integra7on	
  Faults.	
  TSE,	
  2011.	
  
•  Nguyen,	
  Kapur,	
  Weimer,	
  Forrest:	
  Using	
  dynamic	
  analysis	
  to	
  discover	
  polynomial	
  and	
  array	
  
invariants.	
  ICSE	
  2012:	
  683-­‐693	
  
•  Nugyen,	
  Marcheso,	
  Tonella.	
  Automated	
  Oracles:	
  An	
  Empirical	
  Study	
  on	
  Cost	
  and	
  Effec7veness,	
  
ESEC/FSE,	
  2013	
  
•  Pastore,	
  Mariani,	
  Goffi,	
  Oriol,	
  Wahler.	
  Dynamic	
  Analysis	
  of	
  Upgrades	
  in	
  C/C++	
  So?ware.	
  ISSRE,	
  
2012.	
  
•  Pradel,	
  Gross.	
  Leveraging	
  test	
  genera7on	
  and	
  specifica7on	
  mining	
  for	
  automated	
  bug	
  detec7on	
  
without	
  false	
  posi7ves.	
  ICSE	
  2012	
  
•  Raz,	
  Koopman,	
  Shaw.	
  Seman7c	
  anomaly	
  detec7on	
  in	
  online	
  data	
  sources.	
  ICSE.	
  2002.	
  
•  Yang,	
  Evans,	
  Bhardwaj,	
  Bhat,	
  Das.	
  Perracosa:	
  mining	
  temporal	
  API	
  Rules	
  from	
  Imperfect	
  Traces.	
  
ICSE.	
  2006	
  
•  Yilmaz,	
  Paradkar,	
  Williams.	
  Time	
  will	
  tell.	
  ICSE.	
  2010.	
  
	
  
Ques7ons?	
  
ICSE	
  DOCTORAL	
  SYMPOSIUM	
  
S.C.	
  Cheung	
  and	
  L.	
  Mariani	
  
Submission	
  deadline:	
  Nov	
  22,	
  2013	
  
No7fica7on:	
  Feb	
  17,	
  2014	
  
Camera	
  Ready:	
  Mar	
  14,	
  2014	
  
Event	
  Date:	
  Jun	
  3,	
  2014	
  

More Related Content

What's hot

Intro to RxJava/RxAndroid - GDG Munich Android
Intro to RxJava/RxAndroid - GDG Munich AndroidIntro to RxJava/RxAndroid - GDG Munich Android
Intro to RxJava/RxAndroid - GDG Munich AndroidEgor Andreevich
 
Akka.NET streams and reactive streams
Akka.NET streams and reactive streamsAkka.NET streams and reactive streams
Akka.NET streams and reactive streamsBartosz Sypytkowski
 
OSMC 2012 | Neues in Nagios 4.0 by Andreas Ericsson
OSMC 2012 | Neues in Nagios 4.0 by Andreas EricssonOSMC 2012 | Neues in Nagios 4.0 by Andreas Ericsson
OSMC 2012 | Neues in Nagios 4.0 by Andreas EricssonNETWAYS
 
Behind modern concurrency primitives
Behind modern concurrency primitivesBehind modern concurrency primitives
Behind modern concurrency primitivesBartosz Sypytkowski
 
Functional Reactive Programming in Clojurescript
Functional Reactive Programming in ClojurescriptFunctional Reactive Programming in Clojurescript
Functional Reactive Programming in ClojurescriptLeonardo Borges
 
Reactive Programming no Android
Reactive Programming no AndroidReactive Programming no Android
Reactive Programming no AndroidGuilherme Branco
 
apidays LIVE Australia - Strangling the monolith with a reactive GraphQL gate...
apidays LIVE Australia - Strangling the monolith with a reactive GraphQL gate...apidays LIVE Australia - Strangling the monolith with a reactive GraphQL gate...
apidays LIVE Australia - Strangling the monolith with a reactive GraphQL gate...apidays
 
Reactive Programming on Android
Reactive Programming on AndroidReactive Programming on Android
Reactive Programming on AndroidGuilherme Branco
 
Practical RxJava for Android
Practical RxJava for AndroidPractical RxJava for Android
Practical RxJava for AndroidTomáš Kypta
 
Java libraries you can't afford to miss
Java libraries you can't afford to missJava libraries you can't afford to miss
Java libraries you can't afford to missAndres Almiray
 
Reactive Programming on Android - RxAndroid - RxJava
Reactive Programming on Android - RxAndroid - RxJavaReactive Programming on Android - RxAndroid - RxJava
Reactive Programming on Android - RxAndroid - RxJavaAli Muzaffar
 
Concurrency Concepts in Java
Concurrency Concepts in JavaConcurrency Concepts in Java
Concurrency Concepts in JavaDoug Hawkins
 
Don't Make Android Bad... Again
Don't Make Android Bad... AgainDon't Make Android Bad... Again
Don't Make Android Bad... AgainPedro Vicente
 
Non Blocking I/O for Everyone with RxJava
Non Blocking I/O for Everyone with RxJavaNon Blocking I/O for Everyone with RxJava
Non Blocking I/O for Everyone with RxJavaFrank Lyaruu
 
The Road To Reactive with RxJava JEEConf 2016
The Road To Reactive with RxJava JEEConf 2016The Road To Reactive with RxJava JEEConf 2016
The Road To Reactive with RxJava JEEConf 2016Frank Lyaruu
 
The Future of Futures - A Talk About Java 8 CompletableFutures
The Future of Futures - A Talk About Java 8 CompletableFuturesThe Future of Futures - A Talk About Java 8 CompletableFutures
The Future of Futures - A Talk About Java 8 CompletableFuturesHaim Yadid
 
Android architecture component - FbCircleDev Yogyakarta Indonesia
Android architecture component - FbCircleDev Yogyakarta IndonesiaAndroid architecture component - FbCircleDev Yogyakarta Indonesia
Android architecture component - FbCircleDev Yogyakarta IndonesiaPratama Nur Wijaya
 

What's hot (20)

Intro to RxJava/RxAndroid - GDG Munich Android
Intro to RxJava/RxAndroid - GDG Munich AndroidIntro to RxJava/RxAndroid - GDG Munich Android
Intro to RxJava/RxAndroid - GDG Munich Android
 
Akka.NET streams and reactive streams
Akka.NET streams and reactive streamsAkka.NET streams and reactive streams
Akka.NET streams and reactive streams
 
OSMC 2012 | Neues in Nagios 4.0 by Andreas Ericsson
OSMC 2012 | Neues in Nagios 4.0 by Andreas EricssonOSMC 2012 | Neues in Nagios 4.0 by Andreas Ericsson
OSMC 2012 | Neues in Nagios 4.0 by Andreas Ericsson
 
Behind modern concurrency primitives
Behind modern concurrency primitivesBehind modern concurrency primitives
Behind modern concurrency primitives
 
Functional Reactive Programming in Clojurescript
Functional Reactive Programming in ClojurescriptFunctional Reactive Programming in Clojurescript
Functional Reactive Programming in Clojurescript
 
Javantura v2 - All Together Now - making Groovy and Scala sing together - Din...
Javantura v2 - All Together Now - making Groovy and Scala sing together - Din...Javantura v2 - All Together Now - making Groovy and Scala sing together - Din...
Javantura v2 - All Together Now - making Groovy and Scala sing together - Din...
 
Reactive Programming no Android
Reactive Programming no AndroidReactive Programming no Android
Reactive Programming no Android
 
apidays LIVE Australia - Strangling the monolith with a reactive GraphQL gate...
apidays LIVE Australia - Strangling the monolith with a reactive GraphQL gate...apidays LIVE Australia - Strangling the monolith with a reactive GraphQL gate...
apidays LIVE Australia - Strangling the monolith with a reactive GraphQL gate...
 
Reactive Programming on Android
Reactive Programming on AndroidReactive Programming on Android
Reactive Programming on Android
 
Practical RxJava for Android
Practical RxJava for AndroidPractical RxJava for Android
Practical RxJava for Android
 
Java libraries you can't afford to miss
Java libraries you can't afford to missJava libraries you can't afford to miss
Java libraries you can't afford to miss
 
Reactive Programming on Android - RxAndroid - RxJava
Reactive Programming on Android - RxAndroid - RxJavaReactive Programming on Android - RxAndroid - RxJava
Reactive Programming on Android - RxAndroid - RxJava
 
Javantura v2 - Making Java web-apps Groovy - Franjo Žilić
Javantura v2 - Making Java web-apps Groovy - Franjo ŽilićJavantura v2 - Making Java web-apps Groovy - Franjo Žilić
Javantura v2 - Making Java web-apps Groovy - Franjo Žilić
 
Concurrency Concepts in Java
Concurrency Concepts in JavaConcurrency Concepts in Java
Concurrency Concepts in Java
 
Don't Make Android Bad... Again
Don't Make Android Bad... AgainDon't Make Android Bad... Again
Don't Make Android Bad... Again
 
Non Blocking I/O for Everyone with RxJava
Non Blocking I/O for Everyone with RxJavaNon Blocking I/O for Everyone with RxJava
Non Blocking I/O for Everyone with RxJava
 
RxJava on Android
RxJava on AndroidRxJava on Android
RxJava on Android
 
The Road To Reactive with RxJava JEEConf 2016
The Road To Reactive with RxJava JEEConf 2016The Road To Reactive with RxJava JEEConf 2016
The Road To Reactive with RxJava JEEConf 2016
 
The Future of Futures - A Talk About Java 8 CompletableFutures
The Future of Futures - A Talk About Java 8 CompletableFuturesThe Future of Futures - A Talk About Java 8 CompletableFutures
The Future of Futures - A Talk About Java 8 CompletableFutures
 
Android architecture component - FbCircleDev Yogyakarta Indonesia
Android architecture component - FbCircleDev Yogyakarta IndonesiaAndroid architecture component - FbCircleDev Yogyakarta Indonesia
Android architecture component - FbCircleDev Yogyakarta Indonesia
 

Viewers also liked

Consenso europeo ktr adultos uci
Consenso europeo ktr adultos uci Consenso europeo ktr adultos uci
Consenso europeo ktr adultos uci felixvidal
 
25 conception-revu_par_ac
25  conception-revu_par_ac25  conception-revu_par_ac
25 conception-revu_par_acRiadh Harizi
 
« Cri d’alarme » de l’Académie des sciences sur le financement de la recherche
« Cri d’alarme » de l’Académie des sciences sur le financement de la recherche« Cri d’alarme » de l’Académie des sciences sur le financement de la recherche
« Cri d’alarme » de l’Académie des sciences sur le financement de la rechercheCollectif PAPERA
 
Motion control batiment
Motion control batimentMotion control batiment
Motion control batimentGimélec
 
Guidepourlecontroleetlaudit 2006-00607-1-
Guidepourlecontroleetlaudit 2006-00607-1-Guidepourlecontroleetlaudit 2006-00607-1-
Guidepourlecontroleetlaudit 2006-00607-1-Gimélec
 
Unit 12 Future Technologies
Unit 12 Future TechnologiesUnit 12 Future Technologies
Unit 12 Future TechnologiesSonia Osuna
 

Viewers also liked (7)

Consenso europeo ktr adultos uci
Consenso europeo ktr adultos uci Consenso europeo ktr adultos uci
Consenso europeo ktr adultos uci
 
25 conception-revu_par_ac
25  conception-revu_par_ac25  conception-revu_par_ac
25 conception-revu_par_ac
 
« Cri d’alarme » de l’Académie des sciences sur le financement de la recherche
« Cri d’alarme » de l’Académie des sciences sur le financement de la recherche« Cri d’alarme » de l’Académie des sciences sur le financement de la recherche
« Cri d’alarme » de l’Académie des sciences sur le financement de la recherche
 
Motion control batiment
Motion control batimentMotion control batiment
Motion control batiment
 
Guidepourlecontroleetlaudit 2006-00607-1-
Guidepourlecontroleetlaudit 2006-00607-1-Guidepourlecontroleetlaudit 2006-00607-1-
Guidepourlecontroleetlaudit 2006-00607-1-
 
Unit 12 Future Technologies
Unit 12 Future TechnologiesUnit 12 Future Technologies
Unit 12 Future Technologies
 
Notions de emarketing
Notions de emarketingNotions de emarketing
Notions de emarketing
 

Similar to TAROT2013 Testing School - Leonardo Mariani presentation

Using Formal Methods to Create Instruction Set Architectures
Using Formal Methods to Create Instruction Set ArchitecturesUsing Formal Methods to Create Instruction Set Architectures
Using Formal Methods to Create Instruction Set ArchitecturesDVClub
 
Mining Software Archives to Support Software Development
Mining Software Archives to Support Software DevelopmentMining Software Archives to Support Software Development
Mining Software Archives to Support Software DevelopmentThomas Zimmermann
 
Mining Branch-Time Scenarios From Execution Logs
Mining Branch-Time Scenarios From Execution LogsMining Branch-Time Scenarios From Execution Logs
Mining Branch-Time Scenarios From Execution LogsDirk Fahland
 
Hierarchical free monads and software design in fp
Hierarchical free monads and software design in fpHierarchical free monads and software design in fp
Hierarchical free monads and software design in fpAlexander Granin
 
GeeCON 2017 - TestContainers. Integration testing without the hassle
GeeCON 2017 - TestContainers. Integration testing without the hassleGeeCON 2017 - TestContainers. Integration testing without the hassle
GeeCON 2017 - TestContainers. Integration testing without the hassleAnton Arhipov
 
Concurrency in Eclipse: Best Practices and Gotchas
Concurrency in Eclipse: Best Practices and GotchasConcurrency in Eclipse: Best Practices and Gotchas
Concurrency in Eclipse: Best Practices and Gotchasamccullo
 
Event driven javascript
Event driven javascriptEvent driven javascript
Event driven javascriptFrancesca1980
 
Event driven javascript
Event driven javascriptEvent driven javascript
Event driven javascriptFrancesca1980
 
Eric Lafortune - The Jack and Jill build system
Eric Lafortune - The Jack and Jill build systemEric Lafortune - The Jack and Jill build system
Eric Lafortune - The Jack and Jill build systemGuardSquare
 
Fact-Based Monitoring
Fact-Based MonitoringFact-Based Monitoring
Fact-Based MonitoringDatadog
 
Fact based monitoring
Fact based monitoringFact based monitoring
Fact based monitoringDatadog
 
Evan Schultz - Angular Summit - 2016
Evan Schultz - Angular Summit - 2016Evan Schultz - Angular Summit - 2016
Evan Schultz - Angular Summit - 2016Evan Schultz
 
Test First, Refresh Second: Web App TDD in Grails
Test First, Refresh Second: Web App TDD in GrailsTest First, Refresh Second: Web App TDD in Grails
Test First, Refresh Second: Web App TDD in GrailsTim Berglund
 
Test First Refresh Second: Test-Driven Development in Grails
Test First Refresh Second: Test-Driven Development in GrailsTest First Refresh Second: Test-Driven Development in Grails
Test First Refresh Second: Test-Driven Development in GrailsTim Berglund
 
Android RenderScript on LLVM
Android RenderScript on LLVMAndroid RenderScript on LLVM
Android RenderScript on LLVMJohn Lee
 
Reactive programming with examples
Reactive programming with examplesReactive programming with examples
Reactive programming with examplesPeter Lawrey
 
Ft10 de smet
Ft10 de smetFt10 de smet
Ft10 de smetnkaluva
 
Clojure: Practical functional approach on JVM
Clojure: Practical functional approach on JVMClojure: Practical functional approach on JVM
Clojure: Practical functional approach on JVMsunng87
 

Similar to TAROT2013 Testing School - Leonardo Mariani presentation (20)

Using Formal Methods to Create Instruction Set Architectures
Using Formal Methods to Create Instruction Set ArchitecturesUsing Formal Methods to Create Instruction Set Architectures
Using Formal Methods to Create Instruction Set Architectures
 
Mining Software Archives to Support Software Development
Mining Software Archives to Support Software DevelopmentMining Software Archives to Support Software Development
Mining Software Archives to Support Software Development
 
Mining Branch-Time Scenarios From Execution Logs
Mining Branch-Time Scenarios From Execution LogsMining Branch-Time Scenarios From Execution Logs
Mining Branch-Time Scenarios From Execution Logs
 
Hierarchical free monads and software design in fp
Hierarchical free monads and software design in fpHierarchical free monads and software design in fp
Hierarchical free monads and software design in fp
 
GeeCON 2017 - TestContainers. Integration testing without the hassle
GeeCON 2017 - TestContainers. Integration testing without the hassleGeeCON 2017 - TestContainers. Integration testing without the hassle
GeeCON 2017 - TestContainers. Integration testing without the hassle
 
Concurrency in Eclipse: Best Practices and Gotchas
Concurrency in Eclipse: Best Practices and GotchasConcurrency in Eclipse: Best Practices and Gotchas
Concurrency in Eclipse: Best Practices and Gotchas
 
Event driven javascript
Event driven javascriptEvent driven javascript
Event driven javascript
 
Event driven javascript
Event driven javascriptEvent driven javascript
Event driven javascript
 
Eric Lafortune - The Jack and Jill build system
Eric Lafortune - The Jack and Jill build systemEric Lafortune - The Jack and Jill build system
Eric Lafortune - The Jack and Jill build system
 
Fact-Based Monitoring
Fact-Based MonitoringFact-Based Monitoring
Fact-Based Monitoring
 
Fact based monitoring
Fact based monitoringFact based monitoring
Fact based monitoring
 
Evan Schultz - Angular Summit - 2016
Evan Schultz - Angular Summit - 2016Evan Schultz - Angular Summit - 2016
Evan Schultz - Angular Summit - 2016
 
Test First, Refresh Second: Web App TDD in Grails
Test First, Refresh Second: Web App TDD in GrailsTest First, Refresh Second: Web App TDD in Grails
Test First, Refresh Second: Web App TDD in Grails
 
Test First Refresh Second: Test-Driven Development in Grails
Test First Refresh Second: Test-Driven Development in GrailsTest First Refresh Second: Test-Driven Development in Grails
Test First Refresh Second: Test-Driven Development in Grails
 
Android RenderScript on LLVM
Android RenderScript on LLVMAndroid RenderScript on LLVM
Android RenderScript on LLVM
 
Reactive programming with examples
Reactive programming with examplesReactive programming with examples
Reactive programming with examples
 
JDBC Part - 2
JDBC Part - 2JDBC Part - 2
JDBC Part - 2
 
Ft10 de smet
Ft10 de smetFt10 de smet
Ft10 de smet
 
Clojure: Practical functional approach on JVM
Clojure: Practical functional approach on JVMClojure: Practical functional approach on JVM
Clojure: Practical functional approach on JVM
 
From * to Symfony2
From * to Symfony2From * to Symfony2
From * to Symfony2
 

More from Henry Muccini

Human Behaviour Centred Design
Human Behaviour Centred Design Human Behaviour Centred Design
Human Behaviour Centred Design Henry Muccini
 
How cultural heritage, cyber-physical spaces, and software engineering can wo...
How cultural heritage, cyber-physical spaces, and software engineering can wo...How cultural heritage, cyber-physical spaces, and software engineering can wo...
How cultural heritage, cyber-physical spaces, and software engineering can wo...Henry Muccini
 
La gestione dell’utenza numerosa - dalle Segreterie, ai Musei, alle Segreterie
La gestione dell’utenza numerosa - dalle Segreterie, ai Musei, alle SegreterieLa gestione dell’utenza numerosa - dalle Segreterie, ai Musei, alle Segreterie
La gestione dell’utenza numerosa - dalle Segreterie, ai Musei, alle SegreterieHenry Muccini
 
Turismo 4.0: l'ICT a supporto del turismo sostenibile
Turismo 4.0: l'ICT a supporto del turismo sostenibileTurismo 4.0: l'ICT a supporto del turismo sostenibile
Turismo 4.0: l'ICT a supporto del turismo sostenibileHenry Muccini
 
Sustainable Tourism - IoT and crowd management
Sustainable Tourism - IoT and crowd managementSustainable Tourism - IoT and crowd management
Sustainable Tourism - IoT and crowd managementHenry Muccini
 
Software Engineering at the age of the Internet of Things
Software Engineering at the age of the Internet of ThingsSoftware Engineering at the age of the Internet of Things
Software Engineering at the age of the Internet of ThingsHenry Muccini
 
The influence of Group Decision Making on Architecture Design Decisions
The influence of Group Decision Making on Architecture Design DecisionsThe influence of Group Decision Making on Architecture Design Decisions
The influence of Group Decision Making on Architecture Design DecisionsHenry Muccini
 
An IoT Software Architecture for an Evacuable Building Architecture
An IoT Software Architecture for an Evacuable Building ArchitectureAn IoT Software Architecture for an Evacuable Building Architecture
An IoT Software Architecture for an Evacuable Building ArchitectureHenry Muccini
 
Web Engineering L8: User-centered Design (8/8)
Web Engineering L8: User-centered Design (8/8)Web Engineering L8: User-centered Design (8/8)
Web Engineering L8: User-centered Design (8/8)Henry Muccini
 
Web Engineering L7: Sequence Diagrams and Design Decisions (7/8)
Web Engineering L7: Sequence Diagrams and Design Decisions (7/8)Web Engineering L7: Sequence Diagrams and Design Decisions (7/8)
Web Engineering L7: Sequence Diagrams and Design Decisions (7/8)Henry Muccini
 
Web Engineering L6: Software Architecture for the Web (6/8)
Web Engineering L6: Software Architecture for the Web (6/8)Web Engineering L6: Software Architecture for the Web (6/8)
Web Engineering L6: Software Architecture for the Web (6/8)Henry Muccini
 
Web Engineering L5: Content Model (5/8)
Web Engineering L5: Content Model (5/8)Web Engineering L5: Content Model (5/8)
Web Engineering L5: Content Model (5/8)Henry Muccini
 
Web Engineering L3: Project Planning (3/8)
Web Engineering L3: Project Planning (3/8)Web Engineering L3: Project Planning (3/8)
Web Engineering L3: Project Planning (3/8)Henry Muccini
 
Web Engineering L2: Requirements Elicitation for the Web (2/8)
Web Engineering L2: Requirements Elicitation for the Web (2/8)Web Engineering L2: Requirements Elicitation for the Web (2/8)
Web Engineering L2: Requirements Elicitation for the Web (2/8)Henry Muccini
 
Web Engineering L1: introduction to Web Engineering (1/8)
Web Engineering L1: introduction to Web Engineering (1/8)Web Engineering L1: introduction to Web Engineering (1/8)
Web Engineering L1: introduction to Web Engineering (1/8)Henry Muccini
 
Web Engineering L4: Requirements and Planning in concrete (4/8)
Web Engineering L4: Requirements and Planning in concrete (4/8)Web Engineering L4: Requirements and Planning in concrete (4/8)
Web Engineering L4: Requirements and Planning in concrete (4/8)Henry Muccini
 
Collaborative aspects of Decision Making and its impact on Sustainability
Collaborative aspects of Decision Making and its impact on SustainabilityCollaborative aspects of Decision Making and its impact on Sustainability
Collaborative aspects of Decision Making and its impact on SustainabilityHenry Muccini
 
Engineering Cyber Physical Spaces
Engineering Cyber Physical SpacesEngineering Cyber Physical Spaces
Engineering Cyber Physical SpacesHenry Muccini
 
I progetti UnivAq-UFFIZI, INCIPICT, e  CUSPIS
I progetti UnivAq-UFFIZI, INCIPICT, e  CUSPISI progetti UnivAq-UFFIZI, INCIPICT, e  CUSPIS
I progetti UnivAq-UFFIZI, INCIPICT, e  CUSPISHenry Muccini
 
Exploring the Temporal Aspects of Software Architecture
Exploring the Temporal Aspects of Software ArchitectureExploring the Temporal Aspects of Software Architecture
Exploring the Temporal Aspects of Software ArchitectureHenry Muccini
 

More from Henry Muccini (20)

Human Behaviour Centred Design
Human Behaviour Centred Design Human Behaviour Centred Design
Human Behaviour Centred Design
 
How cultural heritage, cyber-physical spaces, and software engineering can wo...
How cultural heritage, cyber-physical spaces, and software engineering can wo...How cultural heritage, cyber-physical spaces, and software engineering can wo...
How cultural heritage, cyber-physical spaces, and software engineering can wo...
 
La gestione dell’utenza numerosa - dalle Segreterie, ai Musei, alle Segreterie
La gestione dell’utenza numerosa - dalle Segreterie, ai Musei, alle SegreterieLa gestione dell’utenza numerosa - dalle Segreterie, ai Musei, alle Segreterie
La gestione dell’utenza numerosa - dalle Segreterie, ai Musei, alle Segreterie
 
Turismo 4.0: l'ICT a supporto del turismo sostenibile
Turismo 4.0: l'ICT a supporto del turismo sostenibileTurismo 4.0: l'ICT a supporto del turismo sostenibile
Turismo 4.0: l'ICT a supporto del turismo sostenibile
 
Sustainable Tourism - IoT and crowd management
Sustainable Tourism - IoT and crowd managementSustainable Tourism - IoT and crowd management
Sustainable Tourism - IoT and crowd management
 
Software Engineering at the age of the Internet of Things
Software Engineering at the age of the Internet of ThingsSoftware Engineering at the age of the Internet of Things
Software Engineering at the age of the Internet of Things
 
The influence of Group Decision Making on Architecture Design Decisions
The influence of Group Decision Making on Architecture Design DecisionsThe influence of Group Decision Making on Architecture Design Decisions
The influence of Group Decision Making on Architecture Design Decisions
 
An IoT Software Architecture for an Evacuable Building Architecture
An IoT Software Architecture for an Evacuable Building ArchitectureAn IoT Software Architecture for an Evacuable Building Architecture
An IoT Software Architecture for an Evacuable Building Architecture
 
Web Engineering L8: User-centered Design (8/8)
Web Engineering L8: User-centered Design (8/8)Web Engineering L8: User-centered Design (8/8)
Web Engineering L8: User-centered Design (8/8)
 
Web Engineering L7: Sequence Diagrams and Design Decisions (7/8)
Web Engineering L7: Sequence Diagrams and Design Decisions (7/8)Web Engineering L7: Sequence Diagrams and Design Decisions (7/8)
Web Engineering L7: Sequence Diagrams and Design Decisions (7/8)
 
Web Engineering L6: Software Architecture for the Web (6/8)
Web Engineering L6: Software Architecture for the Web (6/8)Web Engineering L6: Software Architecture for the Web (6/8)
Web Engineering L6: Software Architecture for the Web (6/8)
 
Web Engineering L5: Content Model (5/8)
Web Engineering L5: Content Model (5/8)Web Engineering L5: Content Model (5/8)
Web Engineering L5: Content Model (5/8)
 
Web Engineering L3: Project Planning (3/8)
Web Engineering L3: Project Planning (3/8)Web Engineering L3: Project Planning (3/8)
Web Engineering L3: Project Planning (3/8)
 
Web Engineering L2: Requirements Elicitation for the Web (2/8)
Web Engineering L2: Requirements Elicitation for the Web (2/8)Web Engineering L2: Requirements Elicitation for the Web (2/8)
Web Engineering L2: Requirements Elicitation for the Web (2/8)
 
Web Engineering L1: introduction to Web Engineering (1/8)
Web Engineering L1: introduction to Web Engineering (1/8)Web Engineering L1: introduction to Web Engineering (1/8)
Web Engineering L1: introduction to Web Engineering (1/8)
 
Web Engineering L4: Requirements and Planning in concrete (4/8)
Web Engineering L4: Requirements and Planning in concrete (4/8)Web Engineering L4: Requirements and Planning in concrete (4/8)
Web Engineering L4: Requirements and Planning in concrete (4/8)
 
Collaborative aspects of Decision Making and its impact on Sustainability
Collaborative aspects of Decision Making and its impact on SustainabilityCollaborative aspects of Decision Making and its impact on Sustainability
Collaborative aspects of Decision Making and its impact on Sustainability
 
Engineering Cyber Physical Spaces
Engineering Cyber Physical SpacesEngineering Cyber Physical Spaces
Engineering Cyber Physical Spaces
 
I progetti UnivAq-UFFIZI, INCIPICT, e  CUSPIS
I progetti UnivAq-UFFIZI, INCIPICT, e  CUSPISI progetti UnivAq-UFFIZI, INCIPICT, e  CUSPIS
I progetti UnivAq-UFFIZI, INCIPICT, e  CUSPIS
 
Exploring the Temporal Aspects of Software Architecture
Exploring the Temporal Aspects of Software ArchitectureExploring the Temporal Aspects of Software Architecture
Exploring the Temporal Aspects of Software Architecture
 

Recently uploaded

UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7DianaGray10
 
Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™Adtran
 
20230202 - Introduction to tis-py
20230202 - Introduction to tis-py20230202 - Introduction to tis-py
20230202 - Introduction to tis-pyJamie (Taka) Wang
 
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDEADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDELiveplex
 
Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.YounusS2
 
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCostKubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCostMatt Ray
 
Nanopower In Semiconductor Industry.pdf
Nanopower  In Semiconductor Industry.pdfNanopower  In Semiconductor Industry.pdf
Nanopower In Semiconductor Industry.pdfPedro Manuel
 
Machine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfMachine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfAijun Zhang
 
UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6DianaGray10
 
UiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation DevelopersUiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation DevelopersUiPathCommunity
 
NIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 WorkshopNIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 WorkshopBachir Benyammi
 
UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8DianaGray10
 
UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1DianaGray10
 
Comparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioComparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioChristian Posta
 
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration WorkflowsIgniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration WorkflowsSafe Software
 
Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Commit University
 
COMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a WebsiteCOMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a Websitedgelyza
 
Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1DianaGray10
 
VoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBXVoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBXTarek Kalaji
 
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfJamie (Taka) Wang
 

Recently uploaded (20)

UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7
 
Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™
 
20230202 - Introduction to tis-py
20230202 - Introduction to tis-py20230202 - Introduction to tis-py
20230202 - Introduction to tis-py
 
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDEADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
 
Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.
 
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCostKubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
 
Nanopower In Semiconductor Industry.pdf
Nanopower  In Semiconductor Industry.pdfNanopower  In Semiconductor Industry.pdf
Nanopower In Semiconductor Industry.pdf
 
Machine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfMachine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdf
 
UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6
 
UiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation DevelopersUiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation Developers
 
NIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 WorkshopNIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 Workshop
 
UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8
 
UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1
 
Comparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioComparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and Istio
 
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration WorkflowsIgniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
 
Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)
 
COMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a WebsiteCOMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a Website
 
Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1
 
VoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBXVoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBX
 
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
 

TAROT2013 Testing School - Leonardo Mariani presentation

  • 1. Automated  Failure  Analysis  in   Absence  of  Specifica7on   Leonardo  Mariani   University  of  Milano  Bicocca   mariani@disco.unimib.it  
  • 2. Analysis  of  So?ware  Behaviors   Analysis  of  So?ware  Failures   (semi-­‐)automa7cally,  when   no  specifica7on  is  available  
  • 3. Automated  Debugging   Fault  localiza7on:  search  for  the  fault  loca7on        (e.g.,  search  for  faulty  program  statements      in  the  program  source  code)   Failure  analysis  (aka  anomaly  detec7on):  search      for  failure  causes        (e.g.,  search  for  erroneous  events  in  the        execu7on  space)  
  • 4. Example   •  known  issue  in  Tomcat  6.0.0  (to  6.0.9)   8  
  • 5. Fault  Localiza7on  (output  obtained  with  Tarantula)   public  class  ChipsListener  implements  ServletContextListener  {          public  ChipsListener()      {  }                  public  void  contextIni7alized(ServletContextEvent  evt)  {              ServletContext  context  =  evt.getServletContext();              JspApplica7onContext  jspContext  =    JspFactory.getDefaultFactory().getJspApplica7onContext(context);                      jspContext.addELResolver(new  ChipsELResolver());            }       n  even  if  related  to  the  bug,  the  bug  is  not  there!   n  this  is  not  the  best  ranked  piece  of  code   n  why  is  this  fragment  of  code  relevant?   9   [Jones  et  al.  Visualiza7on  of  Test  Informa7on  to  Assist  Fault  Localiza7on.  ICSE,  2002.]  
  • 6. Failure  Analysis  (output  obtained  with  BCT)   •  javax.servlet.jsp.JspFactory.getDefaultFactory()  returned  null   •  then   org.apache.tomcat.u7l.modeler.Registry.unregisterComponent(javax.managem ent.ObjectName)  invoked   org.apache.catalina.session.ManagerBase.postDeregister()     •  and   org.apache.tomcat.u7l.modeler.Registry.unregisterComponent(javax.mana gement.ObjectName)    invoked   org.apache.catalina.loader.WebappLoader.postDeregister()     11   [Mariani,  Pastore,  Pezzè.  Dynamic  Analysis  for  Diagnosing  Integra7on  Faults.  TSE,  2011.]   Tomcat  failed  because  
  • 7. How  to  iden7fy  the   events  responsible  for   a  failure  when  no  spec   is  available?  
  • 8. Specifica?on  Mining:   Learn  specifica?ons   from  actual  execu?ons  
  • 9. Specifica7on  mining  for   failure  analysis:  learn   the  regular  behavior   (specifica(on)…  to   detect  anomalous   events  
  • 11. 1 2 a 3 c 4 5 d f eb Applica7on   Traces   Model   Analysis   What  models  can  we  mine?   What  traces  do  we  need?   How  can  we  analyze  a  failure?   Trace   Failure  
  • 12. What  models  can  we  mine?  
  • 13. generate  a   model  that   represents  the   actual  behavior   from  samples   X=-­‐2   X=-­‐9   X=0   X=5   X=7   …   Actual  Behavior   -­‐10<X<10   Mined  behavior   -­‐10<X<10   Specifica7on  Mining  
  • 14. Model  genera7on  is  imprecise…   Over-­‐Generaliza7on   Over-­‐Restric7on   Over-­‐Generaliza7on   and   Over-­‐Restric7on   Actual  Behavior   -­‐10<X<10   Mined  specifica7on   X  >  -­‐100   Mined  specifica7on   -­‐5  <  X  <  5   Mined  specifica7on   -­‐100  <  X  <  5  
  • 15. Real  Specifica7on   -­‐10<X<10   Mined  Specifica7on   -­‐100<X<5   Specifica7on  Mining:  Models  Used  as  Specifica7ons   X  =  100   Correctly  rejected   behavior   X  =  1   Correctly  accepted   behavior   X  =  7   Erroneously  rejected   behavior   X  =  -­‐50   Erroneously  accepted   behavior  
  • 16. Models   1 2 a 3 c 4 5 d f eb x > 0 Full  Ordering  of  Events   Data  Values   Ordering  of  Events  +   Data  Values   Par7al  Ordering  of   Events   open  =>  close  
  • 17. MODELS  THAT  REPRESENT  THE   FULL  ORDERING  OF  EVENTS  
  • 18. Mining  of  Finite  State  Models   •  Trace-­‐based  mining   – State-­‐based  merging   – Behavior-­‐based  merging   •  State-­‐based  mining   Total  =  0   Elem  =  0   Total  =  3   Elem  =  1   Total  =  5   Elem  =  2   Total  =  0   Elem  =  0   onLoad   add   add   empty   onLoad   add   add   empty  
  • 19. kTail  (state-­‐based  merging)   TRACES   PTA   FSA   [Biermann  and  Feldman.  On  the  synthesis  of   finite  state  machines  from  samples  of  their   behavior.  IEEE  ToC,  1972.  ]   a   a   a   b   c   a   b   c   a   a   b   c   a   a   a   a   a   c  
  • 20. Build  the  PTA   TRACES   PTA   a   a   a   b   c   a   b   c   a   a   b   c   a   a   a   a   a   c  
  • 21. k=2   2-­‐future(2)  =  {aa,ab,bc}   2-­‐future(5)  =  {aa,  bc}   2-­‐future(11)  =  {}   2-­‐future(8)  =  {c}   …   2  FUTURES  
  • 22. 2-­‐future(8)  =  {c}   2-­‐future(12)  =  {c}  
  • 24. 2-­‐future(11)  =  {}   2-­‐future(13)  =  {}  
  • 26. 2-­‐future(2)  =  {aa,  ab,  bc}   2-­‐future(3)  =  {aa,  ab,  bc}  
  • 30. K   Over-­‐ restric?on   Over-­‐ generaliza?on   Small  K   Big  K   The  Parameter  K  
  • 31. kBehavior  (behavior-­‐based  merging)   •  incremental   Traces   login   home   checkMsg   logout   [Mariani,  Pastore,  Pezzè.  Dynamic  Analysis  for  Diagnosing  Integra7on  Faults.  TSE,  2011.]  
  • 32. kBehavior  (behavior-­‐based  merging)   Traces   login   home   checkMsg   logout   login   home   checkMsg   7meout  
  • 33. kBehavior  (behavior-­‐based  merging)   Traces   login   home   checkMsg   logout   login   home   checkMsg   7meout  
  • 34. kBehavior  (behavior-­‐based  merging)   Traces   login   home   checkMsg   logout   login   home   checkMsg   7meout   login   home   checkMsg  watchVideo  home   checkMsg   logout   K  =  min  length  of   matched  behavior  
  • 35. kBehavior  (behavior-­‐based  merging)   Traces   login   home   checkMsg   logout   login   home   checkMsg   7meout   login   home   checkMsg  watchVideo  home   checkMsg   logout  
  • 36. kBehavior  (behavior-­‐based  merging)   Traces   login   home   checkMsg   logout   login   home   checkMsg   7meout   login   home   checkMsg  watchVideo  home   checkMsg   logout   login   home   checkMsg   read   home   checkMsg   logout  reply  
  • 37. kBehavior  (behavior-­‐based  merging)   Traces   login   home   checkMsg   logout   login   home   checkMsg   7meout   login   home   checkMsg  watchVideo  home   checkMsg   logout   login   home   checkMsg   read   home   checkMsg   logout  reply  
  • 38. The  Parameter  K   •  K  determines  the  degree  of  generaliza7on   •  Empirically,  behavior-­‐based  merging   generates  models  that  are  more  general  than   state-­‐based  merging  [Lo  et  al.,  JSS,  2012]   State-­‐based  merging   behavior-­‐based   merging  
  • 39. State-­‐Based  Inference  of  FSM  Models   Total  =  0   Elem  =  0   Total  =  3   Elem  =  1   Total  =  5   Elem  =  2   Total  =  0   Elem  =  0   onLoad   add   add   empty   <0   ==0   >0   Abstrac7on   func7on   <0   =0   >0   <0   =0   >0   <0   =0   >0   <init>   Total  ==  0   Elem==0   Total  >  0   Elem  >  0   onLoad   add   add   empty   [Dallmeier,  Lindig,  Wasylkowski,  Zeller:  Mining  Object  Behavior  with  ADABU.  WODA  2006]   [Marcheso,  Tonella,  Ricca:  State-­‐Based  Tes7ng  of  Ajax  Web  Applica7ons.  ICST  2008]   [Mariani,  Marcheso,  Nguyen,  Tonella.  Revolu7on:  Automa7c  evolu7on  of  mined   specifica7ons.  ISSRE.  2012]  
  • 40. The  Abstrac7on  Func7on   •  Quality  of  the  final  model  influenced  by   – Completeness  of  the  state  informa7on  that  is   traced   – The  kind  of  abstrac7on  implemented  by  the   abstrac7on  func7on       numElements   numDis7nctElements   numElements  VS   <0        =0        >0   <-­‐1        =-­‐1        =0        =1      >1  VS  
  • 41. MODELS  THAT  REPRESENT  THE   VALUES  OF  VARIABLES  
  • 42. Program  for  eLectronic  Commerce   Execu7ons   Trace  with   variable  values   totalCost  unitCost   4   3 1 7 …   8   3 8 14 …  
  • 43. totalCost  unitCost   4   3 1 7 …   8   3 8 14 …   _  +  _  =_   _  <  _   _=_   _  >  0   unitCost  =  totalCost   unitCost  <  totalCost   unitCost  <=  totalCost   unitCost  +  totalCost  >  unitCost   …   preserve  expressions  with   perfect  confidence   unitCost  <=  totalCost   unitCost  +  totalCost  >  unitCost   totalCost  >  0     1  >  _   remove  redundant  proper(es   unitCost  <=  totalCost   totalCost  >    0   Daikon  in  a   nutshell   Traces   Template  Expressions   Candidate  Expressions   [Ernst,  Cockrell,   Griswold,  Notkin.   Dynamically   Discovering  Likely   Program  Invariants   to  Support  Program   Evolu7on.  IEEE  TSE   2001]  
  • 44. The  Set  of  Template  Expressions   •  Expressiveness  depends  on  the  template  expressions   •  More  template  expr  =>  more  candidate  expressions   =>  higher  computa7onl  cost   •  Recently  defined  an  approach  to  deal  with   polynomial  and  array  expressions  [Nguyen  et  al.  ICSE  2012]   _  +  _  =_   _  <  _   _=_   _  >  0  1  >  _  
  • 45. MODELS  THAT  REPRESENT  PARTIAL   ORDERING  OF  EVENTS  
  • 46. Mine  Temporal  Rules   Traces   Template  Rules   <pre>   <post>   CONFIDENCE  AND  SUPPORT  THRESHOLDS   Temporal  Rules   [Lo,  Khoo,  Liu.  Mining  temporal   rules  for  so?ware  maintenance.   JSME,  2008]   [Yang,  Evans,  Bhardwaj,  Bhat,  Das.   Perracosa:  mining  temporal  API   Rules  from  Imperfect  Traces.  ICSE.   2006]   …   start   open   close   stop   start   load   stop   start   open   close   stop   begin   end  
  • 47. Mine  Temporal  Rules   start   open   close   stop   start   load   stop   start   open   close   stop   Traces   Template  Rules   <pre>   <post>   CONFIDENCE  AND  SUPPORT  THRESHOLDS   Temporal  Rules   CONFIDENCE  OF  A  RULE   #  traces  rule  holds   …   #traces    pre  applies   start   open   has  67%  confidence   begin   end  
  • 48. Mine  Temporal  Rules   Traces   Template  Rules   <pre>   <post>   CONFIDENCE  AND  SUPPORT  THRESHOLDS   Temporal  Rules   Conf  =  100%   Supp  =  20%   start   stop   close  open   …   …   SUPPORT  OF  A  RULE   #  traces  rule  holds   #traces   start   open   has  50%  support   start   open   close   stop   start   load   stop   start   open   close   stop   begin   end  
  • 49. Template  Rules   •  Expressiveness  depends  on  the  template  rules   •  Confidence  and  Support  for  tuning  the   technique  wrt  imperfect  traces  
  • 50. Steering  FSA  Models  with  Temporal  Rules   kTail  with  k=2   Overgeneraliza7on  problem:   -­‐  locally,  it  sounds  to  be  a   good  decision   -­‐  globally,  it  generates   anomalous  behaviors  
  • 51. Idea:  mine  global  proper7es,  exploit   them  when  taking  decisions  locally   Traces   Mine  Temporal  Rules   Build  PTA   openFile   closeFile   closeConn  connDB   Apply  kTail  (e.g.   with  k=2)   BUT   prevent  state   merges  that  violate   temporal  rules   (LOCAL  DECISIONS)   [Lo,  Mariani,  Pezzè.  Automa7c  Steering  of  Behavioral  Model  Inference.  ESEC/FSE  2009]   GLOBAL  PROPERTIES  
  • 53. FSA  With  Annota7ons  to  Represent   •  Constraints  on  parameter  values    
  • 54. Traces  With  Parameter  Values   addItem   addItem   buy   qt=1   unitCost=1   totalCost=1   qt=2   unitCost=3   totalCost=6   Traces   …  
  • 55. GKTail   merging  similar  traces   EFSM   [Lorenzoli,  Mariani,  Pezzè.   Automa7c  Genera7on  of   So?ware  Behavioral  Models,   ICSE,  2008]   0 1 2 3 4 5 6 m1 0≤x≤15 m1 x=1 m2 x=0 y=0 x=y m3 z={’IT’,’UK’} m1 x=0 m2 x=0 0≤y≤20 8 9 10 11 12 13 m3 z=’UK’ m3 z=’UK’ m2 x=0 y=3 m3 z=’UK’ m1 x=0 m2 x=0 y=15 22 23 24 25 26 27 m1 x=0 m1 x=1 m2 x=0 y=0 x=y m3 z=’IT’ m3 z=’IT’ m2 x=0 y=30 PTA   deriving  guards   X=0   Y=0,   X=0   Y=15,   …   X=0,     0≤Y≤20   DAIKON   m2   m2  
  • 56. GKTail   merging  similar  traces   deriving  guards   EFSM   [Lorenzoli,  Mariani,  Pezzè.   Automa7c  Genera7on  of   So?ware  Behavioral  Models,   ICSE,  2008]   0 1 2 3 4 5 6 m1 0≤x≤15 m1 x=1 m2 x=0 y=0 x=y m3 z={’IT’,’UK’} m1 x=0 m2 x=0 0≤y≤20 8 9 10 11 12 13 m3 z=’UK’ m3 z=’UK’ m2 x=0 y=3 m3 z=’UK’ m1 x=0 m2 x=0 y=15 22 23 24 25 26 27 m1 x=0 m1 x=1 m2 x=0 y=0 x=y m3 z=’IT’ m3 z=’IT’ m2 x=0 y=30 PTA   X=0   Y=0,   X=0   Y=15,   …   X=0,     0≤Y≤20   DAIKON   m2   m2  
  • 57. Mining  Specifica7ons:  Different   Models  for  Different  Aspects   Different  models  can  capture  different  types  of  anomalous  behaviors  
  • 58. Specifica7on  Mining  Tools   •  Synop7c   – hsp://code.google.com/p/synop7c/   •  Perracosa   – hsp://www.cs.virginia.edu/perracosa/   •  Adabu   – hsp://www.st.cs.uni-­‐saarland.de/models/ adabu.php3   •  KLFA   – hsp://www.lta.disco.unimib.it/tools/klfa/  
  • 59. (When)  Are  Mined  Models   Precise  Enough?  
  • 60. Empirical  Studies   -­‐  complexity  -­‐   Length  of  traces/Noise/Number     of  different  events  in  the  traces     Mining  simple  FSA  Mining  extended  FSA   Mining  temporal  rules   Mining  constraints   [Lo,  Mariani,  Santoro,  Learning  extended  FSA  from  So?ware:  An  Empirical  Assessment.     JSS,  2012]   [Yang,  Evans,  Bhardwaj,  Bhat,  Das.  Perracosa:  mining  temporal  API  Rules  from  Imperfect   Traces.  ICSE.  2006]   [Nugyen,  Marcheso,  Tonella.  Automated  Oracles:  An  Empirical  Study  on  Cost  and   Effec7veness,  ESEC/FSE,  2013]  
  • 61. Empirical  Studies   -­‐  sensi7vity  -­‐   Capture  small  differences   Mining  simple  FSA  Mining  extended  FSA   Mining  temporal  rules   Mining  constraints   Capture  major  differences   FSA  good  to  analy7cally   capture  the  behavior  of  small   units  (e.g.,  components)   Temporal  rules  and  constraints   good  to  capture  some  behaviors   in  rela7vely  big  applica7ons  
  • 62. Quality  of  Models  vs  Number  of  Traces   Component/ API/Method   Traces   Ideal  Model   -­‐  Transi7on  coverage   enough  for  mining  good   FSAs  [Lo,  JSS,  2012]   -­‐  Addi7onal  tests  can  be   generated  to  improve   models  [Dallmeier,  TSE,   2012]    
  • 63. Quality  of  Models  vs  Number  of  Traces   Applica7on   Traces   Ideal  Model   -­‐  Good  FSAs  hard  to   mine     -­‐  Other  models:  several   traces  necessary  for   par7cularly  complex   cases  [Nguyen,  ESEC/FSE,   2013]  
  • 64. Take  Home  About  Specifica7on  Mining   •  Think  to  your  research  area   – If  you  need  models  and   specifica7ons…   – …and  you  do  not  have  any,   – but  you  have  a  way  of   execu7ng  your  so?ware   – Specifica7on  Mining  could   an  op7on!  
  • 66. 1 2 a 3 c 4 5 d f eb Applica7on   Traces   Model   Analysis   Trace   Failure  
  • 67. Failure  Analysis  Based  on  Specifica7on   Mining   •  Analysis  of  (Field  and  Regression)  Failures   – BCT    [Mariani  et  al.  Dynamic  Analysis  for  Diagnosing  Integra7on  Faults.  TSE,  2011.]   •  Analysis  of  Regression  Failures   – Radar   [Pastore  et  al.  Dynamic  Analysis  of  Upgrades  in  C/C++  So?ware.  ISSRE,  2012.]   •  Produce  Descrip7ve  Reports   – AVA   [Babenko  et  al.  AVA:  automated  interpreta7on  of  dynamically  detected   anomalies.  ISSTA,  2009.]  
  • 68. BCT:  a  technique  for  automated   iden7fica7on  of  func7onal  faults   1.  Capturing  Behavioral  Data   •  Monitoring  Component  Execu7ons   •  Capturing  Run-­‐Time  Informa7on   2.  Dis(lling  Behavioral  Models   •  I/O  Models   •  Interac7on  Models   3.  Failure  Analysis   •  Regression  failures   •  Field  failure  analysis   addItem(c) c.getTotalCost>=c.getCost c.getQuantity>0 ... getCart() Cart newCart() addItem(c) getCart() ... Catalog imageDB Failure!! unexpected   interac?on!   unexpected   interac?on!   unexpected   interac?on!   unexpected   value!   unexpected   value!   System
  • 69. Capturing  IO  Data   purchase   …   checkCreden7als(usr,  taskType)   makeOrder(cart,  usr)   checkCreden7als(User,  TaskType)   usr.name  =  “Leonardo”   usr.address.streetName  =  “viale  Certosa”   …   taskType.type  =  2   …     usr.name  =  “Carlo”   usr.address.streetName  =  “viale  Manzoni”   …   taskType.type  =  1   …   execu7on  1   execu7on  2   makeOrder(cart,  usr)   purchase(cart)  
  • 70. Capturing  Interac7on  Data   101   purchase   …   checkCreden7als(usr,  taskType)   makeOrder(cart,  usr)   purchase()   Auth.checkCreden7als(User,TaskType)   Shop.makeOrder(Cart,  User)   …     Auth.checkCreden7als(User,  TaskType)   Shop.makeOrder(Cart,  User)   …     Auth.checkCreden7als(User,  TaskType)   Logger.logPermissionDenied(User,  TaskType)   …   execu7on  1   execu7on  2   execu7on  3  
  • 71. Dis7lling  Behavioural  Models   I/O  Data   Interac7on  Data   Daikon   kBehavior   I/O  Model   Interac7on  Model   x != null method1   method2   method3   method4   I/O  and  Interac7on   Models   I/O  and  Interac7on   Models   I/O  and  Interac7on   Models   I/O  and  Interac7on   Models  
  • 72. Run-­‐Time  Verifica(on  and  Failure   Analysis   System Failure!! unexpected   interac?on!   unexpected   interac?on!   unexpected   interac?on!   unexpected   value!   unexpected   value!   103  
  • 73. Filtering   •  Re-­‐execute  tests  and  remove  anomalies  detected  in  both  passing   and  failing  tests     Regression  tes7ng   •  Country==US violated  by  passing  regression  tests  because  the  new   version  of  the  applica7on  is  available    outside  US   •  Viola7ons  of  this  property  can  be  ignored     Field  failures   •  date==20/3/2013 spurious property violated  by  passing   regression  tests   •  Viola7ons  of  this  property  can  be  ignored   Remaining  anomalies  are  re-­‐arranged  according  to  likely  cause-­‐ effects  
  • 74. Run-­‐Time  Verifica(on  and  Failure  Analysis:   Rela(ng  Anomalies   105   start   ini?alize   getValue   I  ini7alize  the   next  component   I  need  a  proper   value  for   ini7aliza7on   I  do  not  know   the  value!!!  I   return  null null  is  not  a   proper  value!  I   return  an  excep7on   we  have  an   excep7on!!!  Let’s  try  to   terminate  safely
  • 75. Run-­‐Time  Verifica(on  and  Failure  Analysis:   Rela(ng  Failures     106   start   ini?alize   getValue   undo   log   closeConnec?on   log  the  event   and  close  the   connec7on
  • 76. Run-­‐Time  Verifica(on  and  Failure  Analysis:   Rela(ng  Failures     107   start   ini?alize   getValue   undo   log   closeConnec?on   one  anomaly  is  the  cause  of  many  others!   return  null  value   throw  excep?on   call  undo   early  close  the   connec?on  
  • 77. Capturing   Clusters   Dynamic  call  tree  for  the  Tomcat  case  study   ini?al  anomaly  graph   109  
  • 78. Capturing   Clusters   Dynamic  call  tree  for  the  Tomcat  case  study   ini?al  anomaly  graph   110  
  • 79. Capturing   Clusters   Dynamic  call  tree  for  the  Tomcat  case  study   ini?al  anomaly  graph   111  
  • 80. Output  Obtained  with  BCT  for  the   Tomcat  Failure   ON  EXIT  from  javax.servlet.jsp.JspFactory.getDefaultFactory()   MODEL  VIOLATED  returnValue  !=  null  =  false   FROM   org.apache.tomcat.u7l.modeler.Registry.unregisterComponent(javax.man agement.ObjectName)     UNEXPECTED  CALL  TO   org.apache.catalina.session.ManagerBase.postDeregister()     FROM     org.apache.tomcat.u7l.modeler.Registry.unregisterComponent(ja vax.management.ObjectName)       UNEXPECTED  CALL  TO     org.apache.catalina.loader.WebappLoader.postDeregister()     112  
  • 81. Eclipse  3.3  Anomaly  Graph   113  
  • 82. Capturing   Clusters   Dynamic  call  tree  for  the  Tomcat  case  study   ini?al  anomaly  graph   114  
  • 83. Eclipse  3.3  Anomaly  Graph   115  
  • 84. Stopping  Criterion   edges  with  weights  greater  than   this  value  are  removed   cohesion(graph)   116   cohesion(graph)  =  avg(cohesion  (CCs))   cohesion(CC)  =  avg(weight  edges)     smaller  value  ==  beser  cohesion    
  • 85. Resul7ng  Graph   …   •   the  components  are   inspected  from  the  biggest  to   the  smallest     •   the  first  two  graphs  are   enough  to  explain  the   problem!   117  
  • 86. Improvements   Make  the  analysis   specific  to  the   type  of   considered  faults   Radar:  failure  analysis   of  regression  problems   Produce  outputs   that  beser  explain   the  reason  of  the   failure   AVA:  automa7c   analysis  of  anomalies  
  • 87. V1   V2   chainItems.size > 0 24   25   27   28   29   31   availableQty   32   34   36   37   28   28   33   Radar  in  a  Nutshell   TEST  SUITE  TEST  SUITE   TRACE   Failed  because   initItems()  has  not   been  invoked  and   chainItem.size  =0  
  • 89. File.open File.write File.close sortFile File.delete File.open   …sortFile   File.delete  File.write   Anomaly  Detec7on  with  FSA  
  • 90. File.open File.write File.close sortFile File.delete File.open …sortFile File.deleteFile.write Anomaly  Detec7on  with  FSA  
  • 91. File.open File.write File.close sortFile File.delete File.open …sortFile File.deleteFile.write Anomaly  Detec7on  
  • 92. File.open File.write File.close sortFile File.delete File.open …sortFile File.deleteFile.write Anomaly  Detec7on   Should the path of the file be the problem? Should the sorting be the problem? Should the content of the file be the problem? May be the file has not been closed!!! …
  • 94. File.open File.write File.close sortFile File.delete File.open …sortFile File.deleteFile.write Anomaly interpretation: Missing event: File.close The  file  has  not  been  closed!  
  • 95. AVA  =  Compare  actual  and   expected  behaviors  (represented   with  a  FSA)  to  produce  informa7ve   outputs  
  • 96. Automata  Viola7ons  Analysis  -­‐  AVA   Iden7fy   Model   Viola7ons   ____   _____   _____   _____   _____   _____   _____       FSA   Trace   Branches Tails Final States
  • 97. Automata  Viola7ons  Analysis  -­‐  AVA   Iden7fy   Model   Viola7ons   Iden7fy   Basic   Interpreta7ons   ____   _____   _____   _____   _____   _____   _____       FSA   Trace   Deletions Insertions Replacements Terminations Branches Tails Final States
  • 98. Automata  Viola7ons  Analysis  -­‐  AVA   Iden7fy   Model   Viola7ons   Iden7fy   Basic   Interpreta7ons   Iden7fy   Composite   Interpreta7ons   ____   _____   _____   _____   _____   _____   _____       FSA   Trace   Deletions Insertions Replacements Terminations Branches Tails Final States Anticipations Postponements Swaps
  • 99. File.open File.write File.close sortFile File.delete File.open   File.close  sortFile   File.delete  File.write   ε File.close Iden7fy  Model  Viola7ons     with  FSA  Extensions   fsa’  =  kBehavior(fsa,t)   extensions  =  diff(fsa,fsa’)  
  • 100. File.open File.write File.close sortFile File.delete File.open File.closesortFile File.deleteFile.write ε Iden7fy  Basic  Interpreta7ons   File.open File.write File.open sortFile File.delete sortFile File.delete File.close File.close File.write File.write sortFile File.deleteFile.close
  • 101. File.open sortFile File.deleteFile.write Iden7fy  Basic  Interpreta7ons   Observed sequence Expected sequences Compare using alignment algorithms File.open File.write File.open sortFile File.delete sortFile File.delete File.close File.close File.write File.write sortFile File.deleteFile.close
  • 102. File.open File.write File.close sortFile File.delete File.open File.write - sortFile File.delete Expected   Observed   The  applica7on  failed   because  File.close  has  not   been  executed     sortFile  is  anomalous  BETTER  THAN  
  • 103. Different  interpreta7ons  can  be  discovered   with  different  alignment  strategies   Deletion EV1 EV2 EV3 EV4 EV5 Insertion EV1 EV2 EV3 EV EV4 EV5 Replacements EV1 EV2 EV3 EV EV5 Terminations EV1 EV2 EV3 Anticipations EV1 EV5 EV2 EV3 EV4 Postponements EV1 EV3 EV4 EV5 EV2 Swap EV1 EV5 EV3 EV4 EV2 SIMPLE  INTERPRETATIONS   COMPOSITE  INTERPRETATIONS  
  • 104. Tools   •  BCT   – hsp://www.lta.disco.unimib.it/tools/bct/   •  Radar   – hsp://www.lta.disco.unimib.it/tools/radar/   •  AVA   – hsp://www.lta.disco.unimib.it/tools/ava/  
  • 106. Specifica?on  Mining     can  be  used  to  enable   several  analyses  in  the   frequent  case  no   specifica?on  is  available  
  • 107. One  interes7ng   domain  is  failure   analysis  
  • 108. Specifica7on  mining  has  been   experienced  in  several  other  contexts   Other  Failure  analysis  approaches   [Hangal,  Lam.  Tracking  down  so?ware  bugs  using  automa7c  anomaly  detec7on.  ICSE   2002]   [Yilmaz,  Paradkar,  Williams.  Time  will  tell.  ICSE.  2010]     Regression  tes?ng   •  Behavioral  Regression  Tes7ng    [Jin,  Orso,  Xie.  Automated  Behavioral  Regression  Tes7ng.  ICST.  2010.]   •  Compa7bility  Tes7ng    [Mariani,  Papagiannakis,  Pezzè.  Compa7bility  and  regression  tes7ng  of  COTS-­‐ Component-­‐based  so?ware.  ICSE.  2007.]     Combined  with  Sta?c  analysis   [Pradel,  Gross.  Leveraging  test  genera7on  and  specifica7on  mining  for  automated  bug   detec7on  without  false  posi7ves.  ICSE  2012]   [Dallmeier,  Zeller,  Meyer.  Genera7ng  fixes  from  object  behavior  anomalies.  ASE.  2009]  
  • 109. But  be  careful  with  “posi7ve”   behavioral  anomalies  
  • 110. References   •  Babenko,  Mariani,  Pastore.  AVA:  automated  interpreta7on  of  dynamically  detected  anomalies.   ISSTA,  2009   •  Biermann  and  Feldman.  On  the  synthesis  of  finite  state  machines  from  samples  of  their  behavior.   IEEE  ToC,  1972.   •  Dallmeier,  Knopp,  Mallon,  Fraser,  Hack,  Zeller.  Automa7cally  Genera7ng  Test  Cases  for   Specifica7on  Mining.  TSE,  2012   •  Dallmeier,  Lindig,  Wasylkowski,  Zeller:  Mining  Object  Behavior  with  ADABU.  WODA  2006   •  Dallmeier,  Zeller,  Meyer.  Genera7ng  fixes  from  object  behavior  anomalies.  ASE.  2009   •  Ernst,  Cockrell,  Griswold,  Notkin.  Dynamically  Discovering  Likely  Program  Invariants  to  Support   Program  Evolu7on.  IEEE  TSE  2001   •  Gabel,  Su.  Tes7ng  Mined  Specifica7ons.  ESEC/FSE.  2012.   •  Hangal,  Lam.  Tracking  down  so?ware  bugs  using  automa7c  anomaly  detec7on.  ICSE  2002   •  Jin,  Orso,  Xie.  Automated  Behavioral  Regression  Tes7ng.  ICST.  2010.   •  Jones,  Harrold,  Stasko.  Visualiza7on  of  Test  Informa7on  to  Assist  Fault  Localiza7on.  ICSE,  2002.   •  Lo,  Khoo,  Liu.  Mining  temporal  rules  for  so?ware  maintenance.  JSME,  2008   •  Lo,  Mariani,  Pezzè.  Automa7c  Steering  of  Behavioral  Model  Inference.  ESEC/FSE  2009   •  Lo,  Mariani,  Santoro,  Learning  extended  FSA  from  So?ware:  An  Empirical  Assessment.    JSS,  2012   •  Lorenzoli,  Mariani,  Pezzè.  Automa7c  Genera7on  of  So?ware  Behavioral  Models,  ICSE,  2008    
  • 111. References   •  Marcheso,  Tonella,  Ricca:  State-­‐Based  Tes7ng  of  Ajax  Web  Applica7ons.  ICST  2008   •  Mariani,  Marcheso,  Nguyen,  Tonella.  Revolu7on:  Automa7c  evolu7on  of  mined  specifica7ons.   ISSRE.  2012   •  Mariani,  Papagiannakis,  Pezzè.  Compa7bility  and  regression  tes7ng  of  COTS-­‐Component-­‐based   so?ware.  ICSE.  2007.   •  Mariani,  Pastore.  Automa7c  Iden7fica7on  of  Failure  Causes  in  System  Logs,  ISSRE,  2008   •  Mariani,  Pastore,  Pezzè.  Dynamic  Analysis  for  Diagnosing  Integra7on  Faults.  TSE,  2011.   •  Nguyen,  Kapur,  Weimer,  Forrest:  Using  dynamic  analysis  to  discover  polynomial  and  array   invariants.  ICSE  2012:  683-­‐693   •  Nugyen,  Marcheso,  Tonella.  Automated  Oracles:  An  Empirical  Study  on  Cost  and  Effec7veness,   ESEC/FSE,  2013   •  Pastore,  Mariani,  Goffi,  Oriol,  Wahler.  Dynamic  Analysis  of  Upgrades  in  C/C++  So?ware.  ISSRE,   2012.   •  Pradel,  Gross.  Leveraging  test  genera7on  and  specifica7on  mining  for  automated  bug  detec7on   without  false  posi7ves.  ICSE  2012   •  Raz,  Koopman,  Shaw.  Seman7c  anomaly  detec7on  in  online  data  sources.  ICSE.  2002.   •  Yang,  Evans,  Bhardwaj,  Bhat,  Das.  Perracosa:  mining  temporal  API  Rules  from  Imperfect  Traces.   ICSE.  2006   •  Yilmaz,  Paradkar,  Williams.  Time  will  tell.  ICSE.  2010.    
  • 113. ICSE  DOCTORAL  SYMPOSIUM   S.C.  Cheung  and  L.  Mariani   Submission  deadline:  Nov  22,  2013   No7fica7on:  Feb  17,  2014   Camera  Ready:  Mar  14,  2014   Event  Date:  Jun  3,  2014