SlideShare a Scribd company logo
1 of 16
Download to read offline
Metrics are usually computed at a low level:
           classes, methods, …




/ W&I / MDSE        23-4-2012 PAGE 0
Multitude of data values obscures a general
      picture of the system maintainability




/W&I / MDSE         23-4-2012 PAGE 1
That we are actually interested in!




/W&I / MDSE          23-4-2012 PAGE 2
You Can't Control the Unfamiliar:
A Study on the Relations
Between Aggregation
Techniques for Software Metrics

 Bogdan Vasilescu
 Alexander Serebrenik
 Mark van den Brand
Two kinds of aggregation
Same metrics, different                Same artifact, different
artifacts                              metrics




/W&I / MDSE         23-4-2012 PAGE 4
Various techniques can be
   found in the literature
Same metrics, different                  Traditional: mean,
artifacts                                median, sum, …



                                       Econometric
                                       inequality indices:
                                       Gini, Theil, Hoover,
                                       Kolm, Atkinson




/W&I / MDSE         23-4-2012 PAGE 5
Various techniques can be
   found in the literature
Same metrics, different                  Traditional: mean,
artifacts                                median, sum, …

                                            Which
                                       aggregation
                                       Econometric
                                         technique
                                       inequality indices:
                                       Gini, Theil, Hoover,
                                         should we
                                       Kolm, Atkinson
                                              use?


/W&I / MDSE         23-4-2012 PAGE 6
Questions

      1. Which and to what extent do the different
         aggregation techniques agree?

      2. What is the nature of the relation between the
         various aggregation techniques?

      3. How does the correlation coefficient change as the
         systems evolve?




/W&I / MDSE            23-4-2012 PAGE 7
Qualitas Corpus 20101126
     • Qualitas Corpus 20101126r, 106 systems
     • FitJava v1.1, 2 packages, 2240 SLOC
     • NetBeans v6.9.1, 3373 packages 1890536 SLOC.




/W&I / MDSE          23-4-2012 PAGE 8
1) Agreement between diff techniques

      • Agreement:
          • Aggregation: Class SLOC  Package
          • Techniques agree if they rank the packages similarly




        We use rank-based correlation coefficient: Kendall’s 


/W&I / MDSE               23-4-2012 PAGE 9
1) Agreement: different inequality indices?
     • Gini, Theil, Hoover, Atkinson – agree
         • aggregates obtained convey the same information
         • Kolm does not!




/W&I / MDSE              23-4-2012 PAGE 10
1) Agreement: traditional and ineq indices?

    • mean
        • Kolm: strong (0,8) and statistically significant (92%)
        • median, standard deviation, and variance


    • sum
        • does not correlate with any other aggregation technique




/W&I / MDSE               23-4-2012 PAGE 11
2) Nature of the relation: Typical patterns




   • Theil is known to be more           • Linear relation with a “fat”
     sensitive to the rich                 head
   • Theil increases faster
     when Gini increases
/W&I / MDSE          23-4-2012 PAGE 12
Which aggregation technique? (1)

      • Theil, Hoover, Gini and Atkinson agree
          • Any can be chosen from the correlation point of view


      • Some might be “better” in each specific case
          • easy to interpret: Gini  [0,1]
          • provide additional insights: Theil (explanation)
          • negative values: Gini, Hoover
               − affects the domain!
          • sensitive for high values: Theil, Atkinson
          • deviations from uniformity: Gini, Hoover




/ W&I / MDSE                  23-4-2012 PAGE 13
Which aggregation technique? (2)

      • Kolm and mean agree
          • Kolm is reliable for skewed distributions
            − better alternative (“by no means”)
          • Not in the paper:
            − agreement observed for NOC
            − but not for DIT!




/ W&I / MDSE               23-4-2012 PAGE 14
Conclusions




/W&I / MDSE         23-4-2012 PAGE 15

More Related Content

Similar to ICSM 2011

An overview on data mining designed for imbalanced datasets
An overview on data mining designed for imbalanced datasetsAn overview on data mining designed for imbalanced datasets
An overview on data mining designed for imbalanced datasetseSAT Journals
 
Icam present silvia
Icam present silviaIcam present silvia
Icam present silviasferna
 
Types of cost ppt @ mba 2009
Types of cost ppt @ mba 2009Types of cost ppt @ mba 2009
Types of cost ppt @ mba 2009Babasab Patil
 
2013 wp evidence-creation-through-knowledge-integration_ho
2013 wp evidence-creation-through-knowledge-integration_ho2013 wp evidence-creation-through-knowledge-integration_ho
2013 wp evidence-creation-through-knowledge-integration_hoWenny Ho
 
Doctoral dissertation presentation 2014
Doctoral dissertation presentation 2014Doctoral dissertation presentation 2014
Doctoral dissertation presentation 2014Xabier Alberdi
 
Technical Consortium Theme 3: Monitoring and evaluation for enhanced resilien...
Technical Consortium Theme 3: Monitoring and evaluation for enhanced resilien...Technical Consortium Theme 3: Monitoring and evaluation for enhanced resilien...
Technical Consortium Theme 3: Monitoring and evaluation for enhanced resilien...ILRI
 
MAD 09 Presentation 2022 MMMM.ppt DR CHARLES (1).ppt
MAD 09 Presentation 2022 MMMM.ppt DR CHARLES (1).pptMAD 09 Presentation 2022 MMMM.ppt DR CHARLES (1).ppt
MAD 09 Presentation 2022 MMMM.ppt DR CHARLES (1).pptEmmanuelStevenKoroma
 
Services for Later Life: Are we any closer to integrating health and social c...
Services for Later Life: Are we any closer to integrating health and social c...Services for Later Life: Are we any closer to integrating health and social c...
Services for Later Life: Are we any closer to integrating health and social c...Age UK
 
Wildfire-Workshop-Presentation-Roundtable-Discussion
Wildfire-Workshop-Presentation-Roundtable-DiscussionWildfire-Workshop-Presentation-Roundtable-Discussion
Wildfire-Workshop-Presentation-Roundtable-DiscussionOECD Environment
 
Using MCDA for HTA, Opportunities, Challenges and Possible Ways Forward
Using MCDA for HTA, Opportunities, Challenges and Possible Ways ForwardUsing MCDA for HTA, Opportunities, Challenges and Possible Ways Forward
Using MCDA for HTA, Opportunities, Challenges and Possible Ways ForwardOffice of Health Economics
 
LENGUAGE TESTING (II Bimestre Abril Agosto 2011)
LENGUAGE TESTING (II Bimestre Abril Agosto 2011)LENGUAGE TESTING (II Bimestre Abril Agosto 2011)
LENGUAGE TESTING (II Bimestre Abril Agosto 2011)Videoconferencias UTPL
 
Constructing A Knowledge Economy Composite Indicator With Imprecise Data
Constructing A Knowledge Economy Composite Indicator With Imprecise DataConstructing A Knowledge Economy Composite Indicator With Imprecise Data
Constructing A Knowledge Economy Composite Indicator With Imprecise Dataronicky
 
The modernisation of European higher education through paying attention to th...
The modernisation of European higher education through paying attention to th...The modernisation of European higher education through paying attention to th...
The modernisation of European higher education through paying attention to th...EduSkills OECD
 
Effective simplicity rotterdam
Effective simplicity rotterdamEffective simplicity rotterdam
Effective simplicity rotterdamsaskiamenkel
 

Similar to ICSM 2011 (20)

An overview on data mining designed for imbalanced datasets
An overview on data mining designed for imbalanced datasetsAn overview on data mining designed for imbalanced datasets
An overview on data mining designed for imbalanced datasets
 
Icam present silvia
Icam present silviaIcam present silvia
Icam present silvia
 
Types of cost ppt @ mba 2009
Types of cost ppt @ mba 2009Types of cost ppt @ mba 2009
Types of cost ppt @ mba 2009
 
2013 wp evidence-creation-through-knowledge-integration_ho
2013 wp evidence-creation-through-knowledge-integration_ho2013 wp evidence-creation-through-knowledge-integration_ho
2013 wp evidence-creation-through-knowledge-integration_ho
 
Doctoral dissertation presentation 2014
Doctoral dissertation presentation 2014Doctoral dissertation presentation 2014
Doctoral dissertation presentation 2014
 
Technical Consortium Theme 3: Monitoring and evaluation for enhanced resilien...
Technical Consortium Theme 3: Monitoring and evaluation for enhanced resilien...Technical Consortium Theme 3: Monitoring and evaluation for enhanced resilien...
Technical Consortium Theme 3: Monitoring and evaluation for enhanced resilien...
 
CBU, Economics
CBU, EconomicsCBU, Economics
CBU, Economics
 
Mirage hh dakar_december2011_0
Mirage hh dakar_december2011_0Mirage hh dakar_december2011_0
Mirage hh dakar_december2011_0
 
Group discussion
Group discussionGroup discussion
Group discussion
 
MAD 09 Presentation 2022 MMMM.ppt DR CHARLES (1).ppt
MAD 09 Presentation 2022 MMMM.ppt DR CHARLES (1).pptMAD 09 Presentation 2022 MMMM.ppt DR CHARLES (1).ppt
MAD 09 Presentation 2022 MMMM.ppt DR CHARLES (1).ppt
 
49417273
4941727349417273
49417273
 
Services for Later Life: Are we any closer to integrating health and social c...
Services for Later Life: Are we any closer to integrating health and social c...Services for Later Life: Are we any closer to integrating health and social c...
Services for Later Life: Are we any closer to integrating health and social c...
 
Wildfire-Workshop-Presentation-Roundtable-Discussion
Wildfire-Workshop-Presentation-Roundtable-DiscussionWildfire-Workshop-Presentation-Roundtable-Discussion
Wildfire-Workshop-Presentation-Roundtable-Discussion
 
Day 2. Co chairs reflections
Day 2. Co chairs reflectionsDay 2. Co chairs reflections
Day 2. Co chairs reflections
 
Using MCDA for HTA, Opportunities, Challenges and Possible Ways Forward
Using MCDA for HTA, Opportunities, Challenges and Possible Ways ForwardUsing MCDA for HTA, Opportunities, Challenges and Possible Ways Forward
Using MCDA for HTA, Opportunities, Challenges and Possible Ways Forward
 
LENGUAGE TESTING (II Bimestre Abril Agosto 2011)
LENGUAGE TESTING (II Bimestre Abril Agosto 2011)LENGUAGE TESTING (II Bimestre Abril Agosto 2011)
LENGUAGE TESTING (II Bimestre Abril Agosto 2011)
 
Constructing A Knowledge Economy Composite Indicator With Imprecise Data
Constructing A Knowledge Economy Composite Indicator With Imprecise DataConstructing A Knowledge Economy Composite Indicator With Imprecise Data
Constructing A Knowledge Economy Composite Indicator With Imprecise Data
 
The modernisation of European higher education through paying attention to th...
The modernisation of European higher education through paying attention to th...The modernisation of European higher education through paying attention to th...
The modernisation of European higher education through paying attention to th...
 
Healthcare Econometrics
Healthcare EconometricsHealthcare Econometrics
Healthcare Econometrics
 
Effective simplicity rotterdam
Effective simplicity rotterdamEffective simplicity rotterdam
Effective simplicity rotterdam
 

More from Bogdan Vasilescu

More from Bogdan Vasilescu (7)

Benevol 2012
Benevol 2012Benevol 2012
Benevol 2012
 
SOS-Evol 2012
SOS-Evol 2012SOS-Evol 2012
SOS-Evol 2012
 
IPA Spring Days 2012
IPA Spring Days 2012IPA Spring Days 2012
IPA Spring Days 2012
 
Sattose 2011
Sattose 2011Sattose 2011
Sattose 2011
 
Benevol 2010
Benevol 2010Benevol 2010
Benevol 2010
 
Master Thesis presentation
Master Thesis presentationMaster Thesis presentation
Master Thesis presentation
 
Seeing the forest for the trees, UMons 2011
Seeing the forest for the trees, UMons 2011Seeing the forest for the trees, UMons 2011
Seeing the forest for the trees, UMons 2011
 

Recently uploaded

Latin American Revolutions, c. 1789-1830
Latin American Revolutions, c. 1789-1830Latin American Revolutions, c. 1789-1830
Latin American Revolutions, c. 1789-1830Dave Phillips
 
How to Use api.constrains ( ) in Odoo 17
How to Use api.constrains ( ) in Odoo 17How to Use api.constrains ( ) in Odoo 17
How to Use api.constrains ( ) in Odoo 17Celine George
 
3.21.24 The Origins of Black Power.pptx
3.21.24  The Origins of Black Power.pptx3.21.24  The Origins of Black Power.pptx
3.21.24 The Origins of Black Power.pptxmary850239
 
Practical Research 1 Lesson 9 Scope and delimitation.pptx
Practical Research 1 Lesson 9 Scope and delimitation.pptxPractical Research 1 Lesson 9 Scope and delimitation.pptx
Practical Research 1 Lesson 9 Scope and delimitation.pptxKatherine Villaluna
 
Easter in the USA presentation by Chloe.
Easter in the USA presentation by Chloe.Easter in the USA presentation by Chloe.
Easter in the USA presentation by Chloe.EnglishCEIPdeSigeiro
 
CapTechU Doctoral Presentation -March 2024 slides.pptx
CapTechU Doctoral Presentation -March 2024 slides.pptxCapTechU Doctoral Presentation -March 2024 slides.pptx
CapTechU Doctoral Presentation -March 2024 slides.pptxCapitolTechU
 
Quality Assurance_GOOD LABORATORY PRACTICE
Quality Assurance_GOOD LABORATORY PRACTICEQuality Assurance_GOOD LABORATORY PRACTICE
Quality Assurance_GOOD LABORATORY PRACTICESayali Powar
 
Human-AI Co-Creation of Worked Examples for Programming Classes
Human-AI Co-Creation of Worked Examples for Programming ClassesHuman-AI Co-Creation of Worked Examples for Programming Classes
Human-AI Co-Creation of Worked Examples for Programming ClassesMohammad Hassany
 
DUST OF SNOW_BY ROBERT FROST_EDITED BY_ TANMOY MISHRA
DUST OF SNOW_BY ROBERT FROST_EDITED BY_ TANMOY MISHRADUST OF SNOW_BY ROBERT FROST_EDITED BY_ TANMOY MISHRA
DUST OF SNOW_BY ROBERT FROST_EDITED BY_ TANMOY MISHRATanmoy Mishra
 
Philosophy of Education and Educational Philosophy
Philosophy of Education  and Educational PhilosophyPhilosophy of Education  and Educational Philosophy
Philosophy of Education and Educational PhilosophyShuvankar Madhu
 
Benefits & Challenges of Inclusive Education
Benefits & Challenges of Inclusive EducationBenefits & Challenges of Inclusive Education
Benefits & Challenges of Inclusive EducationMJDuyan
 
The Stolen Bacillus by Herbert George Wells
The Stolen Bacillus by Herbert George WellsThe Stolen Bacillus by Herbert George Wells
The Stolen Bacillus by Herbert George WellsEugene Lysak
 
Patterns of Written Texts Across Disciplines.pptx
Patterns of Written Texts Across Disciplines.pptxPatterns of Written Texts Across Disciplines.pptx
Patterns of Written Texts Across Disciplines.pptxMYDA ANGELICA SUAN
 
Clinical Pharmacy Introduction to Clinical Pharmacy, Concept of clinical pptx
Clinical Pharmacy  Introduction to Clinical Pharmacy, Concept of clinical pptxClinical Pharmacy  Introduction to Clinical Pharmacy, Concept of clinical pptx
Clinical Pharmacy Introduction to Clinical Pharmacy, Concept of clinical pptxraviapr7
 
Drug Information Services- DIC and Sources.
Drug Information Services- DIC and Sources.Drug Information Services- DIC and Sources.
Drug Information Services- DIC and Sources.raviapr7
 
How to Make a Field read-only in Odoo 17
How to Make a Field read-only in Odoo 17How to Make a Field read-only in Odoo 17
How to Make a Field read-only in Odoo 17Celine George
 
CAULIFLOWER BREEDING 1 Parmar pptx
CAULIFLOWER BREEDING 1 Parmar pptxCAULIFLOWER BREEDING 1 Parmar pptx
CAULIFLOWER BREEDING 1 Parmar pptxSaurabhParmar42
 
AUDIENCE THEORY -- FANDOM -- JENKINS.pptx
AUDIENCE THEORY -- FANDOM -- JENKINS.pptxAUDIENCE THEORY -- FANDOM -- JENKINS.pptx
AUDIENCE THEORY -- FANDOM -- JENKINS.pptxiammrhaywood
 

Recently uploaded (20)

Latin American Revolutions, c. 1789-1830
Latin American Revolutions, c. 1789-1830Latin American Revolutions, c. 1789-1830
Latin American Revolutions, c. 1789-1830
 
How to Use api.constrains ( ) in Odoo 17
How to Use api.constrains ( ) in Odoo 17How to Use api.constrains ( ) in Odoo 17
How to Use api.constrains ( ) in Odoo 17
 
3.21.24 The Origins of Black Power.pptx
3.21.24  The Origins of Black Power.pptx3.21.24  The Origins of Black Power.pptx
3.21.24 The Origins of Black Power.pptx
 
Practical Research 1 Lesson 9 Scope and delimitation.pptx
Practical Research 1 Lesson 9 Scope and delimitation.pptxPractical Research 1 Lesson 9 Scope and delimitation.pptx
Practical Research 1 Lesson 9 Scope and delimitation.pptx
 
Easter in the USA presentation by Chloe.
Easter in the USA presentation by Chloe.Easter in the USA presentation by Chloe.
Easter in the USA presentation by Chloe.
 
CapTechU Doctoral Presentation -March 2024 slides.pptx
CapTechU Doctoral Presentation -March 2024 slides.pptxCapTechU Doctoral Presentation -March 2024 slides.pptx
CapTechU Doctoral Presentation -March 2024 slides.pptx
 
Finals of Kant get Marx 2.0 : a general politics quiz
Finals of Kant get Marx 2.0 : a general politics quizFinals of Kant get Marx 2.0 : a general politics quiz
Finals of Kant get Marx 2.0 : a general politics quiz
 
Quality Assurance_GOOD LABORATORY PRACTICE
Quality Assurance_GOOD LABORATORY PRACTICEQuality Assurance_GOOD LABORATORY PRACTICE
Quality Assurance_GOOD LABORATORY PRACTICE
 
Human-AI Co-Creation of Worked Examples for Programming Classes
Human-AI Co-Creation of Worked Examples for Programming ClassesHuman-AI Co-Creation of Worked Examples for Programming Classes
Human-AI Co-Creation of Worked Examples for Programming Classes
 
DUST OF SNOW_BY ROBERT FROST_EDITED BY_ TANMOY MISHRA
DUST OF SNOW_BY ROBERT FROST_EDITED BY_ TANMOY MISHRADUST OF SNOW_BY ROBERT FROST_EDITED BY_ TANMOY MISHRA
DUST OF SNOW_BY ROBERT FROST_EDITED BY_ TANMOY MISHRA
 
Philosophy of Education and Educational Philosophy
Philosophy of Education  and Educational PhilosophyPhilosophy of Education  and Educational Philosophy
Philosophy of Education and Educational Philosophy
 
Benefits & Challenges of Inclusive Education
Benefits & Challenges of Inclusive EducationBenefits & Challenges of Inclusive Education
Benefits & Challenges of Inclusive Education
 
Prelims of Kant get Marx 2.0: a general politics quiz
Prelims of Kant get Marx 2.0: a general politics quizPrelims of Kant get Marx 2.0: a general politics quiz
Prelims of Kant get Marx 2.0: a general politics quiz
 
The Stolen Bacillus by Herbert George Wells
The Stolen Bacillus by Herbert George WellsThe Stolen Bacillus by Herbert George Wells
The Stolen Bacillus by Herbert George Wells
 
Patterns of Written Texts Across Disciplines.pptx
Patterns of Written Texts Across Disciplines.pptxPatterns of Written Texts Across Disciplines.pptx
Patterns of Written Texts Across Disciplines.pptx
 
Clinical Pharmacy Introduction to Clinical Pharmacy, Concept of clinical pptx
Clinical Pharmacy  Introduction to Clinical Pharmacy, Concept of clinical pptxClinical Pharmacy  Introduction to Clinical Pharmacy, Concept of clinical pptx
Clinical Pharmacy Introduction to Clinical Pharmacy, Concept of clinical pptx
 
Drug Information Services- DIC and Sources.
Drug Information Services- DIC and Sources.Drug Information Services- DIC and Sources.
Drug Information Services- DIC and Sources.
 
How to Make a Field read-only in Odoo 17
How to Make a Field read-only in Odoo 17How to Make a Field read-only in Odoo 17
How to Make a Field read-only in Odoo 17
 
CAULIFLOWER BREEDING 1 Parmar pptx
CAULIFLOWER BREEDING 1 Parmar pptxCAULIFLOWER BREEDING 1 Parmar pptx
CAULIFLOWER BREEDING 1 Parmar pptx
 
AUDIENCE THEORY -- FANDOM -- JENKINS.pptx
AUDIENCE THEORY -- FANDOM -- JENKINS.pptxAUDIENCE THEORY -- FANDOM -- JENKINS.pptx
AUDIENCE THEORY -- FANDOM -- JENKINS.pptx
 

ICSM 2011

  • 1. Metrics are usually computed at a low level: classes, methods, … / W&I / MDSE 23-4-2012 PAGE 0
  • 2. Multitude of data values obscures a general picture of the system maintainability /W&I / MDSE 23-4-2012 PAGE 1
  • 3. That we are actually interested in! /W&I / MDSE 23-4-2012 PAGE 2
  • 4. You Can't Control the Unfamiliar: A Study on the Relations Between Aggregation Techniques for Software Metrics Bogdan Vasilescu Alexander Serebrenik Mark van den Brand
  • 5. Two kinds of aggregation Same metrics, different Same artifact, different artifacts metrics /W&I / MDSE 23-4-2012 PAGE 4
  • 6. Various techniques can be found in the literature Same metrics, different Traditional: mean, artifacts median, sum, … Econometric inequality indices: Gini, Theil, Hoover, Kolm, Atkinson /W&I / MDSE 23-4-2012 PAGE 5
  • 7. Various techniques can be found in the literature Same metrics, different Traditional: mean, artifacts median, sum, … Which aggregation Econometric technique inequality indices: Gini, Theil, Hoover, should we Kolm, Atkinson use? /W&I / MDSE 23-4-2012 PAGE 6
  • 8. Questions 1. Which and to what extent do the different aggregation techniques agree? 2. What is the nature of the relation between the various aggregation techniques? 3. How does the correlation coefficient change as the systems evolve? /W&I / MDSE 23-4-2012 PAGE 7
  • 9. Qualitas Corpus 20101126 • Qualitas Corpus 20101126r, 106 systems • FitJava v1.1, 2 packages, 2240 SLOC • NetBeans v6.9.1, 3373 packages 1890536 SLOC. /W&I / MDSE 23-4-2012 PAGE 8
  • 10. 1) Agreement between diff techniques • Agreement: • Aggregation: Class SLOC  Package • Techniques agree if they rank the packages similarly We use rank-based correlation coefficient: Kendall’s  /W&I / MDSE 23-4-2012 PAGE 9
  • 11. 1) Agreement: different inequality indices? • Gini, Theil, Hoover, Atkinson – agree • aggregates obtained convey the same information • Kolm does not! /W&I / MDSE 23-4-2012 PAGE 10
  • 12. 1) Agreement: traditional and ineq indices? • mean • Kolm: strong (0,8) and statistically significant (92%) • median, standard deviation, and variance • sum • does not correlate with any other aggregation technique /W&I / MDSE 23-4-2012 PAGE 11
  • 13. 2) Nature of the relation: Typical patterns • Theil is known to be more • Linear relation with a “fat” sensitive to the rich head • Theil increases faster when Gini increases /W&I / MDSE 23-4-2012 PAGE 12
  • 14. Which aggregation technique? (1) • Theil, Hoover, Gini and Atkinson agree • Any can be chosen from the correlation point of view • Some might be “better” in each specific case • easy to interpret: Gini  [0,1] • provide additional insights: Theil (explanation) • negative values: Gini, Hoover − affects the domain! • sensitive for high values: Theil, Atkinson • deviations from uniformity: Gini, Hoover / W&I / MDSE 23-4-2012 PAGE 13
  • 15. Which aggregation technique? (2) • Kolm and mean agree • Kolm is reliable for skewed distributions − better alternative (“by no means”) • Not in the paper: − agreement observed for NOC − but not for DIT! / W&I / MDSE 23-4-2012 PAGE 14
  • 16. Conclusions /W&I / MDSE 23-4-2012 PAGE 15

Editor's Notes

  1. Red line – mean, blue line – medianFurther approaches: distribution fitting, quality models (SIG, SQUALE)
  2. Red line – mean, blue line – medianFurther approaches: distribution fitting, quality models (SIG, SQUALE)
  3. % of the systems with statistically significant correlation between the corresponding indices at the 0.05 levelKendall correlation (rank based)